Testing: Art or Science?

​In responding to a post on /r/softwaretesting, I had a thought that I thought was worth sharing.

The poster was starting out on their career in software testing, and was asking the community the best way to go about this. As you might imagine, the responses were either to follow the ISEB / ISTQB “standards” approach to testing, while the rest told them to reject that in favour of the context-driven “school” of Bach, Bolton, et al.

Rather than attemp to futher radicalise this nascent tester, and make them pick a side so early in their career, I attempted to walk the middle path and, in so doing, had my thought.

There are two schools of thought about how best to define what testing is, and how to train testers. Broadly, these are the ISTQB / ISO29119 “standardised” approach, and the “context-driven” approach of James Bach and Michael Bolton.

The former hews to the idea that testing is comprised of standard practices and approaches that can be applied to any product or project, while the latter maintains that the best approach to testing a product or project will be specific to that situation.

While I personally lean more to the context-driven approach, I think there is room for, and things to be gained from, standardisation and best practice. And that’s where the answer to the “Art or Science?” question comes in.

Here’s my (not very well put) answer from my response to the poster: “That’s maybe what a tester is; a creative thinker able to apply their experiences to the software they are testing to develop hypotheses, and then a scientist who applies standard protocols to verifying those hypotheses who then uses those results to develop more hypotheses.​”

In short, it’s both; Art and Science.

In “artist” mode, the tester applies their knowledge, experience, and creativity to the product to determine what questions they should ask of the software. This is the “context-driven” approach; you test in the way that seems most appropriate for the product at that point, and that approach might change as you learn more and more.

In “scientist” mode, the tester designs the test that will enable to them to answer the questions their “artist” created, and executes that test according to scientific principles.

I’m not suggesting for a second that these “modes” are mutually exclusive. Good Testing is a result of simultanteous Art and Science, each informing the other. Some organisations split the artists – the Test Analysts – from the scientists – the Test Engineers, but this breaks down that feedback loop between the two testing roles.

I believe that a tester can only do Good Testing if they are free to perform both roles. Therefore, there is as much a place for a formal grounding in testing techniques and terminology – the “science” – as there is for the creative freedom for a tester to test as they see fit – the “art”.

What is a tester?

From the last post, we have an idea what testing is, but what is a tester? Another good question. However, insofar as a question can be wrong, this one is, which leads to my first answer.

A Tester is not a “What”. A Tester cannot be a “What”. A Tester is a “Who”; a human being. “Why can’t a tester be non-human?” I hear you ask. 

Well, there are two parts to what a tester does; testing, and checking. While these might sound like synonymous terms, they are discrete in a fundamental way.

Testing is a creative process, a process of determining what questions to ask, then determining what the answer is, and if that answer is correct. Checking is a non-creative process of determining whether the answer to those questions is still correct.

Checking can be automated; you’re simply re-asking a question you’ve already formulated, and verifying the result is what you expect. Testing, because it is creative, can only be done by humans. 

Until such time as we get real artificial intelligence, we need humans to formulate those questions; things like “What happens if I enter a date of 0/0/00?” and “What happens if I press this button 20 times in a row on February 29th?” Therefore, a valid answer to the original question “What is a tester?” is “Irreplaceable” (at least for now).

To expand on the point that a tester must be a human, it’s precisely the things that differentiate us from most other forms of life that make good testers; curiosity and intellect.

A tester is someone who, when presented with something new, will come up with hundreds of question starting “What happens if….”, and who then proceeds to find the answers. A tester has a compelling drive to understand, who can express their ignorance in the form of questions, who has the skill to determine how to get an answer, and the perseverance to continue until they do.

The underlying skills, as I have said, are not particular to certain individuals. They are inherent to our species. Those traits are the reason why I am sitting here typing, and not squatting in a cave somewhere. Homo sapiens is the dominant species on this planet because at some point someone thought “What’s over the horizon?” and, much more importantly, someone went to look. 

Testers are the people who have to go look. They can’t not know. And it’s that compulsion to know, to understand, combined with the drive to find the answer that makes a tester.

What is Testing?

“What is testing?” Now there’s a question. Simple, on the face of it. But like many simple-sounding questions, perhaps not so easy to answer. But before I provide an answer , picture this.

You are helicoptered into a remote jungle clearing in the Yucatan peninsula, and told that there is a Mayan pyramid to the North-West. Your job is to navigate to that pyramid, and survey it. You have a map, a compass, and a machete. And that’s it.

On the face of it, you have the necessary tools to complete the task. The path seems pretty clear, but here’s the point. You cannot know what is beyond the treeline. Sure, you have a map, but the map is not the territory. Does the map tell you which trees have jaguars in them? Or which have hostile tribespeople behind them? Or which trees have grown up since the map was made? There are an awful lot of trees out there.

Which brings us to the answer. Testing is exploration. It is learning. It is about answering the question “What’s behind that tree?” It’s also about answering the question “Do I need to know what’s behind that tree?” the answer to both of which is “I don’t know until I go look”.

Another answer; “test” is a verb, not a noun. Testing is an activity, not an artefact. Having a hundred, or a thousand, or a million tests that aren’t run is the same as having no tests. Just planning to look behind every tree has no value. Actually looking has immense value, even if it is just to determine that it wasn’t worth looking.

That learning process takes time, and you don’t have time to look behind every tree. You have a finite period to determine where to look, with only your experience to guide you, and you will only be able to cover a tiny fraction of that jungle. If you could automatically test every combination of 100 Boolean variables at the rate of 1 per second, it would take many orders of magnitude longer than the age of the universe.

Testing is not a journey of a thousand miles. It is an endless search which, like learning, is “a movement of knowing which has no beginning and no end” (Bruce Lee). Testing never ends, it merely stops at a ship event, and the tester is left looking out on an endless jungle of trees that they haven’t looked behind yet. Which is why when you ask a tester if they are finished testing, their answer will – or should – be “I’ve barely even started”.

Testers are required to stand facing a wall of trees, in the middle of millions of square miles of dense jungle with nothing to guide them but a questionable map, a battered compass, and a rusty machete, and know where and, more importantly, how to look. Even more importantly, a tester has the drive and the desire to keep looking, even when they’ve found nothing but leaves and mud for days, keeping their focus and attention because you never know when someone might fire a dart at you.

Testing requires a spirit of adventure, a questioning mind, a desire to know, and a judicious application of experience that allows the tester to interpret the map and compass, wield the machete, and set off into the unknown.

Microsoft’s Surface Pro UK Release Schedule

Late 2012: Microsoft announces they are making the Surface with Windows 8 Professional.

February 2013: Microsoft release the Surface Pro in North America

April 2013: Microsoft release the Surface Pro in China

2020: Technology continues to advance at an geometric rate. There is still no word from Microsoft as to Surface Pro pricing for Europe.

2150: Due to resource pressures, Humanity polarises into two geopolitical behemoths; the United Atlantic Alliance, and the People’s Democratic Federation.

3000: Humanity develop fast interstellar travel, begin colonising nearby systems. The two warring hyperpower blocs compete for resources. The Moon is destroyed. The larger fragments impact the Earth, causing continent-wide destruction and rendering entire hemispheres uninhabitable.

3100: The last humans leave the now dead Earth,

4000: As Humanity harvest the solar system for resources, devouring entire planets, the gravitational instability causes Sol to go supernova. Humanity harnesses all residual matter to fuel their technological advancement.

4500: The last human becomes post-physical. All humanity exists as pure energy beings in the virtual construct known as The Ark.

6000: The post-Human entity known as The Ark begins consuming matter to fuel its growth. It becomes more massive than any known body. Space/time itself begins to contract.

6100: The in-rushing galaxies begin to impact, generating a cloud of galactic fragments accelerating inwards at high fractions of c.

6101: In the last vestige of space/time before it collapses in on itself, and the very concept of space/time ceases to exist, Microsoft release the Surface Pro in the region of space/time formerly occupied by the geopolitical entity known as the United Kingdom

In summary, I’m not prepared to wait that long.

Schrodinger’s Cat Experiment: Mark I

As cultured and educated people, you are no doubt aware of Schrodinger’s Cat. What you are probably not aware of is the distressing truth behind this famous thought experiment and the resulting cover up that has lasted for nearly 80 years.

Knowing the specifics of the experiment, you may reasonably surmise that Erwin was no great lover of cats. On first inspection, it may not be entirely clear why this may be the case. However, I will, in this post, theorise as to the cause of this.

We flashback 80 years to the early 1930s, where we find Erwin Schrodinger jetting between Berlin, Oxford and Princeton, and corresponding with one Albert Einstein about quantum mechanics.

Schrodinger had a theory that needed testing, that at some level, matter can coexist in different states. He had yet to make the leap that it might only apply at the quantum level. Picture the scene; Erwin sits in his office, deep in thought, when his cat Cuddles1 jumps onto his lap. It is there, scratching Cuddles idly between the ears, that he formulates the basis of that famous experiment.

Hang on, you’re thinking to yourselves. Schrodinger’s Cat is a thought experiment. It was never carried out for real. That may indeed be true, and I’m about to tell you why.

As we have already established your intellect and learning, I can assume that you are also familiar with Godwin’s Law, which states that “As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches 1”.

RocketBootKid’s Law, which I am hereafter bequeathing to Mankind, can be stated in similar terms, thusly; “Over time, the probability of a cat owner being randomly attacked by their cat for no rational or logical reason approaches 1”.

Back to Schrodinger. He and his cat are in the box, while Schrodinger ponders how the Observer Effect may alter the results of his experiment. The cat, being a cat and therefore experiencing the multiverse on a plane of existence completely devoid of logic and reason, suddenly sinks five of its six ends into Schrodinger’s tender underparts.

Flashforward a few months, and it is only his dedication to scientific rigor that finds Schrodinger still occupying the box, when the better parts of him, his tender and now swollen underparts in particular, are begging him to develop a less painful experimental paradigm.

At some point, Schrodinger had a final falling out of love with Cuddles. The specifics of this event are wasted to the pages of history, but the ramifications for Cuddles are dire. Schrodinger, in a late-night manic episode, arrives at the specifics of the device with which we are now familiar.

His housekeeper, who services were suddenly dispensed with, would later comment that she hadn’t seen Cuddles around for a while. Perhaps fearing a visit from the RPSCA, Schrodinger was careful to categorise the fate of Cuddles as a “thought experiment” when he published his theory in “Die gegenwärtige Situation in der Quantenmechanik” in 1935.

In a museum somewhere, in a display case, is a box inside of which, most definitely, is a dead cat.

1 Schrodinger may or may not have had a cat, which may or may not have been called Cuddles. I think he’d be satisfied if I said all of the above were possible.

The Ephemeral Nature of Knowledge

Okay, it’s a pretentious title, but you’re just to going to have to deal with it.

This post is at least partly to defend my (annoying?1) tendency to never state anything in definitive terms. There are two reasons for this.

The first is that I find absolute, unilateral, or dictatorial statements inherently distasteful. I was going to say inhuman, but that’s perhaps a bit strong. The reason that the overdeveloped thesaural region in my brain returned that word is that a defining characteristic of humans is our ability to work together, to establish a consensus, to collectively achieve more than the sum of our parts.

A unilateral statement – the product of a single human – is inherently exclusive and therefore destructive to the power of the collective2.

The second is that the very nature of knowledge is fleeting, dynamic, you might even say ephemeral. In fact, someone already did. I remember very clearing taking Physics at school and being told in later years to forget what I had previously been taught. Not because what I had been taught was incorrect, but because it was too high-level, too abstract.

The same is true of all areas of expertise, physics perhaps more than most. There are levels of understanding that are perfectly sufficient for most, but which gloss over the finer, more detailed points that are vital for the development of that subject.

Another factor is that the depths of human knowledge are constantly being explored, only to find that it’s actually a lot deeper than previously thought. Unless you’re keeping abreast of all recent discoveries throughout the entire sphere of human knowledge, you’re going to be at least slightly inaccurate every time you open your mouth.

It is therefore extremely difficult to make any definitive statement about anything, other than that which you know inside and out, without it being based on a incomplete understanding of that subject, and therefore not entirely accurate. Now, most people don’t worry about this, and most of the time it really doesn’t affect much at all.

To the extreme pedants among us, and to those who value community consensus over dictatorial pronouncements, it’s an important distinction, and one that should be accepted.

1 I assume it must be at least slightly annoying, but that’s just a guess.
2 I cannot use that word without the Borg or Communism coming to mind.

The Dilbert Stages of Professional Cynicism

Over the years, I have come to believe that there are three stages to one’s professional career, and that those stages may be defined relative to ones opinion of the work of Scott Adams, specifically ‘Dilbert’.

This theory is borne of my own experiences, but like most of the ideas on here, is unlikely to be terribly original, well thought through, or even succinctly put. In an effort reduce the word count a bit, I’ll apply Ockham’s Razor, shave some words off, and define the stages as follows; 

  • You don’t think Dilbert is funny
  • You think Dilbert is hilarious
  • You think Dilbert is based on your professional life.

Or, to put it another way;

  • You don’t get Dilbert
  • You get Dilbert
  • Dilbert gets you.

These three stages reflect the effect of corporate reality as it slowly eats away at the fresh-faced young employee, heretofore swaddled in the protective nirvana of educational utopia. They are the measure of how much of the child has been replaced by corporate robot, of how much idealism has been replaced by cynicism.
 
Someone I know is very keen that people aren’t cynical and go into things with an open mind, with the attitude that things can be done.As I’ve said before, I consider myself to be both a realist and an idealist. I try to nurture the hope that all things are possible, but I’m not going to stay up all night waiting.

People are cynics for a reason. Cynics are not born; we are made, or rather corrupted. While we may be cast in our mother’s womb, we are forged in the fires of industry, in the furnaces of commerce. It is in this inhospitable environment that the naif in all of us has, at some point, had our eyes forcibly opened a la Malcolm McDowell in A Clockwork Orange.

Lord Acton was only halfway there; Power may corrupt, but its lack is just as harmful, albeit in different ways. Absolute power may make you believe that you can do what you like, but the lack thereof makes you believe that nothing is possible and that, whatever you do, forces beyond your control serve to constrain you.

Eventually, you stop trying. Only the blindest optimist or greatest fool would continue in the face of a life’s experience. Indeed, Einstein defined insanity as “doing the same thing over and over again and expecting different results”.

To return to The Three Stages, the first two stages are merely precusors to the transition to Stage 3, a transition that represents a paradigm shift in the professional outlook of the person in question. A person who has made the transition from Stage 2 to Stage 3 has been “broken”, a term that intentionally mirrors the process by which horses become rideable.

While horses are more useful once broken, broken employees are often less useful. While they are still useful and important members of the team, they are less likely to go the extra mile.

The point at which employees break is often quite tangible. Someone previously level-headed and conscientious will suddenly become outspoken in meetings, or their grin get slightly manic, or “Thursday Afternoon Effect”1 behavior happens on other days of the week.

We all know the signs, and we all silently mourn the passing of their youth, and think “You’re one of us now”.

1 The Thursday Afternoon Effect is the point on Day 11 of a 12-day week full of 10 hour days when everything, even quite sad things, become hilariously funny, and the slightest thing can send you off into wild paroxysms of maniacal laughter.


I am working on an update to the theory that posits a fourth stage, which may be exemplified by the phrase “You know what, fuck that, it doesn’t have to be this way”. Whether this is merely an acute remission in otherwise chronic decline, or the turning of the tide, is the subject of further study.

    Balls.

    Two years ago, the Scottish Government released Abdelbaset Ali Mohmed al-Megrahi, the only man convicted of the Lockerbie bombing of Pan Am Flight 103, on grounds of ill health. He returned to Libya to a hero’s welcome.

    Following the revolt against the Gaddafi regime, al-Megrahi’s family have been unable to give him the necessary cancer medication and have pleaded with the Scottish Government to provide it.

    My first six responses to this request are, uncharacteristically I feel, “Fuck him”.

    That the family have the balls to request that the country that suffered most at his hands – after the US – prolong his life, after he ended so many, feels like a massive boot to Scotland’s balls.

    My seventh thought is that since we released him from prisoner on medical grounds, and continue to monitor him, we retain some duty of care. And since there were doubts around some of the evidence in his trial, it’s possible he’s just some terminal cancer sufferer who got fucked over, in more ways than one.

    All that said, and while it grates against my usual moderate stance, I’m going with Response 1: “Fuck him and his balls”.

    Creating the Universe with Unix Commands

    # In the beginning God created the heaven and the earth
    > su
    Password: jehovah
    > mkdir heaven earth

    # And God said, Let there be light, and there was light.
    > mkdir light

    # And God divided the light from the darkness. And God called the light Day, and the darkness he called Night.
    > cp light Day
    > mv light Night

    # And God said, Let there be a firmament in the midst of the waters, and let it divide the waters from the waters.
    > mkdir waters firmament waters
    mkdir: cannot create directory ‘waters’: File exists

    # And God made the firmament, and divided the waters which were under the firmament from the waters which were above the firmament: and it was so. And God called the firmament Heaven. And the evening and the morning were the second day.
    > mv firmament heaven

    # And God said, Let the waters under the heaven be gathered together unto one place, and let the dry land appear: and it was so. And God called the dry land Earth…
    > mkdir earth

    # .. and the gathering together of the waters called he Seas: and God saw that it was good
    > mv waters seas

    # And God said, Let the earth bring forth grass, the herb yielding seed, and the fruit tree yielding fruit after his kind, whose seed is in itself, upon the earth: and it was so.
    > cd earth
    > mkdir grass seed fruit_tree

    # And God said, Let there be lights in the firmament of the heaven to divide the day from the night; and let them be for signs, and for seasons, and for days, and years
    > cd ../firmament
    gsh: cd: ../firmament: No such file or directory
    > cd ../heaven
    > mkdir lights

    # And God made two great lights; the greater light to rule the day, and the lesser light to rule the night: he made the stars also
    > cd lights
    > mkdir sun moon stars

    # And God said, Let the waters bring forth abundantly the moving creature that hath life, and fowl that may fly above the earth in the open firmament of heaven
    > cd ../waters
    > mkdir creatures
    > cd ../heaven
    > mkdir fowl

    # And God created great whales, and every living creature that moveth, which the waters brought forth abundantly, after their kind, and every winged fowl after his kind: and God saw that it was good
    > cd ../waters/creatures
    > mkdir whales

    # And God blessed them, saying, Be fruitful, and multiply, and fill the waters in the seas, and let fowl multiply in the earth
    > nice whales
    nice: whales: Permission denied

    # And God said, Let the earth bring forth the living creature after his kind, cattle, and creeping thing, and beast of the earth after his kind: and it was so
    > cd ../../earth
    > mkdir creatures
    > cd creatures
    > mkdir cattle

    # And God said, Let us make man in our image, after our likeness: and let them have dominion over the fish of the sea, and over the fowl of the air, and over the cattle, and over all the earth, and over every creeping thing that creepeth upon the earth
    > mkdir man
    > groupadd mankind
    > useradd man -G mankind -m /earth
    > cd /earth
    > chown -rf man:mankind *
    > chmod -r 777 man:mankind *
    > cd /heaven
    > chown -rf man:mankind *
    > chmod -r 777 man:mankind *
    > cd /waters
    > chown -rf man:mankind *
    > chmod -r 777 man:mankind *

    # So God created man in his own image, in the image of God created he him; male and female created he them
    > useradd woman -G mankind -m /earth

    # And God blessed them, and God said unto them, Be fruitful, and multiply, and replenish the earth, and subdue it: and have dominion over the fish of the sea, and over the fowl of the air, and over every living thing that moveth upon the earth
    > nice man
    What manual page do you want?
    > nice woman
    nice: woman: Permission denied

    The Great British Earthquake

    The 2011 earthquake struck in the middle of a sultry English summer afternoon, its epicentre the borough of Tottenham in North London. Within hours, the resultant aftershocks had spread to many other areas of the city and, over the next few days, to other English cities.

    This earthquake, however, was not geological; it was social. The trigger was the shooting by Police of Tottenham man Mark Duggan on Thursday, 4th August 2011. While the trigger was at the Police-Public Tension point of the fault, there are many asperities along the social faultline through British society.

    The earthquake allowed the release of pressure that built up at all the other points of tension as well; Haves vs Have Nots, The Nanny State, the list goes on. However, while the main shock was around Police-Public Tension, the main cause of the ensuing aftershocks – rioting – appear to be centered around the fiscal gap between the Haves and the Have Nots, something that has been given media coverage in the US in light of their current financial crisis, but which has been largely ignored in the UK.

    To quote / paraphrase Joe Friday in Dragnet: “There are those that have it, and those that want it. Those who have it, flaunt it, no matter how they got it. Those who want it can get it by attempting to better themselves in a supportive society cheering them on. Or they can take it the easy way…”

    This is what we’re seeing. The looters are taking things they feel they cannot get legally. They are effectively bootstrapping themselves financially towards the rest of society; the Haves. Leaving the facts that it’s illegal and ruins lives aside for a moment, one could argue that looting is an ultimately stabilising factor in situations like this. As the looters become the Haves, they then become invested in stability and calm, so that they may benefit from their ill-gotten gains. If a non-uniform distribution of wealth is [a|the] cause, then a redistribution of wealth, legal or otherwise, is inherently stabilising.

    Given a destabilising event, those with more to gain than to lose will seize the opportunity. People have to be invested in the success of society in order for that society to survive. To quote from the movies again, in this case Xander Cage from xXx, “if you’re gonna ask someone to save the world, you’d better make sure they like it the way it is”. The longer society fails to address the needs of everyone, then the greater the tensions and the greater the likelihood of seismic events like these.

    But what are the causes of these tensions? As with most things that defy digestible media soundbites, they are legion and exceedingly complex. Let’s concentrate of two areas; 1. Why did it start? and 2. Why did it expand?

    Now, writing this as I do from my resolutely white, middle-class haven in the currently riot-free north of the British Isles, I do not pretend to be anything approaching an expert on the contributory factors, nor am I a psychologist. All this probably means I should keep my trap shut and my opinions to myself. But that’s what this blog is: me keeping my opinions to myself, safe in the knowledge that no-one will read them but me.

    So, the trigger to this situation appears to be tensions between Police and ‘minority’ sections of London’s populace. Interviews with local people suggest that Police intrusion into their lives is constant and disrespectful, fostering a distinct ‘us-and-them’ attitude. In the defence of the Police, the fact that black people are 26 times more likely to be stopped and searched than a white person is bourne of the amount of concealed weapons discovered in these searches.

    Leaving the Police’s attitude aside for another post, what is it that causes these people to carry concealed weapons? One option is that they believe that, generally speaking, society does not look after them, so they have to look after themselves. They band into gangs to gain a sense of belonging, importance and power that society, in it’s current state, does not afford them.

    The other option is that it gives them a sense of power, power that society for the most part denies them, whether that’s the power to elect a representative that will represent them, or the power to determine the course of their own lives through education and employment. In the absence of this sense of empowerment within society, I can understand the attraction to step outside it.

    So, the underlying “why” of the current situation is the same as it has always been. At every stage of human development and society, there have been elements of the population that are unable to make best use of the current nature of society, and so find themselves marginalised.

    The exacerbating factor to all this is that our society has evolved faster than human nature. Underlying our more developed notions are those baser instincts geared towards self preservation. In our modern society, where people are living in ever larger groups and so would benefit from a more collective approach, these base instincts are anathema to the common good.

    This selfish nature, allied with the ability – or lack thereof – to benefit from society, is what drives the wedge between the Haves and Have Nots.

    And therein lies the answer to “Why did it expand?”. Any breakdown in society allows those marginalised by that society their greatest opportunity for gain.

    So, what may have started due to perceived Police brutality, and was hijacked by those looking for personal profit, will naturally peter out. What happens then will
    determine when the next earthquake will strike.


    David Cameron has said that the looters will “feel the full force of the law”. This feels like an empty threat, for the following reasons;
    1. given the number of looters and rioters, there is no way for The Law to catch and prosecute them all; the impunity of numbers.
    2. if they do get them to court, proving that a. it was them and b. they did it maliciously, rather than simply getting swept up in the mob, will be next to impossible. Basically, all that will happen is that the courts will be clogged for years and very few sentences will be handed out.
    3. it is difficult to threaten those who have nothing. The only thing you can remove is their liberty, and the jails are already full. Giving someone with nothing a fine that they can’t pay achieves nothing other than to incentivise more misdeeds.

    Because your whole life is a test