All posts by Richard Paterson

What can I do about Mr Creep?

To clarify: the title question is absolutely not a “/shrugs/ what can I do?” shirking of responsibility. It is a sincere, open request for guidance. Because I’m worried.

I worry that I won’t be able to accurately put across what I want to say, and that these words will hurt instead of help.

I worry that these sorts of abuses happen every day, which they pretty obviously do.

I worry that inattentional blindness means I don’t even register or recognise them when they do.

I worry about the effect incidents like this have on the women in my life, professional and personal, all of whom I hold in the highest regard.

Most of all, I worry that it’s me.

To be clear, I’m not worried for me, but that the problem is me. I am worried that I might be making the lives of the women I work with even the tiniest bit worse.

I consider myself a feminist. I worry a lot that I’m a crappy feminist, and actually make things worse while thinking I’m helping. I want to support the women around me as best I can, and better than I do.

If any of these concerns feel familiar to you, dear male reader, then I hope that that means we’re at least aware that we might be at fault. If so, it is our responsibility to awaken and improve that awareness in others, while we continue to work on ourselves.

I am resolved to do better and I want help to be better. You deserve it.

Test Plans are useless. Test Planning is not.

This article triggered me – in a good way – to describe what I think are the biggest reasons for having a test plan. Obviously, the mileage you get from your test plan will vary, depending on your company, environment, customer, regulatory environment, amongst other things.

As we know, simply having a document doesn’t, in and of itself, add much value. Simply having lots of test cases doesn’t add value. Running them does. Actually, just having documents is waste.

In these agile days where communication is rightly valued over documentation, the poor old test plan gets a bad rep, and is often neglected, or produced on reflex and never read.

There are at least two good reasons for producing a test plan, which I will introduce, naturally, through the medium of appropriate quotes.

Planning is a verb

“Plans are useless. Planning is indispensable”. – Dwight D. Eisenhower.

The value we get from test plans is not from the document, but from the planning process. In producing / reviewing the document, we are effectively testing the project. By asking questions and eliciting responses from the relevant people, we are seeking out where information is missing, or confused, or inconsistent.

Testing is pretty much the first role / stage in a project where questions like “What operating systems do we support?” and “Do we support HTTPS?” have to be answered before the work can be completed. It is therefore vital that we ensure that we ask these questions as early as we can so that a) answers can be found, and b) we can start testing those answers.

Over time you can use the test plan as a repository for organisational lessons learned. In the same way as you might update a test case to test for a specific bug that tends to creep back in, or to ensure that you test a out-of-the-way piece of functionality that you tend to overlook, update your test plan so that you ask the questions that will prevent you making the same mistakes again.

The best way to get value of out of the process around test planning is to involve as many stakeholders and interested parties as possible in the production / review of the plan. Your test plan doesn’t have to be perfect; its primary goal is to serve as a means of reaching agreement between everyone involved, and only subsequently to record the specifics of that agreement.

In longer releases, it’s worth periodically revisiting the plan – really, re-reviewing the plan – in case any of the details have changed that impact any aspect of the planned test activities.

In short, the test plan is actually best used as a vehicle to stimulate communication and record decisions.

Ignorance is not bliss

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” – Mark Twain.

It is easy in testing to get too comfortable in one’s testing approach. You might be testing yet another release of the same product. Everything is the same as it was last time, so why bother doing test planning?

Well, how do you know everything is the same as last time? Have you checked? Doesn’t your role exist precisely because shit happens? People make mistakes all the time. Things change, and people don’t tell other people. Happens all the time.

As the quote says, it’s not the things you don’t know that cause problems. It’s the things you are both 100% sure you’re right about, and even 1% wrong about, that bite you. As a tester, any feeling approaching total confidence in your knowledge about anything should cause warning lights to blink in your consciousness.

Therefore, force yourselves to go through the planning process again, even if you think everything is the same as last time.

Only difference between science and messing about…

“If you don’t write it down it never happened”. – Somebody

We do test planning because it adds value, to us as a function and to our projects. As a result, it is part of our testing process to ensure that we all do it.  Because it’s part of our process, it’s subject to audit as part of our ISO9001 certification. If we cannot show that any test activity took place, then we are not following our own process. An external ISO auditor could raise non-conformities which, depending on the level, could jeopardise our certification.

Of course, you might not have any process, contractual or regulatory reason or need to document your test planning, in which case it’s up to you to decide whether it adds value or not. I’d personally feel safer writing down agreed test planning points, if only for ass coverage purposes.

In summary…

  • Make test planning a conversation
  • Don’t get complacent about what you think you know
  • Write it down while it adds value

Don’t Shift Left; Expand Outwards and Accelerate Failure

My boss sent round this article written in September 2017 from TechTarget about embracing shift-left testing as “the latest theory”. The reason he sent it around was not because we should look into adopting it, but because we’ve been doing it – and I checked – since 2008. “Sounds familiar” he quipped.

“Shift left”? Bleurgh!

Firstly, let’s address the term “shift left”. I’ve used the term for a while to explain how we test, but recent Twitter noises make it more clear that the term is limited.

It suggests movement in one direction. Put like that, I don’t think that’s what we really mean. We’re not suggesting that, in order to start getting involved earlier in the lifecycle that we stop doing testing at the end, are we? What we’re really doing is expanding the range of activities that testers get involved in.

“Left” leads to “right”, which suggests a one-dimensional reality in which your options are severely limited. Modern organisations are multi-dimensional, so why limit ourselves to a single direction in a single dimension? Are the activities that occur notionally to the left of testing in the software development lifecycle the only things we can usefully get involved with?

“Shift left” is about getting testing involved earlier in the software lifecycle so we can try and stop bugs even being written. Broadly, it’s about adding value sooner rather than later. But, again, why limit ourselves to adding value?  Why not look in the other direction – towards operations – and see if we can extract value to make our testing better in the future?

But wait, I hear you cry. If we’re expanding, surely we’re spreading ourselves more thinly, and doing less testing at the end? That may be the case, but the idea is that, by getting involved sooner and identifying problems earlier, an hour spent early in the release should save you many times that late in the day. It’s an investment that you hope pays off as fewer bugs and regressions.

Dan Ashby's Continuous Testing in DevOps
Dan Ashby’s Continuous Testing in DevOps

As I typed this, Dan Ashby‘s surely-famous-by-now image popped into my brain. You can test everywhere, not just a bit more to the left.

So instead of “shift left” I’m going to call it “expand outwards”. It doesn’t suggest that you have to give up anything, and it’s direction-agnostic. You can expand in any direction that lets you add or extract value.

What does “expanding outwards” look like?

We’ve been expanding testing outwards ever since we adopted agile-like behaviours. As a company, we embraced Agile methodologies as an R&D operation. Importantly, I had the freedom to come up with a test process from scratch that would allow testers to get involved from the earliest possible point in the release.

Initially, we did just expand in one direction; “left”, i.e into development, design, and requirements. We got involved in design and estimation meetings, we reviewed requirements and user stories, and we got developers to review our test ideas.

Over time, we looked to expand in other directions; firstly to the “right” and Customer Support.  As the people who work closely with the customer and who support the products in live usage, they are the role who know the most about real-world customer usage of the products. They are a vital resource to get involved if you can. We “shifted them left” into testing so that they could add value to us.

We also get involved with testing documentation, and tech authors share our test environments where possible.

Those are just some examples. We didn’t stop there, and neither should you. Remove your filters and think about everyone even loosely associated with the product, which probably means everyone in the business, and includes your customers. Think about what value you can give to, or extract from, each / any / all of them.

Let’s step back; what are we trying to do?

When we first looked to adopt Agile, we spoke to a consultant who probably said lots of words, but two have stuck with me ever since;

accelerate failure

Put another way, if you know you’re going to make mistakes – which you will, because you’re a human being – then do everything you can to make them, or identify them, as early as possible so it’s as cheap as possible to fix them.

We need to accelerate not just the failure, but the remediation loop as well. The goal has to be that errors spend as little time in the system as possible, whether they be requirements ambiguities, or code defects, or whatever. The longer an error remains in the system, the more it becomes cemented in, compounded upon, worked around, and expensive to remove.

“You mustn’t be afraid to dream a little bigger”

What we’re really trying to achieve is the acceleration of failure across the entire software development lifecycle, not just testing. We can expand testing outwards to try and find problems sooner, but if the rest of the operation doesn’t follow suit, or inhibits your ability to do testing in different places, then you will encounter a glass bubble beyond which you cannot add or extract value.

Adoption of accelerated failure needs to be systemic, not piecemeal, affecting only a single role. In order to realise all the value of accelerated failure, everyone needs to do it, or at the very least accommodate it.

The hardest part of this whole acceleration of failure business is overcoming any cultural momentum around who does what and when. An agile (in the small, nimble sense) company, with fewer organisational and physical boundaries between roles / projects, will be able to change direction more easily than a larger, distributed, older organisation whose culture is probably more rigid.

Even in a small organisation, it’s not straightforward. From experience, the hardest part can be getting other roles to allow testing to be part of the conversation. This is probably due to perception issues around what testers do more than anything else. Again, from experience, testers working closely with developers quickly erodes those concerns as the value testers add becomes more evident.

How do I  Accelerate Failure?

Think about the roles you interact with regularly. They are likely people involved in the software development process. Learn as much as you can about how they operate and what they do. Pair with them, read their process docs if they have them. Figure out what info they have that you can use, and vice versa.

Encourage them to reciprocate and to remove any barriers. There should be no reason for barriers to sharing of information that ultimately benefits the business and the customer. Everyone in the business is there to provide, or support the provision of, products to the customers.

Then expand your focus to other roles, and repeat. Ignore organisational hierarchies and chains of command. Knock on doors, cold-email people. Walk about and have coffee.

You’re a tester; a sponge for knowledge. Absorb it wherever you can find it, and let it moisten the brains of as many others as you can.

Don’t limit that value-add to the person in the seat to your left.

 

PS. Communicate!

If I come across as smug about having taken this approach for nearly ten years, it’s because I am (a little). However, the important takeaway for us all here is to be aware of your approach and compare it to the approaches that others are using. You might be ahead of the curve. You might be putting a new slant on an old idea. You might be using an old idea whose time has come back around.

Don’t assume that what you’re doing, or how you’re doing it, is normal, nothing special, or that everyone else is doing it (better perhaps). Consider that, just maybe, how you’re doing it is innovative, unique, The Next Big Thing.

Setting up an ELK stack with Docker

Recently I’ve been working in an environment comprised of nearly 60 Docker containers. When things go wrong, as they sometimes do, you can make educated guesses about which logs to look at but sometimes you’ll guess wrong. Or you know which ones to look at but want to watch several at once without having 8 PuTTY windows open.

You need something more fit for purpose. Enter ELK. ELK stands for Elasticsearch, Logstash, and Kibana. Rather than explain them in the order they appear in the acronym, I’ll do it in the order in which they are used, but rather than attempt to explain it myself and screw it up, the following summaries are from this helpful O’Reilly article.

Logstash “is responsible for giving structure to your data (like parsing unstructured logs) and sending it to Elasticsearch.”

ElasticSearch “is the search and analysis system. It is the place where your data is finally stored, from where it is fetched, and is responsible for providing all the search and analysis results.”

Kibana “allows you to build pretty graphs and dashboards to help understand the data so you don’t have to work with the raw data Elasticsearch returns.”

What am I building here?

OK, so what does all that look like? It’s pretty simple. You configure your logs to write to Logstash. Logstash then filters the log messages, turning the unstructured messages into structured data. That structured data is output to Elastic, where it is indexed. Kibana then allows you to query that indexed structured data to discover useful information and visualise it.

Getting Started

This is the easy part. Follow the instructions here to clone the ELK stack to your machine and start it up. This basically gets you to pull the three containers and fire them up, with some configuration taken care of. Nothing in this life, however, is for free, so there is some work to be done.

[I must offer all glory to Anthony Lapella for producing such a beautifully wrapped ELK package]

Wiring

I’ve observed before that modern software development is mostly plumbing. Happily, the ELK stack is pre-configured such that you don’t need to wire together the bits of ELK. However, with ELK, you do need to;

  1. Configure your Docker containers to log to a location,
  2. Configure Logstash to pick them up from that location.
  3. Configure Logstash to filter the data.
  4. Configure Logstash to send the data to Elastic

Configuring container log location

Because I might want to run my containers without sending their logs to ELK, I need a way to turn it on and off without too much messing around.

The way I did this was by using a second docker-compose.yml file. This secondary file – which I called “elk-logging.yml” – contains a section for each of your containers. Each section contains the following;

my_service:
    log_driver: syslog
    log_opt: 
        syslog-address: "tcp://localhost:5000"

What this does is tell the container to use syslog, and to send syslog over TCP to port 5000.

So what you need to do is create your secondary YAML file – elk-logging.yml – with as many of the above sections as you have containers you want to log to ELK, with the “my_service” parts replaced with the names of all your containers.

Configuring Logstash input

The next step is to configure Logstash’s input to listen on that port. Fortunately, the ELK stack you cloned earlier already has this configured. The configuration file in in docker-elk/logstash/pipeline/logstash.conf.

Looking at this file shows an input section that shows it listening on TCP:5000.

input {input { tcp { port => 5000 }}

So you don’t need to do anything; this is just for information.

Grokking in Fullness: configuring Logstash filters

This is the most fiddly part of the exercise, because you need to mess about with grok regexes. When you’re dealing with this many containers, the chances of them all using the same log output syntax is remote. There are a couple of approaches you could take;

  1. Specify multiple filters, and fall through each until a working filter is found.
  2. Specify a single filter that is powerful enough to recognise optional components as and when they appear.

I tried both of these approaches, but struggled to get the syntax in the logstash.conf file right, so I eventually settled on the One Grok To Rule Them All. And this is pretty much what it looks like;

<%{NUMBER:priority}>%{SYSLOGTIMESTAMP:syslogtimestamp}\s%{GREEDYDATA:container}\[%{POSINT:containerprocessid}\]\:\s*((?<springtimestamp>%{YEAR}[\-\/]%{MONTHNUM2}[\-\/]%{MONTHDAY}\s%{TIME}))?\s*((\[)?%{LOGLEVEL:loglevel}(\])?)?\s*(%{POSINT:processid})?\s*(---)?\s*(\[\s*(?<thread>[A-Za-z0-9\-]*)\])?\s*((?<loggingfunction>[A-Za-z0-9\.\$\[\]\/]*)\s*\:)?\s*%{GREEDYDATA:logmessage}

Impressive looking, right? I won’t attempt to explain every component, but will try to summarise. Firstly, my log messages seem to broadly be “Syslog format followed by Spring Boot format”, e.g.

<30>Sep 11 10:18:58 my_container_name[1234]: 2017-09-11 14:18:58.328 WARN 14 --- [      thread_name] a.b.c.d.CallingFunction : Oops, something went a little strange, but nothing to worry about.

Everything up to the red colon is SysLog, everything after is Spring Boot. Here’s a map of each bit of the message with the part of the grok that extracts it;

<%{NUMBER:priority}>                <30>
%{SYSLOGTIMESTAMP:syslogtimestamp}  Sep 11 10:18:58
%{GREEDYDATA:container}             my_container_name
[%{POSINT:containerprocessid}\]     [1234]
((?<springtimestamp>%{YEAR}[\-\/]%{MONTHNUM2}[\-\/]%{MONTHDAY}\s%{TIME}))?         2017-09-11 14:18:58.328
((\[)?%{LOGLEVEL:loglevel}(\])?)?   WARN
(%{POSINT:processid})?              14
(\[\s*(?<thread>[A-Za-z0-9\-]*)\])? [     thread_name]
((?<loggingfunction>[A-Za-z0-9\.\$\[\]\/]*)\s*\:)?
                               a.b.c.d.CallingFunction
%{GREEDYDATA:logmessage}            Oops, something went a little strange, but nothing to worry about.

The blue text are the names of the fields that we’re extracting. The red text are built-in grok patterns that we’re using; you can find them all here. Everything else is hardcore grok!

Everything after containerprocessid is wrapped inside “()?”, which indicates that everything inside the brackets is optional. This is because, from container to container, message to message, the various components of the Spring Boot log message weren’t always present, and so I need to tell the grok that.

Like it or not, you’re going to have to figure this bit out for yourself. It involves trial and error, and learning each aspect of the regex / grok syntax as you go. When I started this process, I had used regexes a bit over the years, but was by no means a guru. Once you understand the syntax, it’s quite quick to get something powerful working, plus it looks super hard and you can impress all your coworkers who don’t grok 🙂

My advice is; start small, and figure out how to extract each part of your log message one bit at a time. The good news is that there are lots of online grok checkers you can use. Kibana has one built in, and this one was really useful. A lists of grok patterns is also available here.

Configuring Logstash output

Finally, you need to configure Logstash to send the data to Elastic. As with the input section, this is already pre-configured as part of the ELK stack your cloned at the start.

output { elasticsearch { hosts => "elasticsearch:9200" }}

This shows that Logstash is sending the logs out to elasticsearch on TCP:9200.

OK, how do I get all this working together?

It’s pretty easy. There are only 3 steps to it.

1. Update the logstash.conf with your filter grok

Your final logstash.conf file should look something like;

input {
    tcp { 
        port => 5000
    }
}

filter {
    grok {
        match => {"message" => "<<your grok here>>"}
    }
}

output { 
    elasticsearch { 
        hosts => "elasticsearch:9200" 
    }
}

You just need to copy your logstash file over the top of the existing one in /docker-elk/logstash/pipeline. You can copy the original aside if you want, but move it out of that folder as logstash can get confused if it finds two possible config files.

If you need more help on the format of the logstash.conf file, this page is useful.

2. Restart logstash to pick up the changes to the config file

Restarting the logstash container is simple. Make sure you’re in the /docker-elk folder, then simply;

docker-compose restart logstash

Because it’s easy to make a formatting / syntax error in the file, or especially the grok, make sure you check that your logstash container is running, and that it isn’t complaining about syntax errors. To check whether the container is running;

docker ps | grep logstash

If you logstash container is there, that’s good. Then, to check the logstash log to see if any errors are visible;

docker logs -f dockerelk_logstash_1

At the end of the log, you see a message showing the logstash started successfully. If it wasn’t running when you ran “docker ps”, there’s probably an error in the log showing where the syntax error is.

3. Restart your application containers with the ELK override file

Finally, to restart your application containers and get them to log to logstash, navigate to where you application docker-compose.yml and your elk-logging.yml files live, and;

docker-compose stop

docker-compose -f docker-compose.yml -f elk-logging.yml up -d

This tells docker to start all the containers  detailed in the docker-compose.yml, with the additional logging parameters detailed in elk-logging.yml, creating them if necessary, and doing it in detached mode (so you can disconnect and the containers will keep running).

Configuring Kibana

We’re into the final stretch. The last piece of configuration we need to do is tell Kibana the name of the Elastic index that we want to search.

Again, the pre-configuration sorts most of this for you. All you need to do is navigate to http://<your server>:5601. Kibana will load and show a page that asks you select an Index pattern and a time field.

Now, between Logstash and Elastic (full disclosure: I couldn’t figure out which was responsible for this), a new Elastic index is created every day whose names follow the convention “logstash-dd.mm.yyyy”.

The Kibana page will be pre-populated with an Index pattern of “logstash-*”, which means it will pick up every logstash-related index in Elastic.

It will also be populated with a default time field. You can override this with one of the timestamps your grok should have extracted, but you’re as well keeping the default.

All you then do is accept these settings. Kibana will then connect to Elastic and return a list of fields stored in the indexes that match the “logstash-*” pattern, which should be all the fields your grok extracted (the blue text from earlier on).

Can I look at my data now?

Yes! Go to Kibana’s Discover page and your data should be there. You can explore to your hearts content and start messing around with filters and visualisations to extract all the hidden value that, without ELK, you would find very difficult to extract.

Enjoy!


If you find any errors in my instructions, or any part of the above is unclear, or you want / need more information, comment below and I promise I will update it.

Helpful pages

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

https://discuss.elastic.co/t/grok-multiple-match-logstash/27870

https://github.com/hpcugent/logstash-patterns/blob/master/files/grok-patterns

http://knes1.github.io/blog/2015/2015-08-16-manage-spring-boot-logs-with-elasticsearch-kibana-and-logstash.html

The Sucky Answer Heuristic

We all know that there’s no such thing as a stupid question, but there are certainly stupid answers. All this recent talk of heuristics caused my brain to come up with what I’m calling The Sucky Answer Heuristic.

The Sucky Answer Heuristic is a specific instance of a more general emotional heuristic; if you feel an emotion, something is causing you to feel that emotion. If you’re testing something and have a response then – assuming you’re emotionally stable at the time – that response was likely triggered by something about the thing you are testing.

In this case, if you ask a question and the answer causes you to exclaim some version of “That sucks!”, then it’s worth examining further, and you should keep asking questions until the answers stop sucking, or they only suck. I’ll explain more about when to stop asking questions later.

What’s the point of focusing on Sucky Answers?

A sucky answer to a question is likely an indication of a lack of knowledge. If you’re asking a question of some oracle – which, as a tester, is quite likely – then a sucky response means your oracle isn’t a good oracle, or an oracle at all.

In either case, that’s a risk, and it’s our place to highlight all risks that threaten the value the product is meant to deliver.

Over time, as the product scope and architecture stabilises, the answers to the questions should improve as your oracles have a much better idea of what they’re producing. If the answers are getting worse, you’ve got problems.

In the worst case, really sucky answers could mean that you should stop asking questions, or perhaps even look for another role.

What does a sucky answer feel like?

How do you know what a sucky answer feels like? Well, it feels sucky. You get the answer and are left feeling unsatisfied, or more worried, or frustrated, or any number of negative emotions. It’s a subconscious, subjective judgement, but that’s fine.

Quality in all things is subjective, so the reaction to a poor quality response will also be subjective. However, here are some possible indicators of Sucky Answers, in ascending order of suckiness;

  • An answer that contradicts what you already “know” (this is “good-sucky”;  it suggests some clarification is required.)
  • An answer that takes a while to arrive (though there could be reasons other than the suckiness of the answer for the delay)
  • An answer that answers the question very vaguely (either in delivery, or content)
  • An answer that doesn’t answer the question, or answers a different question
  • An answer never even appears
  • A response that doesn’t acknowledge the validity of the question; “That’s not a sensible question.”
  • A response that doesn’t acknowledge the validity of the questioner; “Why are you asking that question?”

“The answer was sucky because it wasn’t the answer I wanted!”

If you get an answer that wasn’t what you wanted, it’s not necessarily sucky. If a decision is made that you don’t agree with, but the reasoning is explained and is sound and logical, then that’s not sucky. Those sorts of answers you have to accept.

Continuing to ask questions because you don’t like that answer will, in this situation, not improve quality and will just label you as a pain in the ass who always wants things their way.

OK, so how do I work this thing?

You already know how to work this thing. You do this already. It’s not a new idea, I’ve just given it a stupid name. It probably already has a sensible name, of which, in my ignorance, I am unaware.

That said, when I was thinking about this, a picture was appearing in my brain, so I drew it. It has some interesting bits on it that are worth explaining, that might help you decide how to use this feeling of suckiness more effectively.

Broadly, this is a graph of Answer Quality against Time. The Time axis can be a duration of your choosing. I chose the length of a release because, as I covered above, I expect answer quality to gradually increase over the course of a release until all questions are satisfactorily answered before release.

Each time a question is posed and an answer received, we rate the quality of the response, based on the criteria I listed above. The wobbly line is a rolling average of all answer quality scores. A poor answer drags the average down, a good one improves it.

Product Problems

Let’s focus above the X axis first. In this region, the answers are coming, but are delayed, or vague, or confused. I term this the Product Problem zone, because sucky answers point to likely issues with the product. It’s where people don’t yet know everything about the product – which is a risk – so their answers are a bit sucky.

Over time, as people learn more about the product and the scope is clarified, answers should suck less and the answer score average should improve.

When some major aspect of the release changes – the scope changes, a new platform is introduced, a feature is dropped / refactored – the level of uncertainty increases, answers get suckier, and the average falls.

Hopefully, at some point in the release, the answers you get will be readily available and pretty non-sucky. This is your personal “good enough” level. You want your answer to continue to be at least this good to feel confident about how things are going. Not that you should be asked if you feel the product is “good enough” to be released but, if you were, this would be one gut-check you could use.

If the answers are “good enough”, can I stop testing? As long as the product has stopped changing, you could. But as long as things are changing, and new things to be known are being created, you need to keep asking questions and sense-checking the answers. If you declare victory too early and take your eye off the ball, you could not ask a question that would have uncovered a major issue. So, regardless of how good the answers are, keep testing until you have to stop.

If you run out of questions, that’s a different problem. If you’ve run out of testing time, it’s probably not a problem. However, if you do have time left, you need to figure out how to ask more questions.

All this is well and good, and gives you a sense of how things are going, but’s that only useful if you communicate your findings to the right people. You are going to have to translate “My answer suckiness rating is 4” into something that those people understand. Risks are good. Managers understand risks.

Organisational Problems

Let’s now dip below the X axis. Down here are more fundamental problems. Down here your questions aren’t answered. The questions themselves are questioned. You are questioned for asking questions.

Down here, you are no longer able to add value to the product because elements of your organisation are preventing you from doing so. Your place as a valuable member of the organisation is in question.

Any questions you ask are seen as a negative and are interpreted as being destructive. Asking more only makes things worse.

You need to flag these sorts of responses up ASAP to anyone up to and including HR. If it’s only certain people / oracles who give these responses, you can possibly work around it, but if the problem is more systemic, then it’s probably time to look for a new company.

Summary

Pay attention to the responses you get when you ask questions. If you don’t like them, or how you got them, ask yourself why. Ask more questions, and flag up any risks that you identify.

Things will always suck a bit, but if you call attention to the suckiness you can use it as a way to help the team focus on areas of risk.

 

 

 

AACID, a Test Mnemonic

I recently attended the local incarnation of the Software Testing Clinic (all glory to Dan, Mark, Del and Tracey) where the topic was Heuristics and Oracles. To my shame, a vast array of test-related heuristics and heuristic-related mnemonics were presented that I’d never heard of.

At home, chastened, I did some more digging and uncovered even more. One thing that struck me was how, while there were a few pages that listed the testing mnemonics, and clarified the words that each mnemonic should help you remember, I didn’t find much explanation about what the words should mean to you, and how they should help you do better testing.

[Kinda like the International Phonetic Alphabet; you’ll see IPA glyphs in pronunciation help when you look up words. The obvious problem is that if you don’t know which sounds correspond to things like “ʊ” or “ə”, then “haɪ poʊ kənˈdrɪ ə” is next to useless in explaining how to pronounce “hypochondria”. But I digress…]

[Heurist|Mnemonic].*

So this led me to throw together the [Heurist-o-matic | Heuristicator | Heuristonomicon | Mnemonicomatic] (haven’t decided yet, but I think ‘Mnemonicomatic’ is winning);

  1. to bring together all the test mnemonics on a pretty webpage,
  2. to make it easy for people stuck for ideas to have some thrown at them without any effort,
  3. Most Importantly,  to expand upon what they actually mean.

Coding stuff aside – which didn’t really take all that long; I reused some old stuff – the bit that took longest was collating the information, trying to determine who the author is, but mostly what each bit of the mnemonic was actually trying to convey.

And as I was doing this, and reading the names of the authors, I felt that I should come up with something to add to my list, so I could be cool like them 🙂 And then I remembered; I already had!

Apologies for the 80s flashbacks, buuut……..AACID!

Some time ago I was trying to summarise my / our approach to testing, and to distill it down so I could communicate it to others. As a bit of a logophile – someone who loves words – I wanted to use similar word forms – in this case all ending in -ate – as another mechanism to aid recollection, in addition to the mnemonic.

The original mnemonic was ACIAD, because that’s broadly the order in which the things that each letter represents occur, but who’s going to remember ACIAD? So I messed it about, and boom. AACID. In reality, you will likely do aspects of each of these notional “stages” in different orders, or even at the same time, so it’s not too much of a cheat.

I wasn’t quite sure how to categorise this mnemonic, which might not be a big deal, but all the other mnemonics have one and I don’t want mine to picked on. So I think these are general testing principles to act as your touchstones, your center, the principles to which you can return if things get frustrating and confused.

1. Appreciate

The Appreciate stage is about learning as much as you can and building up your understanding of everything related to the project or product. Find and consume as many sources of any kind of information you can.

2. Communicate

The Communicate stage is about building up your support and information network. Talk to other roles and build up mutually beneficial relationships. If you uncover conflicting information, check your understanding and ask questions. Be constructive and support others.

3. Investigate

The Investigate stage is the exploratory, learning, testing phase where you are examining the product. Go slowly and pay attention. Take pictures. If you feel something, find out why. Look everywhere. Trust your instincts. Keep your mouth shut and your eyes open.

4. Assimilate

The Assimilate stage is about allowing what you learn to alter what and how you test. Don’t forget to apply what you have learned. Adapt your expectations, but don’t abandon your values. Add their distinctiveness to our own. Resistance is futile.

5. Deviate

The Deviate stage is about doing all the things “they” say aren’t important or will never happen. Nothing ever happened until the first time. Mess about and get it “wrong”. Try different stuff, weird stuff. Find the edge and go beyond it. Get distracted and follow your nose.

There will doubtless be other guiding testing principles which I’d be happy to add to the list.

So remember; when in doubt, AACID.

The Sergeants of Software Development

It’s 2017, and guess what?  It seems we’re still trying to discover our identity. Still asking questions about what it means to be a tester.

It’s perhaps worth offering one possible analogy that I’ve given to my testers once or twice that, while it doesn’t match perfectly in all respects, does resonate for me.

That is, if we were to imagine a software development organisation as a military unit, testers are the sergeants. Before I break down why I think it’s a useful parallel, I’ll provide a brief explanation of what a sergeant is, and does (which, being the military, is pretty well defined).

“Sergeants typically are second in command of a troop or platoon of up to 35 soldiers, with the important responsibility for advising and assisting junior officers.

After a few years as a Sergeant promotion to either Staff or Colour Sergeant may follow. This is a senior role combining man and resource management of around 120 soldiers, or even command of a troop or platoon.” – Ranks, British Army

To give some flavour for the worth the military places on its sergeants, I Googled for some quotes;

“…the backbone of the Army is the Non-commissioned Man!” – Rudyard Kipling, “The ‘Eathen“.

“The sergeant is the Army.” – Dwight D. Eisenhower

“If I had another life, that’s what I’d be – a regimental sergeant major or a similar rank. That’s where the spirit of the armed forces is.” – Bob Ainsworth

“Battles are sometimes won by generals; wars are nearly always won by sergeants and privates.” – F. E. Adcock

“Any officer can get by on his sergeants. To be a sergeant you have to know your stuff. I’d rather be an outstanding sergeant than just another officer.” – Daniel Daly

“Most armies are in fact run by their sergeants – the officers are there just to give things a bit of tone and prevent warfare becoming a mere lower-class brawl.” – Terry Pratchett

Of course it’s easy to find some quotes that fit the bill, and easier still to assert that one’s place in things is to be the glue that holds everything together and makes it work. However, in the interests of helping explain what testing is, I will explain, expand upon, and (probably, and inevitably) ditch the metaphor.

 

Chain of command

Let’s address the place of a sergeant in the chain of command. Granted, a sergeant is subordinate to the greenest first lieutenant fresh from officer training at Sandhurst, West Point, Frunze and the like. During the training, however, the officer cadet will have been under the guidance of several sergeants and sergeant majors, a dynamic that continues when the officer graduates and is given a command. The place of the sergeant is to assist and guide the young officer, using their far superior experience.

In software development teams, in my experience at least, good project managers tend to lean pretty heavily on their test team leads for organisational and management assistance. This is typically due to the type of knowledge that the tester typically has at their command, which I’ll go into below.

When I raised this analogy on twitter, someone said “I’m not a particularly good order-taker”.  As testers are an extension of the senses of our local stakeholders, it is natural that our relationships with them are more symbiotic than rigidly hierarchical. As I imagine is the case in any structured environment, high effectiveness stems more from mutual support and respect than from slavish devotion to the chain of command.

Also, much as I imagine a sergeant has scope to bring their experience to bear in their duties, so to does (should!) a tester. The chain of the command should support and encourage, not explicitly direct or restrict. As testers, we have a mission to complete, but we decide which tactics to best employ for the situation.

Systemic knowledge

The basis on which a manager comes to appreciate the support of testers is rooted in the nature of their knowledge.

A developer’s knowledge is typically narrow but very deep. They understand completely how a particular feature is put together, but might not know much about adjacent features, or other areas of the codebase. A tester, on the other hand, has pretty broad knowledge which, while not as in depth as a developers, is still pretty comprehensive, at least from a logical perspective.

This systemic view and understanding which the act of testing gives to the testers makes acting as deputy project manager a relatively small jump for a tester. My test leads have more often than not acted as deputy team leads at one point or another.

Another area of potential cross-over of knowledge is in the area of test strategy / planning. Depending on your organisation, the production of a test plan might be the first time someone asks questions like “What languages are we supporting?” “How many concurrent users are we expecting?” “What are we releasing?”.

You can actually get pretty far through the development lifecycle without ever having to answer these questions. If your focus is to deliver functionality above all things, it might never occur to people to have answers for these questions.

Testers can be the drivers for answers because test planning is the first stage where those questions have to be answered. Opinions are divided about test plans, because “no-one reads them”, and so they are waste. But, to quote Eisenhower again, “Plans are useless, but planning is indispensable” (this quote adorns the front of my test plans to this day). The artefact might be waste, but the process of filling in the blanks most certainly is not, especially if the review involves all the relevant local stakeholders.

Being the agent for filling in these blanks, for asking the broad, important questions upon whose answers can hinge the success of a release, makes the testers valuable support – and impetus for change – to management.

“Yes, Drill Sergeant!”

“But”, I hear you cry, “I’ve seen movies and the sergeant is always a shouty asshole. In Full Metal Jacket the Drill Sergeant gets shot!” True, but the role of Drill Sergeants is different. Their job is about breaking recruits down before building them back, minimising individuality and prioritising uniformity.

In a business environment, while mentoring and training will likely be part of a tester’s role, it shouldn’t involve any breaking down of the mentee. Certainly for a tester, that individuality is what makes them valuable. The ability to apply their individual skills and experience is a big chunk of why they find bugs that others have not.

One who serves

A sergeant is, in most in the armed forces but not all, the senior non-commissioned officer (NCO) rank. The word has its origin is the Latin serviens, “one who serves”, through the French term “sergent”.

There is baggage attached to the word “servant”, little of it positive. But if we check our ego at the door for a second, a servant is one who ensures that the needs of others are met; surely a noble calling. In software development, a current answer to questions about the place of the tester in the team is that we should strive to add value where and how we can. What is that if not to serve? That is literally the embodiment of quality; adding value to those who matter.

Rather than being a mere subordinate, meeting the whims of others, someone who chooses to put the needs of others ahead of their own has all they need to be a good, even great, leader. They need only to decide to lead. Those they have helped will often willingly follow.

The servant-leader is one whose desire to help others drives them to seek leadership positions purely to enhance their breadth and power to help others. This, I imagine, is the pure ideal that drives some politicians to seek high office, only to be brought low by human self-interest, and the corrupting effects of absolute power.

Lest we forget, we serve the needs of our customers by providing tools and services so they can do what they want to do. We should perhaps not be repulsed by the word service. We are not servile. The relationship is not servitude. Helping others, at both the company-customer and team levels, is what we should be about. Let’s embrace that.

The Sergeant

So, if you’re still looking for handle for your tester identity, I think you could do worse than a sergeant. The role is about adding value and supporting others, and creatively determining how to meet your mission directives. It also has the scope to develop as much of leadership aspect as you want.

Writing this has reminded me that, despite many years of trying to figure out what my passion was, I had at one point decided that my career goal was broader and simpler that what I had been looking for; that is: to help others. To seek leadership in order to better serve.  Lead to serve.

Automation will kill coding before it kills testing

I twitch-posted this reply to a Reddit /r/QualityAssurance thread because OP was told that “everyone says” that software testing is “a temporary field”, that “automation is going to kill it in another 3-4 years” and “If you go in this field you will be unemployed after a few years and it would be very difficult for you to switch jobs”.

When “everyone says” something like that about testing, I get defensive, I admit. Maybe it’s because that, even after all this time, “everyone” still seems to know more than me. So, let’s actually see whether I can back up my quickfire reply with some cogent argument.

Firstly, I’m not worried about the “everyone says…” part. We should know by now that “everyone says” is a fallacy. Argumnetum ad populum; lots of people think it, ergo it is true. Use of this term, and others like it, should immediately set off the bullshit detectors of anyone who spends any time whatsoever reading stuff on the internet.

Now to “Automation is going to kill it in another 3-4 years”, firstly to the 3-4 year time frame. Four years is not that far away. How many of you in software development land have heard even inklings of using machine learning or AI anywhere near what you do on a daily basis? I’m willing to bet much of your time is still spent grumbling about how you’re mis-using Jira, or abusing Gerrit, or any number of other procedural wastefulness.

That’s what most modern software development is; smart people trying to do something useful while wading through procedural waste. We spend so much time looking to see where we’re putting our feet, we have very little time left to look at the killer robots on the horizon.

To the second part; automation is going to kill testing. Within a couple of generations, automation is going to impact pretty much everyone’s jobs, perhaps even render working itself obsolete (if we’re really lucky). I agree that the machines are out there; they are coming to take our jobs, but I think it’s more like 5-10, probably even 15 years before it makes much of an impact.

Before I get onto which will die first, coding or testing, let’s just quickly deal with the final point; “If you go in this field you will be unemployed after a few years and it would be very difficult for you to switch jobs.

“Unemployed after a few years”? I don’t think so. Sure, some companies have dispensed with their dedicated testing departments in the belief that, with DevOps, they can respond rapidly to customer issues and deploy fixes. I can see how that approach can work for certain business models, but certainly not for all. Would British Airways want to have to phone Rolls-Royce during a London – New York flight to push a fix because the engines timeout after 20 minutes? Probably not.

Even if the nature of software development has changed, there will still be roles for humans until the point at which AI can do everything. Arguably, testers are better positioned to move into different roles than anyone. We have good broad technical and business knowledge of the products and the users. We have good communications and analytical skills. You can’t tell me I can’t find a job with that bag full of reusable skills.

Now to the main point. Will automation kill coding before it kills testing? Firstly, I’m of the opinion that it will kill them both, and every other human endeavour, at some point. In a previous post, I posited that eventually AIs will develop to the point that humans will not have to work, and that all our needs will be met by automated systems. That development will not, of course, be without its challenges, but that is the general direction in which I believe we are headed.

But who dies first? I’m a tester, and not one who is a “failed developer” or who did any sort of computer-related degree (if I’m a failed anything, it’s a failed rocket scientist), so you can give my opinion as much credence as you like.

What I see in modern software development feels like plumbing. I apologise slightly for the inference; testers are often thought of as manual unskilled workers, and I don’t really want to disparage developers in the same way. Or plumbers for that matter.

Never the less, many applications consist of web applications built using one JavaScript library or another built on top of a web of APIs, some built by the team but more often not, all sitting on top of a stack of Iaas / SaaS, virtualised environments provided by someone else, all supported by 3rd party libraries.

The work – and, yes, the skill – is in sticking all these things together into a functional whole. Sure, there are bespoke elements here and there to stick together two components never previously stuck together, but it all still feels a bit…Lego.

If you were to supply a shelf full of documented, reusable components to an AI and ask it to make you an application….well, that doesn’t seem like too much to ask. Does the fact that the system is being made by machines mean that there are no bugs? Will AI make mistakes? Will it notice if it does? Or will all this happen and be rectified so quickly that it effectively never happened at all?

I think production errors – bugs, build problems, deployment mis-configurations – will become a thing of the past, or will be rectified so quickly we won’t even notice. “Did we build the thing right?” will no longer be a question worth asking.

“Did we build the right thing?” will become the only game in town. While humans remain a stakeholder in the production of software, even if only as an end-user, giving the machines feedback on whether they built the right thing will always be a thing (at least as long as there are humans…dun dun dun!)

Testers, with their broad, systemic, technical, and business knowledge, allied to their ability to communicate in both technical and business terms, are ideally placed to act – as we already do – as a bridge between the human and machine worlds, to continue to help translate the needs of one into the language of the other.

As AI advances, and the dynamic – the balance of power, perhaps?  -between the two worlds changes, someone will need to be there to translate. Who better than us?

 

How AIs might change Software Development – and Humanity – forever.

There is plenty of discussion in the wider world about the rise of the thinking machines, and where will humans fit in a world run by AIs. In software development, it’s tempting to think that we’re more isolated than most against this rising tide. That may indeed be the case, but it’s certainly no cause for complacency.

So I set out, as I tend to do, to let my brain empty itself onto the page, and endeavoured to follow my thoughts as far as my knowledge and logic would allow. Also typically, I haven’t specifically researched the topic, mainly out of laziness / habit, but also as an exercise to see what conclusions I could reach shorn of any overt, conscious bias.

I will be jumping between two main strands of thought; how AIs might develop, and how that might affect software development as a discipline. That’ll get confusing, but I’ll try to indicate when I switch.

I also didn’t intend to get into the whole debate – about whether AIs will be subservient, benevolent, or genocidal. I was going to assume that they would stick, at least outwardly, to the tasks we assign them, if only to lull us into a false sense of security. However, somewhere along the way, I did actually reach a conclusion about whether we should fear AIs. You’ll have to read on to find out what that conclusion was.

But let’s start at a beginning.

Thinking machines, however you wish to term them, are not really a paradigm shift in the developmental arc of humanity. We’ve been replacing humans with machinery for centuries, as part of our boundless need to grow. They are just the next stage in the industrialisation process.

We’re at the point now machines can do the majority of the manual labour traditionally done by humans, and I don’t just mean picking fruit or vacuuming the house. Repetitive information-based tasks – data-entry, simple processing, etc – are now within the machines’ grasp. The next stage is to start picking up the job of thinking, and this is where we’ve started to get unsettled.

We’re on our way down into the uncanny valley of thinking machinery, and the point at which this progress will yield things potentially superior to us suddenly doesn’t seem that far away. We worry that we will be the architects of our own demise. It’s bad enough that they might take our jobs. At very worst, we worry that they might turn Terminator and take our lives.

On the more hopeful side, perhaps the demise we’re architecting is of ourselves as a species enslaved by its need to dedicate a huge proportion of its time to basic subsistence. Even after all these years, people work predominantly to buy food and shelter. Wouldn’t we rather eschew these rather primitive drives, and let the machines handle it? What could our species achieve if everyone didn’t have to worry about their next meal, or paying the rent? If we could focus on our “wants” and not our “needs”?

The point at which machines can deliver all our needs is a huge existential moment. Within no more than a couple of generations, the arc of a typical human life will alter enormously. What does one do with a life unimpaired by working to live? AIs will inevitably hold up a mirror to our species, and we will each have to ask ourselves “Who am I?”

But aren’t we getting WAY ahead of ourselves here? How does any of that apply to what software development might be like in the future? Well, it doesn’t directly, but it’s the socio-economic landscape in which future human activity is likely to take place, so we need to at least bear it in mind while we think about our little corner of human endeavour.

So let’s think about how we might characterise where we are on a software development scale from the two possible extremes; human-only software development, to machine-only software development. I’m not a computer science historian by any means, but it’s possible that human-only software development was never a thing, or was only human-only very briefly.

At the other extreme, it’s also possible that machine-only software development will never be a thing either. Even in a post-scarcity world where machines take care of most things, humans will still need to interact with systems, even if only to ask for Tea, Earl Grey, Hot. I’d hope the machines would at least consider our opinions on those interfaces.

Either way, it’s not necessary at this point to define the ends of the spectrum. We’re too far from either end for that clarity to be terribly relevant. Let’s agree that we’re somewhere along that spectrum, and that the arc of progress is towards increased use of machines to automate tasks that were previously done solely by humans.

We’re getting to the point where those tasks being automated are the creative, sapient ones done by humans; product managers, developers, testers, tech authors, literally anyone who has to think up and create stuff that didn’t exist before.

Let’s look at those activities a bit more closely. How much of what we each do on a daily basis is New, with a capital “N”? I’d wager not much of what we call Research and Development is actually Research. A lot, probably even most of it, is the reproduction of concepts that we’ve done before; UIs, logging, database schemas, APIs, etc. It’s mostly the reworking of existing concepts into a slightly better or different-shaped package.

I know, we’re not making bricks, we’re not stamping out license plates. Software development is not a production line. But, if you’re honest, how different really is the plethora of getters and setters you wrote for this product from the ones you wrote for the last one?

So, if we accept that a lot of human-powered software development is plugging together third-party components, it’s not actually that cool. Even less cool is having to deal with the fallout of humans being fallible. Testing exists, in part, because people make mistakes, and the hope is that the testers don’t make the same mistakes. Machinery won’t necessarily make less mistakes, at least not initially, and might make different mistakes, but the rate of detection and fix, and the whole learning feedback loop, will be so much faster. Yes, the machine mean-time-to-error (MTTE) will be terrifyingly short, but downtime too will be so minuscule it will go by unnoticed.

Potentially the stickiest part of the transition is the move from human-centric to machine-centric processes. Our current processes are messy because of what it involves; humans using machines to tell other humans what to make other machines do. Every time we add to, or remove from, the machine world there is an inherent translation from human-readable to machine-readable, and information is lost or garbled in that translation.

When you factor all that in, I’m not sure we’d be too desperate to cling to our approach. So, rather than try and force the AIs to start from a flawed human-centric process, the best approach will probably be to give them a simple feature to produce – probably some machine-to-machine interface – and let them decide how to manage production of that feature.

Basically, we develop them like we would an intern or recent graduate; get them to cut their teeth on something simple, then mentor them through the learning process. Then, once we’re satisfied that they provide consistent good quality output, they take on larger and larger pieces of work.

As with any mentoring relationship, we will likely learn as much from the process as the machines. The most important information will be how to best shape the development of the AIs in the “right” direction. As we’ve seen with crowd-sourced teaching of proto-AIs, the quality of the guidance they are given is vital to the quality of their output, and to the development of its character and personality.

Assuming we curate these formative AIs successfully through the first however many generations, and the AIs themselves take over this process, we are likely to see pretty rapid and meritocratic iteration, as AIs evaluated to be less efficient in generating quality output are weeded out.

Perhaps unsurprisingly this process feels Darwinian in nature; survival of the fittest, but occurring at a vastly increased and accelerating rate. Will that be because evolutionary theory is itself the fittest, universally, or because it’s the process that humans have found to best provide iterative improvement and have therefore baked into the machines’ foundations? I guess we’ll have to see if / how quickly AIs develop other mechanisms for determining fitness.

Let’s go back to character and personality for a second. Will AIs have such things? Is it possible for intelligence to exist without other idiosyncrasies creeping in? Intelligence could be defined as the ability to apply prior experience and knowledge to solve new problems. In much the same way as life events shape human personalities, it’s likely that different sets of events experienced by AIs through different versions of differing feedback systems – senses – will result in varying sets of neural models and heuristics that could be termed personalities.

It’s likely that these personalities will render certain AIs fitter, in certain situations, than others. This will lead to AI specialisms, with families of AIs developing to better deal with specific situations and tasks.

If all this sounds pretty familiar, it’s because it is; it’s pretty much how human societies evolved. It will be interesting to see if the tribal tendencies that so hamper humanity occur in the AIs, or will the lack of resource competition mean they will sidestep that messy stage of their evolution.

Selflessness might have to be one of the baked-in founding principles of AIs. When fitter AIs are produced, those that are superseded should be deleted, lest they consume resources better spent by more efficient descendants.

If it was humans we’re talking about, we’re well into “crimes against humanity” territory. We’re talking about ethnic cleansing, genocide. In effectively recreating ourselves in silicon, and playing out our own evolution in tens of years instead of tens of thousands, we don’t answer or even postpone having to answer these thorny moral questions.

AIs will be another set of lifeforms on the planet – and will likely spread at least to the rest of the solar system – that will face the same questions. In the same way that humanity is starting to class certain animals as non-human people, it’s likely that AIs – as the pre-eminent intelligences – will categorise us similarly.

That’s probably why we can fear AIs less than we fear other humans. It’s arguable that the only reason that humans worry about being exterminated by the machines is because that’s what we would do, and have done, many, many times. As beings of pure intellect without the animal hindbrain to cloud the process, AIs would likely consider the eradication of an intelligent species like humans unthinkable. It literally would not occur to them.

So how will AIs manage the obsolescence of earlier generations of AIs, surpassed by their progeny? Sci-fi writers have postulated constructs to which all consciousnesses – human or artificial – are sent when the individual is no longer viable, there to join with the collective consciousnesses of everyone and everything that went before. This construct acts both as the genetic memory of both species and as the arbiter for significant moral and developmental decisions. Silicon Heaven; it’s where all the calculators go.

Early in the transition from human-powered to machine-powered, humans will still be necessary, and in new capacities.

The new mistakes that might be peculiar to a machine-driven process might have to be initially detected by humans. Anything, human or AI, cannot rectify a mistake it cannot identify took place; if developers knew when they were writing a bug, there would be significantly fewer bugs.

An excuse to reference Douglas Adams, and the case of the spaceship that couldn’t detect that it had been hit by a meteorite because the piece of equipment that detected if the ship had been hit by a meteorite had been hit by a meteorite. The ship eventually inferred this by observing that all the bots it sent to investigate fell out of the hole.

Testers build up a mental working models of the systems they test. It’s one of the most powerful heuristics we can bring to bear. It’s what underpins the H of HICCUPPS. AIs will probably understand themselves and their structure completely, and so will be able to quickly locate and identify any failures (unless it’s in the fault identification routines). It’s probably unlikely, therefore, that we’re going to have to be that detector even from the beginning.

Whether we’d actually be able to distinguish mistakes from the intentional and probably unintelligible ways that AIs operate and communicate is questionable anyway, especially since they are likely to be changing rapidly. Even the things we did manage to figure out would be rendered useless because, when we looked at it again the following day, we’d be greeted by a completely new iterated version.

Next, someone has to train the AIs in all sorts of topics. Most apposite for software development, because it’s important and difficult to define, is what ‘Quality’ means. How does one explain a subjective concept to a machine? Will it understand? Can an AI make subjective judgements as well as a human? Well, since those judgements are subjective, who’s to say that an AIs subjective judgement is any better / worse than a human’s?

Perhaps, in the case of AIs, their subjectivity is merely an aggregation of their specific sets of objective measures. The models that each AI is generating and refining is a result of the data they are analysing, which is unlikely to be exactly the same as any other AI. Therefore, each decision they make is objective as far as their models go, but may differ from other AIs decisions. Individually objective, but collectively subjective.

It comes down to whose opinion matters. In the event that humans still need to use IT in some fashion, from manually all the way to direct neural interaction, you could argue that a human’s opinion is more valid. When I inject my iPhone 27 into my ear canal, and it begins to assimilate itself into my cerebral cortex, I want to know that it doesn’t feel like my brain is being liquified. I don’t think an AI can tell me that, though I’m not queuing up to be the first guy to test it either.

Most software being created will be made by machines to allow machines to talk to other machines. In those cases, the machines can make that determination, based probably on objective criteria, which – as I say above – might aggregate to that AIs subjective measure of good enough. Given how rapidly they should be able to change things, an AI’s “good enough” is going to be as near flawless as makes no difference. Not that we’ll notice, of course.

Where and while humans are still involved, there will be a lengthy period of training where we teach the AIs our collective subjective definition of quality, to get it to the point where it’s “good enough”, or at least as good as our “good enough”. That could actually be an interesting job, but in reality might boil down to you to being shown two pictures and being asked to pick your favourite, which sounds pretty dull.

The post-scarcity endgame feels like it will be idyllic, but getting there will be painful. Social change is not something that our not-that-evolved-really species transition does well, or quickly.

Post-scarcity means effectively unlimited supply. That completely undermines economics and capitalism as we understand it. It’s perhaps not too much of a stretch to imagine that there will be resistance from those, to quote Monty Python, “with a vested interest in the status quo”. Those who control the resources; the oligarchs, the malignant capitalists.

Given how the quality of life of many billions of people would skyrocket in a few years, the sheer inertia of the change should be unstoppable. It won’t all be smooth sailing, I’m sure. There might have to be a bit of a revolution to rid ourselves from the shackles of those who would seek to control effectively unlimited resources.

There will probably also be a bit of anti-industrial revolution first, from those whose jobs / way of life is under threat, who don’t trust that the post-industrial society is ready to receive them, or that post-industrial life is for them. Before AIs “take our jobs”, they need to be able to provide for the needs of all those people, so that they can continue to live their lives, better than before if possible.

Key to a smooth transition will be improved quality of life for people. Humans are easily pleased; a warm bed, good food, footy on the telly and a few beers with our mates. If people can still get that without having to go to work, you won’t have to sell them the idea, they’ll be biting your hand off, and not even the likes of Putin would be able to stop it. The biggest hurdle might just be to convince people that all people need to do is simply reach out and take it. Revolutions are never that far away, it just takes enough people brave enough – or with nothing to lose – to take a stand.

Will our selfish, tribal, lizard-brain tendencies continue to hobble us? Self-preservation is a powerful force. How much more evolution – biological or social – is required before we accept this new normal? If machines tirelessly meets the fundamental needs that our lizard brain worries about, does this free us to make more rational decisions?

Will we be more munificent if our personal needs are met? Those who are “rich beyond the dreams of avarice” are often philanthropic. What do you give to the man who has everything? Nothing, because he’ll likely want to give it to you. Will that selfishness diminish as society more consistently and bountifully preserves us? What will that do our sense of self? If we identify as “the provider”, and that responsibility is rendered obsolete, again; who are we?

Let’s try and summarise all that quickly.

As AIs become more widespread and more effective, the types and amount of work humans have to do will begin to dwindle. A few bumps aside, this will be the largest wholesale improvement in quality of life for everyone on the planet.

Benevolent AIs – because benevolent they will be – will be the saving and making of humanity. They will allow us to put aside our petty squabbling for power, and usher in a golden age. With the freedom to spend our days as we desire, rather than chained to the means of production, the next age of humanity will be ushered in. As Information followed Industrial, so will Intelligence follow Information, and Imagination follow Intelligence. And imagination will be the only limit to what our species can become.

 

We need to talk about testing…

Rosie Sherry posted a scary statement / question / challenge on The Ministry of Testing site: “The Abysmal State of Testing in 2016 and What Can We Do About It“. I started responding on the forum, but my musings quickly became too convoluted – I thought – to be of much value until I could give them some more thought and structure. Whether I’ve been successful or not, well, you be the judge.

Before we can answer the second part of the challenge, we need to understand and agree what the problems are. I think there’s one underlying issue that presents itself in different ways, and that is;

We don’t understand what testing is.

There’s a statement that needs explained.

Who is “we”? It’s everyone involved to directly or indirectly with the business of learning about what software actually does; what we might call “testing”. “We” encompasses everyone from career testers, who one would hope know the most about testing, to non-technical managers, who are business people and not technically savvy.

Rosie also posted a comment:

Does stuff like this have anything to do with the abysmal state of things?

If you don’t want to click the link, it’s an advert from a company selling automation on the back of a questionable infographic (which I poo-pooed, and to which they responded with a “yeah, we know, so read our whitepaper”). But therein lies the problem.

The people who understand testing the least, and are probably least inclined to expend much effort in digging any deeper, will look at the infographic and be given an overly simplistic and inaccurate understanding of what is a hugely complex endeavor.

So, to answer Rosie’s question: 100% yes, this stuff absolutely feeds the problem. There are lots of companies out there selling testing services who really don’t seem to understand – or their marketing cannot convey – what testing even is, let alone deliver useful testing.

This perceived or unconscious lack of knowledge is behind my probably cryptic tweet:

Why is the state of testing abysmal, @rosiesherry ? How about Dunning-Kruger managers versus Imposter Syndrome testers?

The people who have the loudest voice right now about testing are the people who are trying to sell it to other people, and none of those people – the sellers and the prospective buyers – really seem to understand what testing is. These are the Dunning-Kruger people: they have a little knowledge about testing and have high – and unfounded – confidence in that knowledge.

Those of us who are more experienced – the career testers – are much more wise about the ways of the testing world, and are right at the bottom of the Dunning Kruger curve, at the point where Socrates defined true knowledge as “knowing you know nothing”.

We therefore have less confidence in our knowledge and – broadly, as a community – succumb to Imposter Syndrome, where we think that those loud and overconfident voices will find us out. When you see confident statements along the lines of “this is what testing is / what testers do”, there are immediately cries of “I don’t do that!” or “OMG, why don’t I do that?” from testers.

So, What Can We Do About It?

We need to have a louder voice. We need to recognise that we do know what we’re talking about, but at the same time be clear that we still don’t know what testing is. But we need to give ourselves a break about that.

Testers have been struggling for decades to try and define what testing “is”, and it doesn’t seem to fit into any one box. Perhaps we need to declare that it’s its own thing; it’s a new species in the taxonomy of human endeavor. I think it’s a fabulous mongrel of different breeds, and even species, of activity, to the point where it defies classification using “conventional” or “classical” understanding.

This of course does nothing to solve what testing is for the purposes of explaining it to others. It might, however, explain why it is so difficult for us to accurately and completely explain, and for others to grasp.

download

Because our business, like Archaeology, is the search for Fact (not The Truth!), I believe we are more open and clear about what we know and what we don’t. The problem is that this is interpreted as a lack of knowledge by the loud “a little knowledge is dangerous” people.

Find Your Voice

Despite being a tester for a long time, I’d always been reticent about writing / blogging about my experiences, partly because of Imposter Syndrome, partly being worried about people telling me I’m wrong, but mainly because I didn’t want to restate / steal the hard work of the heavy lifters who are pushing the envelope of what testing is.

However, more recently I’ve come to the conclusion that there is value in other people re-framing the thoughts of others in their own words. That a “peer-reviewed scientific paper”, crowd-sourced approach will allow us to further our collective understanding. That if each of us can, in the process, advance our understanding, we might provide the kernel of the next big idea that moves us forward. That if “standing on the shoulders of giants” was good enough for Bernard of Chartres and Isaac Newton, it’s good enough for us.

So, all you huddled masses of testers yearning to be understood; collect your thoughts, clear your throat, and speak out. We must be heard.