The Fourth Amigo

A couple of things converged in the last week that has finally prompted me to put these thoughts together.

The first was preparing for a company-internal AMA I’m doing about building / supporting the creation of a “shifted-left” embedded test organisation. The second was a sequence of tweets about the threshold of value of seemingly minor observed defects, one example being typos in the UI / documentation.

The thoughts centre around the relative position of the technical author role in a company, and how testers can and should develop a close working relationship with them where possible.


From working closely with technical authors – admittedly, some time ago now – it’s apparent that there are (at least) a few similarities between how testers and technical authors operate;

  • they are often siloed off into specialist teams, struggle to get information about the product, and have their work squeezed by slipping code and unmoving deadlines
  • when embedded into development teams, they don’t necessarily get invited to all the meetings as a matter of course, resulting in them missing out on vital information
  • they initially base their work from requirements / user stories, have to learn and react to the ways the built software might be different, and document the result
  • they need software to be deployed and usable (to test systems) in order to produce their work products
  • they communicate how the software operates
  • they ask questions when the product doesn’t operate as expected.

Work Products

A tester’s primary job is to highlight where the software does not meet the expectations its users might have, in any of the many qualities that software provides to its users.

A technical author’s primary job is to document exactly how the software needs to be used to meet the expectations its users might have, in any of the many qualities that software provides to its users.

(One of) A tester’s primary work product is a list of the ways the product doesn’t meet one or more expectations. A technical authors primary work product is a list of ways the product does meet one or more expectations, and specifically how it can meet them.

These are very similar processes resulting in not dissimilar outputs, albeit with different target audiences (bugs wholly internal, docs largely external);

  • test scripts – either (shudder) manual or automated – are effectively a documented path through some features of the software, with things to do and see along the way. They are a document of how to interact with the product from a user workflow perspective.
  • product documents (there are many types, but I’m referring to user guides here) are a map of the software with instructions how to interact with the product on each new interface. They are a document of how to interact with the product from a product hierarchy perspective.

Both roles’ primary work products are records of the software behaviour from two different perspectives and expressed in slightly different ways.

Product Knowledge

In the nearly 15 years since I started trying to push testing left in my organisation, one of the main lessons was that “left” is not the only direction in which one can or should push. In an old post on the subject, I suggested a more accurate but less catchy maxim: “Expand Outwards”.

I think we can all see why that would never catch on. However, it does remind one to not just look left for activities and people with whom to foster closer working relationships. Look “right” and work with Customer Services; I’d argue no-one in your org knows the software, and the associated woes of customers, better.

Tying with testers for the broad product knowledge silver, I’d put technical authors. They literally wrote the (usually many) books on the product. They have documented how to interact with every control in every dark and dusty corner of the product. They have taken screenshots of every dialog.

Shifting Left

Given that testers and technical authors both produce similar work products and require wide and deep product knowledge, it’s logical to shift both roles “left”, not just testers.

If you’re in a meeting (or trying to get into meeting) where product information is being shared / created, bring your technical author colleague with you. The things that make testers’ jobs easier can also make a technical authors job easier.

It can initially be difficult to make the case for inclusion, as testers can be outwardly passive in these meetings (because they are actively listening / learning). However, it will not take long before a tester’s spidey sense starts tingling and they start asking questions that makes it clear there is not yet a good answer.

Technical authors can absolutely interact in the same way; actively listening and learning about upcoming features, as well as questioning and clarifying the implementation.

“The Fourth Amigo”

The term “three amigos” has been around for some time, typically referring to close working and information-sharing relationships between product managers, developers, and testers. It’s high time that technical authors are added to those conversations.

Technical Authors are a reliable source of product information who are professional communicators; they can clearly explain the often complex concepts built into the product. If they struggle to communicate a concept, that’s a smell the team should not ignore.

If they are able to engage in parallel with development and testing, Technical Authors are another vital source of information into, and perspective on, the development process. You owe it to them, yourself, your team and your customers to utilise their knowledge and communication prowess.


Taxonomy of Testing

I recently tweeted about my desire to (help) push the boundaries of testing. I feel my personal knowledge and understanding of testing has stagnated, and I struggle to engage with the content in the testing community (though that’s mostly a reflection on me and not it).

In thinking about how one might go about pushing the boundaries of understanding testing, I thought it might be useful to understand the current state of the art. Where is the testing envelope right now? What shape is it? What are its components? Are there bits that have been neglected?

The testing envelope is a fluffy cloud. Its shape is amorphous, its edges  poorly defined, like the dotted lines on maps between countries disputing the border.

Progress might be accelerated by understanding where we are, so we can pose better questions that help us understand where we might go.

But how to document or depict the envelope? My first thought was to utilise taxonomy, which is the practice and science  of categorization or classification. The taxonomy I’m most aware of is that one used to categorize living things. There are many others, including some attempts already in the field of computing. Howeer, some difficulties already present themselves.

The taxonomy for living things, while very large and containing many, many branches, is simple in that each category – kingdom, phylum, and so on – have one parent. The evolutionary nature of life means that a species may, in time, result in two or more successor species, but a species cannot have two or more ancestor species.

Depicting such a taxonomy is relatively straightforward. You start with a small number of categories, and you just keep categorising differences until you run out.

The things produced by humans, by comparison, aren’t so neat and simple. While we certainly split activities into categories and specialisations, we also regularly combine activities and endeavours to create all new things.

Human society has reached its current state (for better or worse) by assimilating ideas and concepts from every thread of human endeavour. The resulting “fabric” is so complex and intertwined, depicting it is probably impossible, or at least wildly impractical. A decipherable model would probably be so simplistic as to be practically useless.

So, is a taxonomy of testing possible? If it was, would it be understandable? If it was, would it be valuable? These are all (good) questions.

An underlying principle of the endeavour to “push the testing envelope” should be to ask questions with an open mind, without knowing the answer, without judgement of the responses.

Will we know if the responses are pushing the envelope? We won’t know until we try. Will a taxonomy of testing have any utility whatsoever? You tell me.


BASKETS: Ep. I: Hardware Components

In this post, I’m going to list the hardware that comprises the BASKETS, with some notes on why I chose it and roughly how much it cost.

However, before I get into that, it’s worth putting up a quick diagram to explain what I’m trying to achieve. This may evolve as I learn more, or people tell me problems with this, but this is where I’m starting from.#

I haven’t mentioned this yet, but the computational hardware for the BASKETS will be Raspberry PI 2 Model Bs. When I started the project, the RP2 was the latest. If I was starting now, the Raspberry Pi 3 is now the one to get.

If I was starting today, I don’t think I’d use Raspberry Pi Nano Ws. While a lovely little machine – smaller and cheaper with built in Wifi – they’re single core and have half the RAM of the RP2/3. Since I’ll be running databases and servers, I need the power of a full size Pi.


I will have a Primary Pi fitted with a 802.11 Wireless USB dongle that will enable me to SSH into it from my home network. It will also be wired to a network switch, through which it will connect to (up to) 4 other Secondary Pis.

The intention is that the Primary Pi will act as the gateway – to allow me to connect to the BASKETS – and will act as domain controller / DNS + DHCP server, monitor, etc as required.

The Secondary Pis will either host security tools (one will be running Kali Linux) or target applications (one will be running OWASP Juice Shop or similar).

Let’s go through my bill of materials;

Computer – Raspberry Pis x5, from £32

You clearly don’t need as many as all this. For reasons I don’t recall, I decided to have 5 Raspberry Pis in my environment. I’m sure it made sense at the time.

In terms of getting cheap computers for a project such as this, it’s hard to beat the Raspberry Pi for price, power, and size.

Storage – 16 / 32Gb microSDHC Class 10 memory cards x 5, from £6

You need somewhere to store your operating systems and data. I got a mix of 16Gb and 32Gb cards Class 10. Get the fastest cards you can afford; the fastest 90Mbps cards start from £50 and are probably overkill. Class 10 (10Mbps) are fine for our purposes.

Case – 5L Really Useful Box, from £5

Unless you have an old Xbox 360 you’re willing to gut (which was my original plan for the case), this is the next best thing. It’s solid (so you can screw stuff to it, or cut it easily), won’t cut your hands to bits, and it’s meant to be opened.

Case – Transparent plastic Raspberry Pi cases x5 – from £3 each

Not strictly necessary, but makes it easier when you’re trying to  pack box-shaped things inside a larger box-shaped thing. Plus it gives you nice flat surfaces to help stick them together.

Network – TP-LINK TL-SG108 8 Port Metal Gigabit Ethernet Switch, from £20

I need a switch to handle at least 5 machines, and this fit the bill. It’s solid and well made, and not too costly, and it has room for expansion.

Network – Wireless USB Network Adaptor [Optional] – from £5

You could use one of the spare switch ports and wire the BASKETS to your home network, but I wanted it to be wireless, just because it’s one less wire and more flexible.

I only need one; to connect the Primary Pi to my home network. The Pis will be connected to each other via the TPLink switch.

The price of these things varies depending on the supported bands and frequencies. Make sure you get one that matches your home Wifi. The one pictured is 802.11n

Network – 20cm Cat5 Ethernet Cables x5 – from £2 each

Each Pi will be wired via the TPLink switch. To keep the cost – and clutter – to a minimum, I bought the shortest cables I could find, with a different colour for each Pi. Because what is life without whimsy?

Power – Anker Quick Charge 3.0 – from £30

These Anker USB charging blocks are great. I have one for everyone in the house. They’re not super cheap, but they’re small, well made, and reliable.

Power – Anker 6-pack USB charging cables, 30cm – from £10

Obviously all the Pis need power from the Anker, and if you’re lucky you can get a package deal for the Anker power block plus cables. You can pay less for USB A > micro B cables, but Anker ones are the sturdiest I’ve found.

Since these will be protected inside a sturdy plastic box and not unplugged much, you could probably save a little cash here.

Power – 2-gang 2m extension lead – from £7

I needed to power both the Anker USB power supply and the TPLink switch, but I didn’t want to have two power cables outside the box. So I bought a simple two plug extension cable to keep all that cable mess inside the box.

Icing on Cake – Raspberry Pi 7-Inch Touch Screen Display – from £60

This is very much a nice to have, but I had this spare from another incomplete project, so it made sense to re-purpose it for this. This will act as the display from the primary Pi and is intended to display status information about the environment.

That comes to about £300 (before the very optional extras)

By reducing the number of Pis, it should be possible to get this down to £200 for a smaller setup. So, the cost is non-trivial, but it’s all stuff that can be used for other things; nothing is sacrificed or used up.

In the next post, I’ll go over how I intend to package all that together into the Really Useful Box.

Building a self-contained environment for testing security (or BASKETS. Kinda)

I’ve been a tester for a while now, and I’ve done most types of testing for a little while. However, there is one really glaring hole in my skillset (among many other, slightly less glaring ones). and that is security testing.

While I’ve used code scanners (AppScan, AppSpider, Checkmarx, OWASP Dependency Checker) and automated them with Jenkins in a DevSecOps fashion, I don’t really consider that security testing. Using code scanners is testing in the same way that automated testing is testing; i.e. not.

They can find things they know to look for, but can’t recognise or highlight anything else, unless it gets in the way of what they’ve been told to do. Sure, code scanners make available the work of security researchers around the world, but they don’t do anything to educate those who use them. If all I have to do is push a button and then send the report to the devs, what have I learned? Nowt.

Some time ago, I decided I wanted to make my own security testing environment. The reason for this is mostly to learn how, but the other major driver was that setting up a security testing environment in a corporate IT infrastructure is problematic.

No matter how friendly your IT folks are – and mine are very friendly and exceedingly accommodating – they will quite sensibly baulk at letting some n00b tester install security software into their lovely corporate environment.

Security testing tools use the same techniques and look to exploit the same vulnerabilities that malicious hackers would, so performing security testing on the network will (or should) cause their network monitoring and intrusion detection systems to light up like a Christmas tree.  Sure, in the initial reconnaissance phase you’re not doing anything naughty, but which IT guy really wants that noise and hassle in their life?

My initial plan was to rip the guts out of an old Xbox 360 (who had suffered the dreaded RROD) and use that as the case, but it was a bitch to work with; lots of sharp metal edges and, unsurprisingly, hard to get into, with losing patience and layers of skin.

The project died, but not until after I had bought the initial pieces of hardware.

In the next post I’ll go into the hardware I am using, with some idea of how it all fits together into one package.


UKSTAR 2018: A Retrospective

Right, since I’m sat at home on a Wednesday all bunged up with lurgy, let’s try and get some thoughts down about UKSTAR 2018 before I blow them out of my nose.

Travelling to That London Down There

I’m not a huge fan of travelling, mainly due to not liking not knowing exactly where I’m going, and what’ll happen when I get there. While both traveling and testing are about encountering new things and making the unknowns known, travelling makes me nervous.

However, the reality usually is mundane and straightforward, and in this instance I was accompanied by my awesome colleagues Sindu Kumar and Jehane Penfold-Ward who kept me safe.

Plus London, in small doses, is pleasant enough.

First test conference

I’ve been to a couple of other conferences, but this was my first time at a testing conference. There was a great vibe around the event; lots of positive energy, so must give many thanks to the UKSTAR organisers and the committee for putting on such a great event.

It was great to be around people who talk my language and who care enough about testing to find the time and resources to attend the event.

The venue was great and the food – apart from some slightly suspect-looking duck on Day 2 – was excellent (the duck was good, don’t get me wrong, it’s just that green meat raises flags for me 😉

Social anxiety & starstruck-ness

It’s a stereotype that people in IT do so because it’s easier talking to computers than people. I’m certainly an introvert usually, but being surrounded by fellow testers, I certain felt more at home than I usually do in a room full of people.

One big takeaway for conference n00bs is; introduce yourself to people. It might feel unnatural – certainly does for me – but a room full of testers all there for the same reason you are is about as friendly and non-intimidating an environment as you’re likely to encounter.

Thankfully, I did manage to meet some people, I think in every instance because they had the guts to come and say hello to the giant with the beard. So, kudos and thanks to Dan Billing, Ali Hill, Victor Slavchev, Jit Gosai, Stuart Crocker and James Thomas (and, I’m sure, others) for introducing themselves. It was lovely to meet you all.

There were also several times where I saw a face or name I recognised from the twitter- or blogo-spheres, and I didn’t introduce myself. In these cases, your first thought might be – as mine was – “OMG, that’s Dorothy Graham (for example), who am I to just walk up to her?”

In this, I didn’t learn one of the points I made in my talk; if you only know one thing that another person doesn’t know, it’s worth talking about it. So this is a reminder to be brave and introduce yourself. You will both benefit from making that connection, now and in the future. You owe it to future you.

A little interaction to underscore the point; at one point, I think in a workshop, I told someone my twitter handle, and they exclaimed “Oh, you’re that Richard Paterson”. I’m not any sort of famous, so we’re all really in the same boat.

First talk

I’d been submitting (the same) talk ideas to a small number of conferences for a year or so, so when the email arrived from UKSTAR asking me to speak, my email to my boss read simply “Holy shit!”

All of a sudden it was real, not just a possibility. Fortunately, the topic of my talk was about the process of deciding to propose, and eventually delivering, a conference talk. Every thing that I did in preparation was material for the talk, and vice versa.

I had my initial slides put together about 9 months before my talk, and I delivered in internally to my testing colleagues in a meeting where we launched our own internal test conference. The aim was mainly to embolden them to put forward  ideas for our test conference, but it also gave them – fine testers all – a chance to find the bugs in my presentation.

Over the months leading up to UKSTAR, I endeavoured to practice my talk every week. I setup a meeting room to what I imagined the real thing would be like. I put my slides up on the screen behind me. I video’ed each run-through using Webex on my laptop. After each run through, I recorded my thoughts on how it went, and what needed improved.

It was this preparation and visualisation over several months that meant that my heart only started beating faster about 15 minutes before my talk. Despite my preparation, standing in front of a room full of people looking at you is not the same as standing in an empty meeting room, no matter how hard you visualise.

In retrospect, I feel like it wasn’t until halfway through my talk that I felt in-the-flow enough to stop looking at the ground and engage with the audience, and that helped. I made eye contact, I saw nods and smiles, and that helped me realise that I was actually doing it, and people were listening.

It was great to get lots of questions at the end, although I felt like I didn’t answer them that well; one of many things to improve upon next time. And several people were kind enough to say hello afterwards and say nice things, which was much appreciated.


My thoughts on my own journey from deciding to try to actually delivering the talk? A very worthwhile trip, with lots to learn along the way, and a great payoff at the end. It’s a journey I hope to make again.


All the speakers did a great job, and I picked up at least one pearl of wisdom from each, usually lots more than one. The standouts for me were Hristo Gergov, Viktor Slavchev, Yann Person, Dorothy Graham, Isabel Evans and Harry Girlea. A wide variety of experience and experiences in testing represented.

An observation worth making, which I also had at our internal conference, is that the most engaging speakers are those who let their own personality shine through in their presentation. I don’t want to feel like you’re just saying words, such that it could be anyone up there. I want to hear you imparting your knowledge and experience in your own words, in your own style.


Following on from his workshop with Alan Richardson, Gustav Kuhn’s keynote was a wonderful mixture of magic and psychology, reminding us that, while our brain is our first and most valuable testing tool, it’s not always paying as much attention as we’d like, and what we think we perceive may not actually be what happened.

Michael Wammes’ keynote was a great reminder that we’re doing all this for humans – ourselves, but mostly our customers. Tech is there to enrich and improve lives, not confuse and frustrate. Like the conference, it’s about connecting people.


As someone in a testing group who thinks we’re doing good testing, I’m very keen to check for any signs that I’m lying to myself, that we’re in fact fooling ourselves. Conferences – and other resources – are vital to give you a sense of what the industry is doing, to give you a benchmark for what you’re doing and, more importantly, not doing.

As a first time conference-goer and first time speaker, it was a really enjoyable experience, made so by my fellow testers. So, here’s to you all, thank you, and hope to see you soon.


What can I do about Mr Creep?

To clarify: the title question is absolutely not a “/shrugs/ what can I do?” shirking of responsibility. It is a sincere, open request for guidance. Because I’m worried.

I worry that I won’t be able to accurately put across what I want to say, and that these words will hurt instead of help.

I worry that these sorts of abuses happen every day, which they pretty obviously do.

I worry that inattentional blindness means I don’t even register or recognise them when they do.

I worry about the effect incidents like this have on the women in my life, professional and personal, all of whom I hold in the highest regard.

Most of all, I worry that it’s me.

To be clear, I’m not worried for me, but that the problem is me. I am worried that I might be making the lives of the women I work with even the tiniest bit worse.

I consider myself a feminist. I worry a lot that I’m a crappy feminist, and actually make things worse while thinking I’m helping. I want to support the women around me as best I can, and better than I do.

If any of these concerns feel familiar to you, dear male reader, then I hope that that means we’re at least aware that we might be at fault. If so, it is our responsibility to awaken and improve that awareness in others, while we continue to work on ourselves.

I am resolved to do better and I want help to be better. You deserve it.

Test Plans are useless. Test Planning is not.

This article triggered me – in a good way – to describe what I think are the biggest reasons for having a test plan. Obviously, the mileage you get from your test plan will vary, depending on your company, environment, customer, regulatory environment, amongst other things.

As we know, simply having a document doesn’t, in and of itself, add much value. Simply having lots of test cases doesn’t add value. Running them does. Actually, just having documents is waste.

In these agile days where communication is rightly valued over documentation, the poor old test plan gets a bad rep, and is often neglected, or produced on reflex and never read.

There are at least two good reasons for producing a test plan, which I will introduce, naturally, through the medium of appropriate quotes.

Planning is a verb

“Plans are useless. Planning is indispensable”. – Dwight D. Eisenhower.

The value we get from test plans is not from the document, but from the planning process. In producing / reviewing the document, we are effectively testing the project. By asking questions and eliciting responses from the relevant people, we are seeking out where information is missing, or confused, or inconsistent.

Testing is pretty much the first role / stage in a project where questions like “What operating systems do we support?” and “Do we support HTTPS?” have to be answered before the work can be completed. It is therefore vital that we ensure that we ask these questions as early as we can so that a) answers can be found, and b) we can start testing those answers.

Over time you can use the test plan as a repository for organisational lessons learned. In the same way as you might update a test case to test for a specific bug that tends to creep back in, or to ensure that you test a out-of-the-way piece of functionality that you tend to overlook, update your test plan so that you ask the questions that will prevent you making the same mistakes again.

The best way to get value of out of the process around test planning is to involve as many stakeholders and interested parties as possible in the production / review of the plan. Your test plan doesn’t have to be perfect; its primary goal is to serve as a means of reaching agreement between everyone involved, and only subsequently to record the specifics of that agreement.

In longer releases, it’s worth periodically revisiting the plan – really, re-reviewing the plan – in case any of the details have changed that impact any aspect of the planned test activities.

In short, the test plan is actually best used as a vehicle to stimulate communication and record decisions.

Ignorance is not bliss

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” – Mark Twain.

It is easy in testing to get too comfortable in one’s testing approach. You might be testing yet another release of the same product. Everything is the same as it was last time, so why bother doing test planning?

Well, how do you know everything is the same as last time? Have you checked? Doesn’t your role exist precisely because shit happens? People make mistakes all the time. Things change, and people don’t tell other people. Happens all the time.

As the quote says, it’s not the things you don’t know that cause problems. It’s the things you are both 100% sure you’re right about, and even 1% wrong about, that bite you. As a tester, any feeling approaching total confidence in your knowledge about anything should cause warning lights to blink in your consciousness.

Therefore, force yourselves to go through the planning process again, even if you think everything is the same as last time.

Only difference between science and messing about…

“If you don’t write it down it never happened”. – Somebody

We do test planning because it adds value, to us as a function and to our projects. As a result, it is part of our testing process to ensure that we all do it.  Because it’s part of our process, it’s subject to audit as part of our ISO9001 certification. If we cannot show that any test activity took place, then we are not following our own process. An external ISO auditor could raise non-conformities which, depending on the level, could jeopardise our certification.

Of course, you might not have any process, contractual or regulatory reason or need to document your test planning, in which case it’s up to you to decide whether it adds value or not. I’d personally feel safer writing down agreed test planning points, if only for ass coverage purposes.

In summary…

  • Make test planning a conversation
  • Don’t get complacent about what you think you know
  • Write it down while it adds value

Don’t Shift Left; Expand Outwards and Accelerate Failure

My boss sent round this article written in September 2017 from TechTarget about embracing shift-left testing as “the latest theory”. The reason he sent it around was not because we should look into adopting it, but because we’ve been doing it – and I checked – since 2008. “Sounds familiar” he quipped.

“Shift left”? Bleurgh!

Firstly, let’s address the term “shift left”. I’ve used the term for a while to explain how we test, but recent Twitter noises make it more clear that the term is limited.

It suggests movement in one direction. Put like that, I don’t think that’s what we really mean. We’re not suggesting that, in order to start getting involved earlier in the lifecycle that we stop doing testing at the end, are we? What we’re really doing is expanding the range of activities that testers get involved in.

“Left” leads to “right”, which suggests a one-dimensional reality in which your options are severely limited. Modern organisations are multi-dimensional, so why limit ourselves to a single direction in a single dimension? Are the activities that occur notionally to the left of testing in the software development lifecycle the only things we can usefully get involved with?

“Shift left” is about getting testing involved earlier in the software lifecycle so we can try and stop bugs even being written. Broadly, it’s about adding value sooner rather than later. But, again, why limit ourselves to adding value?  Why not look in the other direction – towards operations – and see if we can extract value to make our testing better in the future?

But wait, I hear you cry. If we’re expanding, surely we’re spreading ourselves more thinly, and doing less testing at the end? That may be the case, but the idea is that, by getting involved sooner and identifying problems earlier, an hour spent early in the release should save you many times that late in the day. It’s an investment that you hope pays off as fewer bugs and regressions.

Dan Ashby's Continuous Testing in DevOps
Dan Ashby’s Continuous Testing in DevOps

As I typed this, Dan Ashby‘s surely-famous-by-now image popped into my brain. You can test everywhere, not just a bit more to the left.

So instead of “shift left” I’m going to call it “expand outwards”. It doesn’t suggest that you have to give up anything, and it’s direction-agnostic. You can expand in any direction that lets you add or extract value.

What does “expanding outwards” look like?

We’ve been expanding testing outwards ever since we adopted agile-like behaviours. As a company, we embraced Agile methodologies as an R&D operation. Importantly, I had the freedom to come up with a test process from scratch that would allow testers to get involved from the earliest possible point in the release.

Initially, we did just expand in one direction; “left”, i.e into development, design, and requirements. We got involved in design and estimation meetings, we reviewed requirements and user stories, and we got developers to review our test ideas.

Over time, we looked to expand in other directions; firstly to the “right” and Customer Support.  As the people who work closely with the customer and who support the products in live usage, they are the role who know the most about real-world customer usage of the products. They are a vital resource to get involved if you can. We “shifted them left” into testing so that they could add value to us.

We also get involved with testing documentation, and tech authors share our test environments where possible.

Those are just some examples. We didn’t stop there, and neither should you. Remove your filters and think about everyone even loosely associated with the product, which probably means everyone in the business, and includes your customers. Think about what value you can give to, or extract from, each / any / all of them.

Let’s step back; what are we trying to do?

When we first looked to adopt Agile, we spoke to a consultant who probably said lots of words, but two have stuck with me ever since;

accelerate failure

Put another way, if you know you’re going to make mistakes – which you will, because you’re a human being – then do everything you can to make them, or identify them, as early as possible so it’s as cheap as possible to fix them.

We need to accelerate not just the failure, but the remediation loop as well. The goal has to be that errors spend as little time in the system as possible, whether they be requirements ambiguities, or code defects, or whatever. The longer an error remains in the system, the more it becomes cemented in, compounded upon, worked around, and expensive to remove.

“You mustn’t be afraid to dream a little bigger”

What we’re really trying to achieve is the acceleration of failure across the entire software development lifecycle, not just testing. We can expand testing outwards to try and find problems sooner, but if the rest of the operation doesn’t follow suit, or inhibits your ability to do testing in different places, then you will encounter a glass bubble beyond which you cannot add or extract value.

Adoption of accelerated failure needs to be systemic, not piecemeal, affecting only a single role. In order to realise all the value of accelerated failure, everyone needs to do it, or at the very least accommodate it.

The hardest part of this whole acceleration of failure business is overcoming any cultural momentum around who does what and when. An agile (in the small, nimble sense) company, with fewer organisational and physical boundaries between roles / projects, will be able to change direction more easily than a larger, distributed, older organisation whose culture is probably more rigid.

Even in a small organisation, it’s not straightforward. From experience, the hardest part can be getting other roles to allow testing to be part of the conversation. This is probably due to perception issues around what testers do more than anything else. Again, from experience, testers working closely with developers quickly erodes those concerns as the value testers add becomes more evident.

How do I  Accelerate Failure?

Think about the roles you interact with regularly. They are likely people involved in the software development process. Learn as much as you can about how they operate and what they do. Pair with them, read their process docs if they have them. Figure out what info they have that you can use, and vice versa.

Encourage them to reciprocate and to remove any barriers. There should be no reason for barriers to sharing of information that ultimately benefits the business and the customer. Everyone in the business is there to provide, or support the provision of, products to the customers.

Then expand your focus to other roles, and repeat. Ignore organisational hierarchies and chains of command. Knock on doors, cold-email people. Walk about and have coffee.

You’re a tester; a sponge for knowledge. Absorb it wherever you can find it, and let it moisten the brains of as many others as you can.

Don’t limit that value-add to the person in the seat to your left.


PS. Communicate!

If I come across as smug about having taken this approach for nearly ten years, it’s because I am (a little). However, the important takeaway for us all here is to be aware of your approach and compare it to the approaches that others are using. You might be ahead of the curve. You might be putting a new slant on an old idea. You might be using an old idea whose time has come back around.

Don’t assume that what you’re doing, or how you’re doing it, is normal, nothing special, or that everyone else is doing it (better perhaps). Consider that, just maybe, how you’re doing it is innovative, unique, The Next Big Thing.

Setting up an ELK stack with Docker

Recently I’ve been working in an environment comprised of nearly 60 Docker containers. When things go wrong, as they sometimes do, you can make educated guesses about which logs to look at but sometimes you’ll guess wrong. Or you know which ones to look at but want to watch several at once without having 8 PuTTY windows open.

You need something more fit for purpose. Enter ELK. ELK stands for Elasticsearch, Logstash, and Kibana. Rather than explain them in the order they appear in the acronym, I’ll do it in the order in which they are used, but rather than attempt to explain it myself and screw it up, the following summaries are from this helpful O’Reilly article.

Logstash “is responsible for giving structure to your data (like parsing unstructured logs) and sending it to Elasticsearch.”

ElasticSearch “is the search and analysis system. It is the place where your data is finally stored, from where it is fetched, and is responsible for providing all the search and analysis results.”

Kibana “allows you to build pretty graphs and dashboards to help understand the data so you don’t have to work with the raw data Elasticsearch returns.”

What am I building here?

OK, so what does all that look like? It’s pretty simple. You configure your logs to write to Logstash. Logstash then filters the log messages, turning the unstructured messages into structured data. That structured data is output to Elastic, where it is indexed. Kibana then allows you to query that indexed structured data to discover useful information and visualise it.

Getting Started

This is the easy part. Follow the instructions here to clone the ELK stack to your machine and start it up. This basically gets you to pull the three containers and fire them up, with some configuration taken care of. Nothing in this life, however, is for free, so there is some work to be done.

[I must offer all glory to Anthony Lapella for producing such a beautifully wrapped ELK package]


I’ve observed before that modern software development is mostly plumbing. Happily, the ELK stack is pre-configured such that you don’t need to wire together the bits of ELK. However, with ELK, you do need to;

  1. Configure your Docker containers to log to a location,
  2. Configure Logstash to pick them up from that location.
  3. Configure Logstash to filter the data.
  4. Configure Logstash to send the data to Elastic

Configuring container log location

Because I might want to run my containers without sending their logs to ELK, I need a way to turn it on and off without too much messing around.

The way I did this was by using a second docker-compose.yml file. This secondary file – which I called “elk-logging.yml” – contains a section for each of your containers. Each section contains the following;

    log_driver: syslog
        syslog-address: "tcp://localhost:5000"

What this does is tell the container to use syslog, and to send syslog over TCP to port 5000.

So what you need to do is create your secondary YAML file – elk-logging.yml – with as many of the above sections as you have containers you want to log to ELK, with the “my_service” parts replaced with the names of all your containers.

Configuring Logstash input

The next step is to configure Logstash’s input to listen on that port. Fortunately, the ELK stack you cloned earlier already has this configured. The configuration file in in docker-elk/logstash/pipeline/logstash.conf.

Looking at this file shows an input section that shows it listening on TCP:5000.

input {input { tcp { port => 5000 }}

So you don’t need to do anything; this is just for information.

Grokking in Fullness: configuring Logstash filters

This is the most fiddly part of the exercise, because you need to mess about with grok regexes. When you’re dealing with this many containers, the chances of them all using the same log output syntax is remote. There are a couple of approaches you could take;

  1. Specify multiple filters, and fall through each until a working filter is found.
  2. Specify a single filter that is powerful enough to recognise optional components as and when they appear.

I tried both of these approaches, but struggled to get the syntax in the logstash.conf file right, so I eventually settled on the One Grok To Rule Them All. And this is pretty much what it looks like;


Impressive looking, right? I won’t attempt to explain every component, but will try to summarise. Firstly, my log messages seem to broadly be “Syslog format followed by Spring Boot format”, e.g.

<30>Sep 11 10:18:58 my_container_name[1234]: 2017-09-11 14:18:58.328 WARN 14 --- [      thread_name] a.b.c.d.CallingFunction : Oops, something went a little strange, but nothing to worry about.

Everything up to the red colon is SysLog, everything after is Spring Boot. Here’s a map of each bit of the message with the part of the grok that extracts it;

<%{NUMBER:priority}>                <30>
%{SYSLOGTIMESTAMP:syslogtimestamp}  Sep 11 10:18:58
%{GREEDYDATA:container}             my_container_name
[%{POSINT:containerprocessid}\]     [1234]
((?<springtimestamp>%{YEAR}[\-\/]%{MONTHNUM2}[\-\/]%{MONTHDAY}\s%{TIME}))?         2017-09-11 14:18:58.328
((\[)?%{LOGLEVEL:loglevel}(\])?)?   WARN
(%{POSINT:processid})?              14
(\[\s*(?<thread>[A-Za-z0-9\-]*)\])? [     thread_name]
%{GREEDYDATA:logmessage}            Oops, something went a little strange, but nothing to worry about.

The blue text are the names of the fields that we’re extracting. The red text are built-in grok patterns that we’re using; you can find them all here. Everything else is hardcore grok!

Everything after containerprocessid is wrapped inside “()?”, which indicates that everything inside the brackets is optional. This is because, from container to container, message to message, the various components of the Spring Boot log message weren’t always present, and so I need to tell the grok that.

Like it or not, you’re going to have to figure this bit out for yourself. It involves trial and error, and learning each aspect of the regex / grok syntax as you go. When I started this process, I had used regexes a bit over the years, but was by no means a guru. Once you understand the syntax, it’s quite quick to get something powerful working, plus it looks super hard and you can impress all your coworkers who don’t grok 🙂

My advice is; start small, and figure out how to extract each part of your log message one bit at a time. The good news is that there are lots of online grok checkers you can use. Kibana has one built in, and this one was really useful. A lists of grok patterns is also available here.

Configuring Logstash output

Finally, you need to configure Logstash to send the data to Elastic. As with the input section, this is already pre-configured as part of the ELK stack your cloned at the start.

output { elasticsearch { hosts => "elasticsearch:9200" }}

This shows that Logstash is sending the logs out to elasticsearch on TCP:9200.

OK, how do I get all this working together?

It’s pretty easy. There are only 3 steps to it.

1. Update the logstash.conf with your filter grok

Your final logstash.conf file should look something like;

input {
    tcp { 
        port => 5000

filter {
    grok {
        match => {"message" => "<<your grok here>>"}

output { 
    elasticsearch { 
        hosts => "elasticsearch:9200" 

You just need to copy your logstash file over the top of the existing one in /docker-elk/logstash/pipeline. You can copy the original aside if you want, but move it out of that folder as logstash can get confused if it finds two possible config files.

If you need more help on the format of the logstash.conf file, this page is useful.

2. Restart logstash to pick up the changes to the config file

Restarting the logstash container is simple. Make sure you’re in the /docker-elk folder, then simply;

docker-compose restart logstash

Because it’s easy to make a formatting / syntax error in the file, or especially the grok, make sure you check that your logstash container is running, and that it isn’t complaining about syntax errors. To check whether the container is running;

docker ps | grep logstash

If you logstash container is there, that’s good. Then, to check the logstash log to see if any errors are visible;

docker logs -f dockerelk_logstash_1

At the end of the log, you see a message showing the logstash started successfully. If it wasn’t running when you ran “docker ps”, there’s probably an error in the log showing where the syntax error is.

3. Restart your application containers with the ELK override file

Finally, to restart your application containers and get them to log to logstash, navigate to where you application docker-compose.yml and your elk-logging.yml files live, and;

docker-compose stop

docker-compose -f docker-compose.yml -f elk-logging.yml up -d

This tells docker to start all the containers  detailed in the docker-compose.yml, with the additional logging parameters detailed in elk-logging.yml, creating them if necessary, and doing it in detached mode (so you can disconnect and the containers will keep running).

Configuring Kibana

We’re into the final stretch. The last piece of configuration we need to do is tell Kibana the name of the Elastic index that we want to search.

Again, the pre-configuration sorts most of this for you. All you need to do is navigate to http://<your server>:5601. Kibana will load and show a page that asks you select an Index pattern and a time field.

Now, between Logstash and Elastic (full disclosure: I couldn’t figure out which was responsible for this), a new Elastic index is created every day whose names follow the convention “”.

The Kibana page will be pre-populated with an Index pattern of “logstash-*”, which means it will pick up every logstash-related index in Elastic.

It will also be populated with a default time field. You can override this with one of the timestamps your grok should have extracted, but you’re as well keeping the default.

All you then do is accept these settings. Kibana will then connect to Elastic and return a list of fields stored in the indexes that match the “logstash-*” pattern, which should be all the fields your grok extracted (the blue text from earlier on).

Can I look at my data now?

Yes! Go to Kibana’s Discover page and your data should be there. You can explore to your hearts content and start messing around with filters and visualisations to extract all the hidden value that, without ELK, you would find very difficult to extract.


If you find any errors in my instructions, or any part of the above is unclear, or you want / need more information, comment below and I promise I will update it.

Helpful pages

The Sucky Answer Heuristic

We all know that there’s no such thing as a stupid question, but there are certainly stupid answers. All this recent talk of heuristics caused my brain to come up with what I’m calling The Sucky Answer Heuristic.

The Sucky Answer Heuristic is a specific instance of a more general emotional heuristic; if you feel an emotion, something is causing you to feel that emotion. If you’re testing something and have a response then – assuming you’re emotionally stable at the time – that response was likely triggered by something about the thing you are testing.

In this case, if you ask a question and the answer causes you to exclaim some version of “That sucks!”, then it’s worth examining further, and you should keep asking questions until the answers stop sucking, or they only suck. I’ll explain more about when to stop asking questions later.

What’s the point of focusing on Sucky Answers?

A sucky answer to a question is likely an indication of a lack of knowledge. If you’re asking a question of some oracle – which, as a tester, is quite likely – then a sucky response means your oracle isn’t a good oracle, or an oracle at all.

In either case, that’s a risk, and it’s our place to highlight all risks that threaten the value the product is meant to deliver.

Over time, as the product scope and architecture stabilises, the answers to the questions should improve as your oracles have a much better idea of what they’re producing. If the answers are getting worse, you’ve got problems.

In the worst case, really sucky answers could mean that you should stop asking questions, or perhaps even look for another role.

What does a sucky answer feel like?

How do you know what a sucky answer feels like? Well, it feels sucky. You get the answer and are left feeling unsatisfied, or more worried, or frustrated, or any number of negative emotions. It’s a subconscious, subjective judgement, but that’s fine.

Quality in all things is subjective, so the reaction to a poor quality response will also be subjective. However, here are some possible indicators of Sucky Answers, in ascending order of suckiness;

  • An answer that contradicts what you already “know” (this is “good-sucky”;  it suggests some clarification is required.)
  • An answer that takes a while to arrive (though there could be reasons other than the suckiness of the answer for the delay)
  • An answer that answers the question very vaguely (either in delivery, or content)
  • An answer that doesn’t answer the question, or answers a different question
  • An answer never even appears
  • A response that doesn’t acknowledge the validity of the question; “That’s not a sensible question.”
  • A response that doesn’t acknowledge the validity of the questioner; “Why are you asking that question?”

“The answer was sucky because it wasn’t the answer I wanted!”

If you get an answer that wasn’t what you wanted, it’s not necessarily sucky. If a decision is made that you don’t agree with, but the reasoning is explained and is sound and logical, then that’s not sucky. Those sorts of answers you have to accept.

Continuing to ask questions because you don’t like that answer will, in this situation, not improve quality and will just label you as a pain in the ass who always wants things their way.

OK, so how do I work this thing?

You already know how to work this thing. You do this already. It’s not a new idea, I’ve just given it a stupid name. It probably already has a sensible name, of which, in my ignorance, I am unaware.

That said, when I was thinking about this, a picture was appearing in my brain, so I drew it. It has some interesting bits on it that are worth explaining, that might help you decide how to use this feeling of suckiness more effectively.

Broadly, this is a graph of Answer Quality against Time. The Time axis can be a duration of your choosing. I chose the length of a release because, as I covered above, I expect answer quality to gradually increase over the course of a release until all questions are satisfactorily answered before release.

Each time a question is posed and an answer received, we rate the quality of the response, based on the criteria I listed above. The wobbly line is a rolling average of all answer quality scores. A poor answer drags the average down, a good one improves it.

Product Problems

Let’s focus above the X axis first. In this region, the answers are coming, but are delayed, or vague, or confused. I term this the Product Problem zone, because sucky answers point to likely issues with the product. It’s where people don’t yet know everything about the product – which is a risk – so their answers are a bit sucky.

Over time, as people learn more about the product and the scope is clarified, answers should suck less and the answer score average should improve.

When some major aspect of the release changes – the scope changes, a new platform is introduced, a feature is dropped / refactored – the level of uncertainty increases, answers get suckier, and the average falls.

Hopefully, at some point in the release, the answers you get will be readily available and pretty non-sucky. This is your personal “good enough” level. You want your answer to continue to be at least this good to feel confident about how things are going. Not that you should be asked if you feel the product is “good enough” to be released but, if you were, this would be one gut-check you could use.

If the answers are “good enough”, can I stop testing? As long as the product has stopped changing, you could. But as long as things are changing, and new things to be known are being created, you need to keep asking questions and sense-checking the answers. If you declare victory too early and take your eye off the ball, you could not ask a question that would have uncovered a major issue. So, regardless of how good the answers are, keep testing until you have to stop.

If you run out of questions, that’s a different problem. If you’ve run out of testing time, it’s probably not a problem. However, if you do have time left, you need to figure out how to ask more questions.

All this is well and good, and gives you a sense of how things are going, but’s that only useful if you communicate your findings to the right people. You are going to have to translate “My answer suckiness rating is 4” into something that those people understand. Risks are good. Managers understand risks.

Organisational Problems

Let’s now dip below the X axis. Down here are more fundamental problems. Down here your questions aren’t answered. The questions themselves are questioned. You are questioned for asking questions.

Down here, you are no longer able to add value to the product because elements of your organisation are preventing you from doing so. Your place as a valuable member of the organisation is in question.

Any questions you ask are seen as a negative and are interpreted as being destructive. Asking more only makes things worse.

You need to flag these sorts of responses up ASAP to anyone up to and including HR. If it’s only certain people / oracles who give these responses, you can possibly work around it, but if the problem is more systemic, then it’s probably time to look for a new company.


Pay attention to the responses you get when you ask questions. If you don’t like them, or how you got them, ask yourself why. Ask more questions, and flag up any risks that you identify.

Things will always suck a bit, but if you call attention to the suckiness you can use it as a way to help the team focus on areas of risk.




Because your whole life is a test