Monday, 14 July 2014

A Map for Testability

Here Be Dragons

Chris Simms (@kinofrost) asked me on Twitter last week whether I'd ever written anything about raising awareness of testability using a mind map. Apparently Chris had a vague recollection of me mentioning this. This is certainly something that I did, however I couldn't remember where I had discussed it. I've not posted about it, which is surprising as it is a good example of using a technique in context to address a testing challenge. As I mentioned in my Conference List, I have a testability target for this year. It therefore feels like an opportune moment to write about the idea of raising awareness of testability and an approach to this that I found effective.

Promoting Testability

As I wrote in my post Putting your Testability Socks On there are a wealth of benefits to building testability into your software. Given this, it is somewhat surprising that many folks working in Software don't consider the idea of testability. In environments where this is the case it is a frustrating task getting testability changes incorporated into the product, as these are inevitably perceived as lower priority than more marketable features. As Michael Bolton stated in his recent post, testers should be able to ask for testability in the products they are testing. The challenge comes in promoting the need for testability, particularly in products where it has not been considered during early development. This is a responsibility which will, in all likelihood, fall on the tester.

A great way that I found, almost by accident, to introduce the idea of testability in my company was to run a group session to the whole department on the subject. I say by accident as I'd initially prepared the talk for a UKTMF quarterly meeting and so took the opportunity to run a session on the subject internally in a company off-site meeting by way of a rehearsal for that talk. The internal presentation was well received. It prompted some excellent discussions and really helped to introduce awareness of the concept of software testability across the development team.

The Way Through the Woods

Even with a good understanding of testability in the organisation it is not always plain sailing. As I mentioned in my previous post, developments that persist without the involvement of testers are most at risk of suffering from lack of the core qualities of testability. It is hard to know how to tackle situations, such as the one I was facing, where lack of testability qualities are actually presenting risks to the software. The job title says 'software tester' so as long as we have software we can test, right?

On that occasion I took a somewhat unconventional approach to raise my concerns with the management team and present the problems faced in attempting to test the software. I created a mind map. As anyone who has read To Mind Map or not to Mind Map will know that I don't tend to use mind maps to present information to others. In this case I generated the map for personal use to break down a complex problem, and the result turned out to be an appropriate format for demonstrating the areas of the system that were at risk due to testability problems to others.

The top level structure of the map was oriented around the various interfaces or modes of operation of the software features. This orientation was a critical element in the map's effectiveness as it naturally focussed the map around the different types of testability problem that we were experiencing. The top level groupings included command line tools, background service processes, installation/static configuration and dynamic management operations such as adding or removing servers.

  • The installation/static configuration areas suffered from controllability problems due to their difficulty to automate and harness
  • The asynchronous processes suffered from lack of controllability and visibility around which knowing which operations were running at which time
  • The dynamic management operations lacked simplicity and stability due to inconsistent workflows depending on the configuration.

One of the key benefits of mind maps, as I presented in my previous post on the subject, is to allow you to break down complexity. After creating the map I personally had a much clearer understanding of the specific issues that affected or ability to test. Armed with this knowledge I was in a much better position to explain my concerns to the product owners, so the original purpose of the map had been served.

Presenting the Right Image

What I have said in my previous post on mind maps was that I don't tend to use them to present information to others, but if they are to be used for this purpose then they need to be developed with this in mind. In this case I felt that the map provided a useful means to assist in developing a common understanding between the interested parties and so tailored my personal map into a format suitable for sharing. I used two distinct sets of the standard Xmind icons, one to represent the current state of the feature groups in terms of existing test knowledge and harnessing, and the second representing the testability status of that area.

Mind Map Key

The iconography in the map provided a really clear representation of the problem areas.

Mind Map

Driving the conversation around the map helped to prompt some difficult decisions around where to prioritise both testing and coding efforts. I won't claim that all of the testability problems were resolved as a result. What I did achieve was to provide clear information as to the status of the product and the limitations that were imposed on the information we could obtain from our testing efforts as a result.

Highlighting the testability limitations of a system in such a way opens up the possibility to getting the work scheduled to address these shortfalls. It is difficult to prioritise testability work without an understanding amongst the decision makers of the impact of these limitations on the testing and development in general.

In an agile context such as mine then legacy testability issues can be added to the backlog as user stories. These may not get to the top of the priority list, but until they do there will at least be an appreciation that the testing of a product or feature will be limited in comparison to other areas. What's more it is far more effective to reference explicit backlog items, rather than looser desirable characteristics, when trying to get testability work prioritised.


Hopefully this post has prompted some ideas on how raise awareness of testability, both proactively and in light of problems that inhibit your testing. As well as this I think that the key lesson here is about coming up with the most appropriate way to present information to the business. In this case, for me, a mind map worked well. In all likelihood a system diagram would have been just as effective. Some of the customers that I work with in Japan use infographic type diagrams to great effect to represent the location of problems within a system in a format which works across language boundaries - something similar could also have been very effective here.

Testing is all about presenting information and raising awareness. The scenarios that we face and the nature of the information that we need to convey will change, and it pays to have a range of options at your disposal to present information in a manner that you feel will best get it across. There's absolutely no reason why we should restrict these skills to representing issues that affect the business or end user. We should equally be using our techniques to represent issues that affect us as testers, and testability is one area where there should be no need to suffer in silence.


Both my previous post Putting your Testability Socks On and Michael Bolton's recent Testability post ask for testability contain good starting references for further research on Testability.

Monday, 23 June 2014

The Conference List

Conference List Notebook

I'm really pleased to be presenting a talk at EuroSTAR again this year. Having spoken before, I know this is a great opportunity and with Paul Gerrard as conference chair I'm sure it will be a fantastic event in Dublin.

There are many benefits to speaking at a conference, the most obvious being the opportunity attend a high profile testing events without having to pay for a ticket. There is also a lot to gain from discussing your work with your peers, as I discussed in my post sparing the time .

There are some less obvious benefits too. These are useful to be aware of, particularly for permanent employees such as myself who are not looking to achieve any marketing value for their product or service from attending. One particularly subtle positive for me results from the shift in perception that arises at the thought of presenting my work to others, and my response to looking at my work more critically.

Getting your house in order

One of the hidden benefits for me in speaking at a conference comes from the extra effort that I put in to completing my in-house background projects leading up to a speaking engagement. In order to attend and present to other testers I need to be coming from a position of confidence in the work that I'm doing. Whilst it should be the case that I have confidence in the testing that we do at all times, it is also the case that for inherently self critical individuals like me, things are rarely exactly as I want them. I always have projects in the pipeline that are aimed at improving the way that we work and filling the gaps that I see in our testing approach. Some of these may be background improvements to our processes and tools, some may be areas of testing areas that I think need attention. Whatever the situation, a speaking deadline provides an excellent incentive to get my house in order and progress those areas that I feel need improvement before I can discuss them with others.

What's on my Conference List?

Here are a few of the things that are on my list to try to do before EuroSTAR this November :

  • Team adoption of stochastic SQL generation

    Last year I set myself the task of creating a tool capable of 'randomly' generating SQL queries based on a data and syntax model designed by the tester. In order to do this I spent some time teaching myself Ruby, as I find including the learning of a new skill helps to maintain enthusiasm for any personal project. I'll save the details for another post, but just an at the stage now where I'd like to get more people involved and enthused about this, with the aim of making it part of our standard testing activities. I'm kicking this off this week with an introductory session with query team.

  • Customised Ganglia monitoring on all test machines

    [Ganglia] ( "ganglia") cluster monitoring has become a core element in our soak and scale testing activities. It supports both generic monitoring of operating system resources and also custom monitoring of metrics relevant to our software operation. In our case this includes, amongst other things, the memory of our processes; the disk space utilised in key areas; the numbers of tasks in our processing and pending work queues. Of course we need to be careful of the Observer Effect here. The collection of metrics on the behaviour and performance of processes and resources inevitably impacts that which it is monitoring. From discussions with our customers I know that the software can start to impact application performance as you increase metrics and machines,. Ganglia has proved very useful in pinpointing resource problems, and I intend to get it installed and running on all of our other test servers in the next few weeks.

  • Improved testability in background services.

    As I wrote about in my post Putting your Testability Socks On we did introduce some testability issues into parts of the system a while ago. Whilst many of these have since been tackled, there are some background processes which still exhibit issues with controllability and observability that I hope to resolve. Splitting a maintenance process down into a series of individually callable operations will allow us to explicitly trigger each of the operations managed by the process. This should provide more control for tests involving those operations, but also prevent the introduction of non-deterministic behaviour into other tests which are currently volatile. Needless to say we will also have the full process running for many other tests.

  • Replace our bug tracking system

    Our bug tracking system was adequate when we were a single team company with few customers. Now that we have a larger engineering department with multiple teams, many customers and, most importantly, multiple supported release branches, the system is struggling and I want to replace it with something more suitable. (Of course we could move towards not using a tracking system at all - Gojko Adzic's old post Bug Statistics are a Waste of Time is a good starting point for anyone wanting to go down that road).

What's your Conference List?

in my experience it is often the background projects that individuals work on that provide the greatest advances in the way we work. The user stories or developments that we are working rarely improve our work processes directly. It is the ideas that arise laterally out of my work that triggers the best ideas, yet these need cultivating by motivated individuals and can languish uncompleted without some targets to deliver.

Of course, does not have to be conferences that we might establish as arbitrary targets to deliver some longer term goals. Presenting on testing at internal company meetings, meeting other testers at testing meetups, recruiting new team members or even family personal milestones (My wife and I have baby number 4 due in July) can act as a target for completing long term tasks. Even in an open culture it is often hard to prioritise personal or background goals in the face of more immediate business needs. Establishing a personal deadline helps me in giving the incentive to deliver, particularly when that target involves presenting on your testing in front of a room full of testing experts. If you are struggling to deliver your background projects - try looking at your calendar and put some targets in place around events that you have coming up - it might just provide the push you need to turn that great idea into reality, and may even provide some material for your next talk.

Wednesday, 28 May 2014

The Friday Puzzle


Every Friday I put together an email entitled the 'Friday Report' which I send round to all of my teams. In it I include relevant articles and blog posts of interest to the various roles within the teams, including testing, documentation and support. As well as these links I also try to include a puzzle for people to tackle.

The original intention of including a puzzle was, as one might expect, as a bit of fun to encourage people to discuss the problem and find a solution. As I researched and presented more puzzles I started to see the scope for a less obvious, yet potentially more valuable benefit to these. I found that the most interesting discussions around the puzzles arose when individuals move beyond searching for the expected answer and start to apply critical thinking around the premises of the puzzles themselves. Through some simple questions I've been able to encourage some excellent debate around the puzzles themselves, calling into question the validity of the assumptions, the integrity of the constraints, and the viability of the solutions themselves.

  • Is the puzzle logically sound?
  • is the solution "on the card" the only one that fits?
  • Is it possible to come up with a different solution if we remove the assumed constraints?

Puzzles share many common characteristics with the situations that we are presented with working in a software company. They are usually based around some simplistic model of the world and the individuals within it that restrict behaviour, in a similar way that behaviour is modelled when developing software solutions. Puzzles may be presented with a limited set of information on the scenario in question, just as problems requiring diagnosis are rarely presented with all of the salient facts up front. Puzzles are presented with carefully chosen language to lead or mislead, just as customers, suppliers and other parties we interact with may sometimes use language to define their problems with the intention of leading us to the conclusions that they feel are relevant. In questioning the puzzles we are practising valuable skills in critical thinking that can be applied directly in our working contexts to ensure we maintain an open mind and consider options beyond the obvious ones presented to us.

I'm really pleased how enthusiastically everyone tackles these puzzles. On most Fridays now the discovery of the given answer is only the beginning of an excellent display of questioning and critical thinking around the problem. I've included here a few choice examples of puzzles and subsequent discussions that I think exhibit the exact characteristics that I'm really hoping for in this kind of exercise.

Choose your execution

This was the first puzzle in which I really started to see and introduce the idea of questioning the integrity of the puzzle itself as opposed to simply looking for the 'correct' answer. The interesting thing here is that the basis of the puzzle is that the prisoner escapes through manipulating the logic of his situation, however for me the language of the puzzle scenario provided scope for exploring other ideas.

Puzzle: General Gasslefield, accused of high treason, is sentenced to death by the court-martial. 
He is allowed to make a final statement, after which he will be shot if the statement is false 
or will be hung if the statement is true. Gasslefield makes his final statement and is released. 
What was his statement?**
  • A: I think the General said “I’m going to be shot”.*
  • Me: Well done, that’s the solution given. I personally would not have let him go – anyone see a way round letting him go?
  • J: Yes – I don’t see any reason why the alternative to “shot” or “hung” had to be “set free”. That’s not stated anywhere. It could have been continued incarceration – or some other method of execution. Anyway, given the statement, as soon as you take one or other of the two prescribed actions, either the statement or the action becomes invalid under the rules. If you don’t take an action, then we don’t yet know whether the statement is true of false – so he could continue to be held.
  • Me: They say they’ll shoot him if his statement is false but don’t explicitly say they won’t shoot him if it is true (i.e. it is an OR not an XOR) I’d have hung him then shot him.
  • S: Or may be shoot him on the leg and then hang him.

The Smuggler

This is a great example of something that is seen a lot in software circles. A solution that, on first examination, appears plausible, but that does not stand up well to closer scrutiny considering the practicalities of the situation.

A man comes up to the border of a country on his motorbike. He has three large sacks on his bike. The customs officer at the border crossing stop him and asks, “What is in the sacks?” 
“Sand,” answered the man.
The guard says, “We’ll see about that. Get off the bike.”
The guard takes the sacks and rips them apart; he empties them out and finds nothing in them but sand. He detains the man overnight and has the sand analysed, only to find that there is nothing but pure sand in the bags. The guard releases the man, puts the sand into new bags, lifts them onto the man’s shoulders and lets him cross the border.
A week later, the same thing happens. The customs officer asks, “What have you got?”
“Sand,” says the man.
The officer does another thorough examination and again discovers that the sacks contain nothing but sand. He gives the sand back to the man, and the man again crosses the border.
This sequence of events repeats every day for the next three years. Then one day, the man doesn’t show up. The border official meets up with him in a restaurant in the city. The officer says, “I know you’re smuggling something and it’s driving me crazy. It’s all I think about. I can’t even sleep. Just between you and me, what are you smuggling?”
What was he smuggling?
  • A: Smuggling gold dust in the sand?
  • L: I think he’s smuggling motorbikes.
  • Me: That is the 'answer on the card' - he was smuggling motorbikes. Any flaws in the puzzle?
  • O: Finding pure sand seems amiss as sand has absorbing properties. The analysis should have uncovered the sand plus all that it had absorbed e.g. water, sack fibre etc.
  • N: Surely the border guard would have noticed that it was a different bike each time?
  • S: could be the same model/colour with false number plates?
  • A: The official must have been suspicious if he was riding what appeared to be the same bike for 3 years but not showing any signs of wear and tear
  • S: He drove at night time; with poor lighting :)
  • Me: If he was smuggling motorbikes into the country - how did he get back? Would the lack of a motorbike on the return journey not have triggered some suspicion?
  • N: Maybe he smuggled cars or bicycles in the opposite direction? Or maybe he came back by bus or train.
  • S: The customs officer won’t be working 24 hours, so on his return on foot, a different officer is working and always sees him on foot.
  • Me: So the guy goes across every day and the customs officer is convinced that he's smuggling something to the extent that it is "driving him crazy" - but he doesn't tell his colleagues to check how the guy gets back and what he has with him? I wouldn't want to hire that customs officer for anything important.
  • S: We know little about him coming back – maybe he is doing the same in reverse…..(different model/make) and the customs officers did talk about it, but the ‘crazy’ one assumed it would be the same bike.

Predicting the score

This is a great example of a puzzle that is trying so hard to be clever that it has omitted the glaringly obvious, that the simplest answer may just be correct. Also included for a great piece of lateral thinking in the final response which is actually more pleasing than the rather unsatisfactory answer on the card.

Bill bets Craig £100 that he can predict the score of the hockey game before it starts. 
Craig agrees, but loses the bet. Why did Craig lose the bet?
  • S: Because the score of a hockey game before it starts is 0:0
  • Me: That's the answer on the card - everyone happy with that?
  • W: Because Bill guessed the correct score.
  • Me: That is some seriously twisted lateral thinking.
  • J: You can predict anything. Bill predicted the score. He didn't bet he would predict it correctly.

The Woman with half boys

This is a great example of where the language of the puzzle included unintended ambiguity. The answer on the card is based on an unsatisfactory premise that you can have half a boy. In this case, as with the previous one, I think that the team came up with a fast more elegant solution than the answer on the card.

 A woman has 7 children, half of them are boys. How can this be possible?
  • A: Maybe “half” means at least half, in which case she has at least 4 boys. Maybe she is expecting another one but doesn’t know the gender, so 4 of the 7 are boys, 3 are girls, and the one on its way could be either.
  • J: The statement is only that half are boys, there's no saying what the other half are - they could be boys too. If so, half are boys, a quarter are boys - any fraction. However, half of 7 of anything seems odd when the individual items are logically indivisible. Of course it could be in a specific application of the term "boy". So if one of the woman's children was a man or a woman rather than a boy, that could leave 3 and 3 boys and girls. That seems really inconsistent, though - being pedantic about "boy" but loose with "children" (and indeed loose with "them").
  • J: The "them" could include the woman, at which point, there are 8 people and if 4 are boys we're there.
  • Me: Well done J for coming up with an answer which, in my opinion, is actually neater than the one on the card (which was that they were all boys)

Carrying Sacks

This one I chose specifically due to ambiguity in the question that I thought would prompt interesting debate. It is a poorly worded puzzle and the team had no problem in dismantling it, and interestingly using it as an opportunity to explore the socio-political implications of the problem scenario!

A farmer and his hired help were carrying grain to the barn. 
The farmer carried one sack of grain and the hired help carried two sacks. 
Who carried the heavier load and why?
  • A: The farmer… the hired help had empty sacks… well it doesn't specify there was grain in them.
  • N: Even if there WAS grain in the hired help’s sacks, it would depend on how much grain was in each sack. So it could be either of them.
  • S: "A farmer and his hired help were carrying grain to the barn." This suggests that both were carrying grain. We don’t know the size or content of the sacks, or the weight of an empty sack. As it stands I do not think we can tell.
  • A: My thoughts exactly. Or maybe they were carrying different types of grain that weighed different amounts. Or maybe one was carrying dry grain and one was carrying waterlogged grain. Without further information it’s all just guesswork!
  • L: I’d hope that if the farmer had paid the guy, the hired man would have been of some help so therefore should be carrying at least as much weight as the farmer.
  • N: Depends what he or she was hired for. It might have been to feed the chickens and collect their eggs.
  • Me: Well done to N and L for getting the answer on the card. I'm with S and N - the first sentence states that they were both carrying grain. The second does not exclude the possibility of the hired help carrying grain, so the question is ambiguous and unanswerable. For me the most sensible answer is the 'obvious' one that the hired help carried more as the probability is that the sacks were full and of equal size, and as L points out, he was hired to help not just mess about with sacks.
  • E: But isn’t it more of a political/economic/philosophical question? The “hired help” represents the oppressed masses, forced to sell their labour for little or no reward. The “farmer” represents the capitalist oppressor. The burden of the hired help will always be the greater, in the exploitative capitalist society in which the farmer and the hired help notionally co-habit. My only question is why the farmer is carrying anything at all.

Riding into the Forest

Another one where the simplistic model of the problem doesn't stand up to practical considerations. Whilst something of an easy target, I really like the answers here.

How far can a horse run into a forest?
  • N: The answer to the puzzle is “half way”. Any further and the horse will be running OUT of the forest.
  • M: Depending on the density of the forest, I’d have to say “as far as the first tree”
  • Me: N has the answer on the cards, although Michael has applied some practical considerations to the issue….any other thoughts?
  • S: With regard N’s answer, that assumes the horse is running in a
  • straight line. It might be going in circles. A few answers from me:
    • As soon as it crosses the boundary into the forest, it is in, so it can’t run into it any further.
    • Until it stops running
    • Until it runs out of the forest
  • N: If the horse was riderless, it probably wouldn’t run into the forest at all. It would run over open ground if it could – less hazardous!
  • L: I agree with N - if a horse was riderless, it would not run through a forest. It would walk. Unless it was being chased, at which point I guess the answer is ‘until it got eaten’.

A Valuable Distraction

Whilst you could argue that these exercises distract from 'real' work I think that they are invaluable in promoting team debate around a subject and, more importantly, engender a team culture of collaborative criticism. I believe in encouraging an environment where it is acceptable to question the invisible boundaries that may have been established. Whether through our own assumptions or the dictate of others, were are often faced with the situation where we are introduced to scenarios with pre-established constraints, and the easiest option is usually to accept these and operate within them. I hope that by practising the process of questioning the scope and boundaries of every situation helps to ensure that we don't just blindly accept the information presented to us. I don't want to work with testers who only look for problems in the functionalities presented to them, I want to work with testers who question the functionality itself. I want to work with support agents who question the conclusions on that have been made by customers on what has caused the problems that they are facing. I want to work with technical authors who question behaviour and how it fits with the product set as a whole rather than simply documenting what has been done. Luckily for me that is exactly what I have, something I'm reminded of every Friday.


Some nice puzzle links can be found here:-