Sunday 25 November 2012

Blood in the Water - why bug 'feeding frenzies' are a good sign

I've been fortunate that my company has enjoyed a successful year and I've been in a position to take on new testers. As the team has grown an interesting phenomenon has become more apparent relating to patterns of bugs that are raised against our system . At first glance it is an apparently worrying trend, however if we look deeper I think it is something that gives me confidence that we're hiring the right folks and ultimately provides an indication of a healthy testing culture.

A feeding frenzy

We enjoy a highly collaborative relationship between the testers and developers across the team. One way that this manifests itself is through open discussion on any issues that we find through testing. Rather than limiting our communication to playing tennis with the bugs database, we'll discuss emerging problems openly with developers. We'll chat about the risks relating to an issue and decide whether to tackle immediately or raise in the tracking system for future prioritisation. Most importantly, though, we'll try to identify the contributory factors behind the issue to pin down exactly what has caused it to occur.

The phenomenon that I find interesting is that, after one of the testers in the team has been discussing an issue in the office, over the next few hours or days I will often see more bugs which have common characteristics to the original issue being raised. The interesting part is that these will come, not from the original tester, but by other members of the team not immediately connected to the original discussion. These will always differ sufficiently from the original issue to not be mere bug plagiarism, but I'll see a clear correlation between the original bug and the subsequent ones to know that the occurrence of the latter was directly influenced by the former.

A Sign of Distraction?

So are the testers getting distracted? That is one way of looking at it. Another is that they are simply copying each other, but given the fact that I place no value on the number of issues a tester raises, this would be pointless. I take a different view. An extremely useful source but rarely mentioned source of testing information is simply listening in on conversations that take place in the office, eavesdropping if you will. If you were to ask me to list my top 10 most powerful testing techniques then eavesdropping would probably rank in there somewhere. When a crop of issues is raised around an area which has recently been the subject of discussion this tells me that the testers in my team have their ears open and are listening to the information that is being generated and shared between their colleagues daily. Not only that but they are using that information, processing it and applying it to their testing.

The Birth of a Heuristic

Testers use heuristics to generate testing ideas. These will be based on combined information from a variety of sources including documentation, conversations, articles and publications, testing theory and simple prior experience. Many of the 'major' issues that I and my team have identified in my system over recent years have been rooted very much in the specifics of the application and its design rather than in programmatical error. The use of context relevant testing heuristics has certainly allowed us to identify and resolve such issues that may otherwise have gone undetected through more traditional 'requirements traceability matrix' based approaches (As I discussed in this post an effective way of sharing these heuristics between team members is to generate and maintain a team heuristics sheet on a Wiki). The fact that a discussion around a new problem area can result in a flurry of related issues being generated to me is an indication that the team are mentally processing new information to generate new heuristics, and then using their instincts to identify other areas of the system where their application might be relevant. This process of heuristic evolution via 'mutation' to me is a valuable sign of critical thinking on the part of my team and is certainly a positive behaviour that should be encouraged. This is why, whenever I see a series of issues being raised within the team that looks like the testers have been copying each others homework, it always makes me smile. After all, you can't turn off instinct in the best hunters, and when there's blood in the water expect a feeding frenzy to follow.

Image: Wikipedia

Phil Kirkham said...

Great post. As a tester of 1 I don't get the feeding frenzy you describe here but after going to a Testers Metup or reading a blog post it can trigger off ideas for me to go and test to see if they cause problems on the systems I am testing

Adam Knight said...

Thanks Phil, I can appreciate your situation having worked as a lone ranger tester before. I totally agree that blogs and publications are a fantastic source of ideas, and meeting great testers such as yourself at meetups is a rich source of testing expertise. As I said in my previous post - "you never know what is going to be relevant"

Thanks for commenting,



Adam Knight said...

Thanks to @chrisgaletti for retweeting and pointing out that 'heuristic mutation' is more appropriately application to a different context. The concept that I'm trying convey is the evolution of new heuristics through adoption, mutation and application by the team. I've tweaked the wording to hopefully reflect that better.

Thanks again to Chris,


Rob Lambert said...

Great post Adam. I've never thought to define this as a heuristic but it's clearly a good one.

This is why I'm so keen to have each development team sat together so that the Testers can hear the discussions about implementation, problems, different views etc. It gives them invaluable information and insights in to areas of interest.

Great post. Thanks for sharing this.

Adam Knight said...

Thanks Rob,

What I'm describing here isn't in itself a heuristic but a process by which new heuristics can be generated.

All heuristics are applicable in specific contexts, and I'd suggest that many originate from an initial discovery that evolves into a heuristic through re-use by the individual or more widescale adoption. Whilst there are many generally applicable testing heuristics, I feel it is important in any context to expand on these to generate more specific ones based on your greater knowledge of the system at hand. For example, in Hendrickson, Emery and Lynsday's excellent Test Heuristic Cheat Sheet they mention a range of numbers that might cause issue in many software systems. In addition to these we have a further set of numbers and numeric precisions in our custom heuristic Wiki that we know have specific relevance to our system. These have developed through initial discovery in testing and subsequent sharing and adoption of that knowledge by the rest of the team.

The behaviour that I describe above shows to me that, whether or not you've had the chance to explicitly share them or document them, heuristics can evolve within a team culture when the individuals have their ears and minds open.

Thanks for reading and commenting.


halperinko said...

Great article,
As testing is based mainly on testers experience with cases he learned about or have seen in the past, we can enhance this basis by reading each other's bugs.
of course being in a small team allows more face to face discussions which may be even better - but in larger organizations we can just take a look at the bug E-Mails passing by.
An error made in one part of the system, may just as well be inflicted on another part, and an idea one tester had from his experience - can give another tester some twist of same idea to verify on his own part of the system / feature.

Another way, is to discuss one interesting bug in a weekly meeting, and raise ideas where else to look.

@halperinko - Kobi Halperin.

Whatsapp Button works on Mobile Device only

Start typing and press Enter to search