Tuesday 25 October 2011

A Problem Halved - using a context specific cheat sheet to share product test heuristics

A term that is often used when highlighting the skilled nature of the job of software testing, particularly exploratory testing, is the use of testing heuristics. Heuristics are experience based rules of thumb that we use selectively to guide our approach to a problem, such as testing software features. Over the course of a testers career, (which, as I discuss here can be viewed as its own minor evolution) they will evolve a set of personal heuristics. We can apply these selectively in the appropriate contexts to increase our chances of quickly exposing the potential or actual problems with a software solution. The posession of an excellent set of heuristics can make a huge difference in the effectiveness of testing performed within a given timeframe.

Every solution is different


Over the course of time working on testing a product a tester will develop not only their general testing knowledge but also an excellent set of testing heuristics that relate solely to the context of that product. This knowledge is non-portable and valuable only within its present testing context, however within this context these heuristics can be the most valuable resources for targeted testing of the product in question. Some examples from my current context
  • There is an internal boundary within our NUMERIC libraries between NUMERICs of precision 18 and 19 which adds extra boundary cases to any NUMERIC type testing
  • Some validation on data imports is client side and some is server side, error handling and server logging testing needs to consider both types
  • The query system follows a different execution path when operating on few (<10) data partitions so testing needs to consider both paths
  • For our record level expiry, different paths are followed depending on whether some records in a partition, all records in a partition or all records in an import are expired
The knowledge of these facts adds a richness to any testing beyond that which might be obvious from the external, documented behaviour of product.

(Shared) knowledge is power


For some, their heuristics may be closely guarded secrets, the knowledge that elevates them above the competition or renders them invaluable within the organisation. My opinion, however, is that our goal is to produce the best software that we can as a team, and the sharing of knowledge that has arisen through our combined experience is essential.

Having read Elisabeth Hendrickson, James Lyndsey and Dale Emery's excellent Test Heuristics Cheat Sheet, I found this to be a great method of communicating simple high level heuristics. I felt that a context specific heuristics cheat sheet would be an excellent tool for sharing our internal testing knowledge and for use as a guide/checklist in exploratory testing of our product features. Using a similar layout to the testobsessed cheat sheet, I created a simple framework on our Wiki for our own context specific version for my team to use to document our internal heuristics. We break down by product area and then individual functions or operations. These are then annotated with key-word points that might need to be considered when testing that aspect of the product. If a concept merits further explanation then we link to a separate "glossary" where terms can be expanded upon to avoid cluttering up the sheet. This helps to keep the sheet as a lightweight reference tool rather than anything more involved. The page is maintained by the team, with each member updating the list if they encounter any behaviour that might be useful to others if testing that feature in the future. For example, an entry might look like

Record Level Operations
  • Record Expiry - Expire Subset of Records from Tree ; Expire All in Tree ; Expire All in Import ; Record Level Delete ; Records On Legal Hold ; Purged Records
We've been working with this tool for a few months now and have found it to be very useful. We reference the heuristics in exploratory testing, in elaboration meetings and also charter reviews of story coverage as a mental checklist and to prompt new ideas. If new problems are discussed in reviews then we suggest adding them to the sheet so that it is maintained by the team and for the team. (We have not had to maintain the sheet for long enough to drop features or consider versioning which may cause minor headaches). Of course no checklist or cheat sheet should be considered exhaustive and we have to apply our own personal intelligence to every testing challenge, however, as a simple tool for sharing testing experiences and driving new ones, I can recommend it.
image : http://de.wikipedia.org/wiki/Benutzer:KMJ
Phil said...

Nice post - I use the Test Heuristics Cheat Sheet a lot so adding context specific ones to it seems a good idea, I'll try it out

Adam Knight said...

Thanks Phil,

you'll know from the Test Heuristics one, maintaining information in the form of a cheat sheet really helps to keep the information to hand and is lightweight enough to walk through in meetings or reviews as a high level checklist. If you make it printer friendly you could even keep a hardcopy to take to whiteboard sessions.

Adam

halperinko said...

How about exporting and sharing that knowledge base?

I think creating a simple randomizer which when stuck presents you with an idea to follow (when writing tests and/or when exploring), may be great.
(This can also be incorporated within existing test tools, legacy or SBTM)

If we define a common format (CSV or XML for instance), then we could share even cheat sheets of specific technical areas - wanted to do that as part of training for new testers, but never got to promote it.
(Posted several links to good test cheat sheets in XQual forum - the tool I wanted to base it on, as it had free demo server)

Kobi (@halperinko)

Adam Knight said...

Hi Kobi, thanks for commenting.

I don't think there would be any value in exporting the heuristics as, as the title suggests, these are context specific and only relevant to our product. Elisabeth's excellent Heuristics sheet is publicly available and generically applicable, she's already done such a great job already!

Cheers,

Adam

Whatsapp Button works on Mobile Device only

Start typing and press Enter to search