Thursday, 11 March 2010

The power of the tester in requirement meetings

It is my firm belief that Testers should get involved in the development of requirements at the earliest opportunity. In a recent discussion with Gojko Adzic he pointed out the similarities in the process we use at RainStor and that of Chris Matts model of Feature Injection. I have had some occasions recently which provide excellent examples of the presence of a Tester making an invaluable contribution to requirements discussions for the purposes of Breaking the Model.

Scenario 1 - Tester, Developer and Product Manager Discuss a deletion requirement

Product Manager (pm): The requirement is to support the physical removal of items from the system

dev: Great, we've got an existing model for a logical delete using a system query to identify the items to delete. We can use the same model to identify items to physically delete.

tester: What happens in the situation where you've already logically deleted an item? It would not then be visible to the user to query so they wouldn't know that they needed to physically delete that item so the model won't work.

dev: Oh yeah.

The interesting point here was that the developer was tempted to fit the new functionality into an existing model based on current functionality. By breaking the model the tester highlighted the need for a new model, that was not based on existing functionality, and further discussion with the customer.

Scenario 2 - Product Manager, Developer and Tester Discuss a legal hold requirement

pm: The customer wants to be able to flag an object as on hold or not. This is to to prevent it from being deleted if a person or company is involved in a legal investigation.

dev: This just requires a simple boolean flag to identify the items that are held. The easiest way to tackle this is if the item is held then we'll prevent that entire container for that item from being expired or logically deleted

tester: Didn't we have a requirement to physically delete stuff for legal reasons? What happens if the same container contains 1 item that needs deleting and 1 that is held.

dev: Oh yeah, OR for that matter, what happens if the same item is involved in two investigations, once the first is closed the boolean hold flag could be switched off, even though there is still an active investigation. We need a reference count instead.

pm: That sounds better

tester: A reference count would mean that unsetting the flag for a set of items could not be re-applied in the case of failure or cancellation, as you could accidentally dereference the same item twice.

dev: Oh yeah - we need some kind of case ID to apply the hold under, then we can apply and remove based on the case reference, then this would never impact any other cases that have that same item on hold.

pm: Good thinking - I'll run this by the customer.

What I found particularly interesting in this case was that, the original discussion was based around implementing what the customer had asked for. Once the tester had started to introduce the idea of breaking this model, the developer actually went on to identify further examples that highlighted flaws in the original request. In this case the presence of a tester raising issues with the proposal shifted the focus of the session. Instead of discussing how to deliver the customer requirement, the focus moved to finding a model that delivered the business value they needed. This was a good example of challenging requirement, a subject which Gojko presented on at the Agile Testing and BDD Exchange last year. The result was a far more robust model for implementation, and one which addressed a series of issues which the customer had not considered themselves.

In both examples the value of having the Tester present in the early discussions around the requirement were clear. The product manager and developer gained the benefit of the expertise of the tester in critiquing proposed models before committing to a solution. The tester gained an insight into the customer requirement that was not biased by the developers having already started work on a specific solution.

Copyright (c) Adam Knight 2009-2010

Friday, 5 March 2010

Finding the right balance

What should one look for when recruiting for a test team?
Do you take the company profile of "Software Tester", which probably reads something like:-

  • Bachelors degree in computer science or a scientific/mathematical subject
  • 2 or more years experience of software testing
  • Experience of automated testing tools
  • Experience of your documentation tool of choice
  • ISEB foundation certificate

and hand this to the recruitment agents?

Personally I try to avoid this approach. I see little benefit in populating a team with multiple versions of the same skill set. Instead I prefer to take the opportunity to put together a team who together have a range of skills, the members of which can all make valuable and different contributions to testing the product.

In order to do this, the first question that must be tackled is:-

What qualifies someone as a software tester?

My answer would be:-

Anyone who can make an educated assessment of the value of the system under test to a stakeholder of that system, or anyone who can increase the ability of the team as a whole to achieve that assessment.

The first category could contain a number of different abilities such as:-

1. Experience in software testing, particularly working with the technologies in question
2. Experience of working in the industry or business area in which the product is targeted
3. Anyone with experience of using the systems and tools (rival or complementary) which the targeted users of the system will have experience of.
4. Experience of administering the environments, operating systems or types of network within which the system will operate
5. Anyone with experience of working in the legal or legislative environment in which the system will operate

The second may contain abilities such as:-

1. The ability to create and maintain automated test harnesses/suites
2. The ability to configure test environments

Now, clearly if we are putting a team together to critically assess and automate ongoing checking of a system then again just putting together a team of testers with specific domain knowledge but no ability to automate or maintain test infrastructure then that is not going to achieve the team goals. What we require is a team whose combined skills will achieve the goals of the team as a whole.

Creating a skills matrix to identify needs

My approach to identifying the skills required to build such a team revolves around the creation of a skills matrix:-

First I create a list of skills which I feel are relevant to the team and the work that we have to do. I generally group these under headings such as "Testing Skills", "Domain Knowledge", "Test Automation" or similar. I will obtain the input of the current team members on this as well. The result will be a list of skills E.g.
Test Scripting
Team Admin
Test Automation
Performance Testing
Multi-User/Load testing
Web Testing
Test Technical
Unix Shell scripting
Windows Shell scripting
Dot Net
Database Related
SQL Server

Then, I rate these skills in terms of importance to the team. I use 2 ratings, both on a scale of 1-3:-

- Relevance - How relevant this skill is to the team
- Weight - how frequently the skill is required. This essentially translates to the number of team members that are required to possess the skill

Then I rate each member of the team according to their ability. I do this in discussion with each individual, according to the following ratings:-

0 - I would not know where to start doing/using this
1 - I would need help if asked to do/use this
2 - I would be able to do/use this on my own but am no expert
3 - I would be able to do/use this without any help and know what I was doing

By putting these ratings into a spreadsheet and using simple formulae of

total score=SUM(team ratings) - (relevance X weight)
peak score=MAX(team ratings) - (relevance)

The Total column gives us a score relating to the coverage of that skill in our team relative to the weight. The "Max" column is simply the difference between the relevance of the skill and the ability of the highest scoring team member, thus identifying if there is a general ability in the team but we may be lacking a specialist.

A score of zero and above will indicate adequate coverage, below zero indicates a need in the team. I appreciate that the scoring system is arbitrary,it is a useful indicator of need. If it is not indicating useful information then I suggest adjusting the weighting/relevance ratings. The result of this analysis will be a matrix similar to:-
AreaSkillRelevanceWeightAlanBobCarl TotalMax
Test Scripting32333 30
Estimation32332 20
Test Automation32322 10
Performance Testing31222 3-1
Multi-User/Load testing21112 20
Web Testing12211 21
Domain Knowledge 0
SQL9232122 -1-1
ODBC Connectivity22313 31
Business Intelligence32120 -3-1

This provides a simple yet highly visible representation of the current position of the team and where the shortfalls are relative to where you want the team to be.

This approach to identifying the skill needs of the team is useful immediately for personal development plans, to see if the existing team members have an interest in learning or developing on one of the new skills.

The approach is especially useful at the point of recruiting new members to the team. Whereas the temptation is often to recruit based on a role "template" as described at the start of this post, an exercise such as this can sometimes reveal unexpected results in that the needs of the team lie in complementary areas to the traditional testing skills. Recently I recruited an excellent addition to my team who had very little testing experience at all. He did, however, possess system administration skills in the operating environments that our software is installed into, which gives him an excellent insight into the desirable characteristics of the system from the perspective of a valuable stakeholder, the system admin. Similarly another of the best testers in my team has a high level of specific domain knowledge in the area of databases, which is a desirable skill in critically assessing the behaviour of the product from the perspective of a DBA.

I would add the caveat that I have only used this on small teams so am not sure how well it scales, although on larger projects it could be interesting to apply the same approach recursively at the individual and team levels to identify team skills and needs as part of a larger group.

This approach takes some courage, as it requires looking at the combined skills of a team to meet the challenges faced rather than the skills of individual members. There may also be some push back from management, who prefer to see things simply by fitting individuals into role shaped compartments. However the benefits to the company that can be brought by putting together a multi-skilled team to address a multi-faceted task will pay dividends in the long term.

Copyright (c) Adam Knight 2009-2010