Friday, 5 March 2010

Finding the right balance

What should one look for when recruiting for a test team?
Do you take the company profile of "Software Tester", which probably reads something like:-

  • Bachelors degree in computer science or a scientific/mathematical subject
  • 2 or more years experience of software testing
  • Experience of automated testing tools
  • Experience of your documentation tool of choice
  • ISEB foundation certificate

and hand this to the recruitment agents?

Personally I try to avoid this approach. I see little benefit in populating a team with multiple versions of the same skill set. Instead I prefer to take the opportunity to put together a team who together have a range of skills, the members of which can all make valuable and different contributions to testing the product.

In order to do this, the first question that must be tackled is:-

What qualifies someone as a software tester?

My answer would be:-

Anyone who can make an educated assessment of the value of the system under test to a stakeholder of that system, or anyone who can increase the ability of the team as a whole to achieve that assessment.

The first category could contain a number of different abilities such as:-

1. Experience in software testing, particularly working with the technologies in question
2. Experience of working in the industry or business area in which the product is targeted
3. Anyone with experience of using the systems and tools (rival or complementary) which the targeted users of the system will have experience of.
4. Experience of administering the environments, operating systems or types of network within which the system will operate
5. Anyone with experience of working in the legal or legislative environment in which the system will operate

The second may contain abilities such as:-

1. The ability to create and maintain automated test harnesses/suites
2. The ability to configure test environments

Now, clearly if we are putting a team together to critically assess and automate ongoing checking of a system then again just putting together a team of testers with specific domain knowledge but no ability to automate or maintain test infrastructure then that is not going to achieve the team goals. What we require is a team whose combined skills will achieve the goals of the team as a whole.

Creating a skills matrix to identify needs

My approach to identifying the skills required to build such a team revolves around the creation of a skills matrix:-

First I create a list of skills which I feel are relevant to the team and the work that we have to do. I generally group these under headings such as "Testing Skills", "Domain Knowledge", "Test Automation" or similar. I will obtain the input of the current team members on this as well. The result will be a list of skills E.g.
Test Scripting
Team Admin
Test Automation
Performance Testing
Multi-User/Load testing
Web Testing
Test Technical
Unix Shell scripting
Windows Shell scripting
Dot Net
Database Related
SQL Server

Then, I rate these skills in terms of importance to the team. I use 2 ratings, both on a scale of 1-3:-

- Relevance - How relevant this skill is to the team
- Weight - how frequently the skill is required. This essentially translates to the number of team members that are required to possess the skill

Then I rate each member of the team according to their ability. I do this in discussion with each individual, according to the following ratings:-

0 - I would not know where to start doing/using this
1 - I would need help if asked to do/use this
2 - I would be able to do/use this on my own but am no expert
3 - I would be able to do/use this without any help and know what I was doing

By putting these ratings into a spreadsheet and using simple formulae of

total score=SUM(team ratings) - (relevance X weight)
peak score=MAX(team ratings) - (relevance)

The Total column gives us a score relating to the coverage of that skill in our team relative to the weight. The "Max" column is simply the difference between the relevance of the skill and the ability of the highest scoring team member, thus identifying if there is a general ability in the team but we may be lacking a specialist.

A score of zero and above will indicate adequate coverage, below zero indicates a need in the team. I appreciate that the scoring system is arbitrary,it is a useful indicator of need. If it is not indicating useful information then I suggest adjusting the weighting/relevance ratings. The result of this analysis will be a matrix similar to:-
AreaSkillRelevanceWeightAlanBobCarl TotalMax
Test Scripting32333 30
Estimation32332 20
Test Automation32322 10
Performance Testing31222 3-1
Multi-User/Load testing21112 20
Web Testing12211 21
Domain Knowledge 0
SQL9232122 -1-1
ODBC Connectivity22313 31
Business Intelligence32120 -3-1

This provides a simple yet highly visible representation of the current position of the team and where the shortfalls are relative to where you want the team to be.

This approach to identifying the skill needs of the team is useful immediately for personal development plans, to see if the existing team members have an interest in learning or developing on one of the new skills.

The approach is especially useful at the point of recruiting new members to the team. Whereas the temptation is often to recruit based on a role "template" as described at the start of this post, an exercise such as this can sometimes reveal unexpected results in that the needs of the team lie in complementary areas to the traditional testing skills. Recently I recruited an excellent addition to my team who had very little testing experience at all. He did, however, possess system administration skills in the operating environments that our software is installed into, which gives him an excellent insight into the desirable characteristics of the system from the perspective of a valuable stakeholder, the system admin. Similarly another of the best testers in my team has a high level of specific domain knowledge in the area of databases, which is a desirable skill in critically assessing the behaviour of the product from the perspective of a DBA.

I would add the caveat that I have only used this on small teams so am not sure how well it scales, although on larger projects it could be interesting to apply the same approach recursively at the individual and team levels to identify team skills and needs as part of a larger group.

This approach takes some courage, as it requires looking at the combined skills of a team to meet the challenges faced rather than the skills of individual members. There may also be some push back from management, who prefer to see things simply by fitting individuals into role shaped compartments. However the benefits to the company that can be brought by putting together a multi-skilled team to address a multi-faceted task will pay dividends in the long term.

Copyright (c) Adam Knight 2009-2010


  1. Adam,

    Nice, clean, and actionable recommendations about assembling a balanced testing team.

    I like it and am retweeting it on Twitter.

    You focus on "hard skills" which makes sense for the scope of your post and is where most people do and should start when looking to hire for open positions. Having said that, I've been impressed by team leaders who also explicitly add a similar filter for soft skills and testing philosophies to increase the diversity of viewpoints on their testing teams.

    Examples of these could include: exploratory-testing biased vs. scripted bias; general preference for "small and simple" tests vs. "large and complex" tests, Myers Briggs personality types, etc.

    Justin Hunter

  2. Justin,

    Thanks for taking the time to read and make your kind comment on this post. I appreciate the retweet too, I'm delighted that you feel this post worth sharing.

    You make an excellent point about the various personalities and sets of soft skills required to make up a good team. It may prove more difficult to quantify some such qualities to be able to apply a matrix type measurement to these skills, although certainly abilities such as experience in exploratory testing techniques and exposure to certain testing methodologies can be measurable.

    Once the skills you require have been established then a more subjective approach can be layered on top to identify personality traits that would best complement the members that you have in the team. Personally I feel that this type of decision is best made on a candidate by candidate basis, to allow for the unpredictability of personalities and the fact that we might not identify desirable traits until they are placed in front of us.


  3. Hi, This is really nice blog. I like it. Thanks for sharing this useful information through this article.


Thanks for taking the time to read this post, I appreciate any comments that you may have:-