Friday, 25 March 2011

Without stabilisers - why writing your own test harnesses really is an option

When considering utilising test automation to assist in your testing, one of the most important decisions that will need to be made at some point is what technologies or tools to use to automate the testing. For many the obvious approach will be to look to the software market to identify the tools which are most appropriate. Making the assumption that third party tools are the way to go could be excluding the perfectly valid approach of building custom automation in house. I can understand the perceived benefits of using an out-of-the-box tool to run your tests, and writing your own test harness can appear to be a frightening alternative, but the creation of your own test harness can be an excellent approach to introducing automation into your testing effort and should not be overlooked.

Advantages of writing your own harness

You can start as small as you like

Writing a harness for automation can be a daunting task, however one of the major benefits of in house developed test harnesses is that you can start very simply. Many systems have some kind of command line interface (See Rolling with the Punches for some good places to start.) Something as simple as a script to take data input files to drive a command line function and verify an expected result can be a powerful addition to your testing capabilities if used sensibly (I'm not going to discuss the relative risks of automated approaches here). By starting small and proving the benefits you can then use the results as evidence when requesting time/help to expand your efforts and implement something on a larger scale. No matter how small you start, I'd recommend adopting a set of automation principles like the ones we use here to ensure that you start with a structure that can work long term.

Driven by your needs

It is unlikely that an out of the box tool is going to integrate seamlessly with your team methods and implementing a tool may involve a level of compromise on your ideal approach to testing your product. Any compromises and workarounds may introduce more risk of the automation not meeting your long term needs or proving more troublesome and error prone than the system being tested. While I have been writing this post (it has taken a while) a couple of posts have been published by Liz Keogh and Elisabeth Hendrickson highlighting the dangers of focussing on tools first. By writing your own tool your product and the team processes and interactions drive your automation rather than the other way around.

You control the interfaces and reporting

When creating your own test harnesses you have more control over the information that is reported from the system and the format of it. You have more control in terms of what you want to measure and how and where you want to put this information, such as in a format for loading into a database.

Issues are resolved in tool not tests

If you find an issue with compatibility between the test tool and the application, you change the tool. There is no need to compromise on the test or amend your approach to due to inflexibility with a tool that you are not in a position to modify.

Quick resolution on problems

If you hit an issue with the test harnesses, you can prioritise a fix and resolve it on your own timescales. You are not beholden to another organisation to resolve issues holding up the test development.


As you extend the scope of our testing you can extend the harnesses to suit different scenarios. Having control over your own tool allows features to be added on your priority basis. Initially our test harness covered simple query execution and results comparison. Over time it has grown to cover many other scenarios such as
  • system administration and user management
  • bulk repeat data load
  • parallel operations on clustered servers
  • data schema progression
  • memory monitoring and timing

Tools need development anyway

Many of the testing tools that are available still require a high level of scripting or programming in order to implement fully into a development process. If you are going to have to do this anyway then it may be worth considering going the extra mile and creating your own harness as well to provide extra flexibility.

Advantages of using third partly tools

There are some key benefits that third party tools do provide which need to be considered in the automation decision.

Not reinvesting the wheel

If you create your own automation harness you do so safe in the knowledge that a great deal of what you are doing has been done before, and probably better. Implementing a tools saves time developing infrastructure items such as scheduling, reporting interfaces and results comparison.

Starting feature set

Using third party tools provides a set of functionality out of the box that you can (hopefully) quickly utilise to start testing different aspects of the system. When writing your own harness, functionality is pulled into the system on a priority basis so some lower priority features may be waiting a long time to be implemented.

Availability of outside expertise

If it all starts to go wrong, with a third party testing tool then you have the confidence that there are forums, consultants and user groups out there with experiene implementing the exact tools that you are using and have probably encountered many of your issues before. There is no such support network when your harness is developed in-house.

Externally Supported and Maintained

A test harness is a piece of software and comes with all of the baggage that software development entails. Tools are developed and maintained by people whose job is to do exactly that. Developing your own tool may be an additional overhead on already busy development/test team.

Tools can still be modified

The counterpoint to the last in the list of advantages of writing your own harness, many tools are open source or are highly scriptable to make them more flexible in meeting the needs of the team.

Why my circumstances suit harness creation

So what circumstances suit creating your own harness? I can only speak from my own experience but I'd suggest that creating our own harness has worked well in my current role for the following reasons:-

  • Command line/programmatial API interfaces
  • Most of our server functionality can be operated through command line and file inputs, plus through programmatical APIs. If I wanted to test through a more complex interface such as web then a third party tool would likely be more suitable for any automation of those interfaces.
  • Unique workflow
  • The nature of our interactions with the system mean that "written english" like automation structures would be very difficult to use. ("Given that I've imported this specific 10 million line data file, when I execute this 100 line SQL statement then it yeilds these 1000 results within 10 seconds" doesn't exactly roll off the tongue). Instead we have implemented a layered metadata based approach which allows us to store the intention of each test and step alongside the physical input files and document the "live specification" that our tests provide.
  • Large Volume Data
  • The scale of input data and results of our system mean that even specific database test systems such as DBFit do not scale sufficiently to execute tests matching our customer requirements. By writing our own harnesses we can ensure the scalability of results generation and checking.
  • Scripting/programming experience in team
  • As I stated in my list of automation principles I don't think that testers should necessarily need coding/scripting knowledge to be able to create tests, however developing your own test harnesses does require this knowledge. This can come either from the testers themselves or from others in the organisation helping. We are lucky to possess the relevant skills in our team.

Whatever approach taken to automation will certainly involve some trade-off between the benefits of using a tool versus an in-house development. If you feel that some automation implementation would be appropriate for your testing context but do not know where to start, doing some small scale automation in house may be the perfect way to get you started. You can demonstrate if it really is for you and discover more about your specific needs to aid with the decision on how to progress. It might just be that you decide that an in-house system is the right approach for you long term.

Copyright (c) Adam Knight 2009-2011

Friday, 18 March 2011

Not Going it Alone - using the power of groups to implement change

A recent discussion on twitter relating to having the right individuals in the right roles and the power of individuals working within systems prompted me to write:-
"groups & groupthink powerful force in persisting or changing system, right individual can be fulcrum for change"
I thought I'd expand on this with some examples from my own experience to illustrate how harnessing the power of groups and group thinking can be a far more effective mechanism to implement change than what could otherwise be achieved by an individual attempting to implement the same changes alone.

The perils of going it alone

As someone who takes an active interest in ways of improving software development processes, it is easy to look at the organisation that you are working in and see problems everywhere that need addressing. The temptation is to try to address all of these perceived problems head on, and raise the subject of the changes that need to be made at every opportunity to anyone who will listen. In my experience this approach is not the most effective way of implementing change and can actually result in resistance to ideas that could potentially benefit the organisation:-
  • The existing process is there for a reason
  • The way that things are done now are because at some point someone thought they were a good idea. The phenomenon of GroupThink can be a powerful force in resisting change in the current system. This is the name given by psychologist Irving Janis to the situation whereby people fail to assess appropriate alternatives in preference of maintaining group cohesion, validating their own behaviour through reference to the behaviour of the larger group. Attempting to incite change by criticising the way things are done can be perceived as criticising the group as a whole and can result in the team adopting a separate position as an act of self defence to maintain cohesion.
  • Reputation as an evangelist can diminish your message
  • No matter how valid your points, if you develop a reputation for constantly preaching about the ills of your working practices then this can diminish your ability to facilitate change. Former colleagues of mine have suffered from this problem. One, for example, repeatedly raised the same issues relating to code quality in every development meeting that he attended to the extent that developers in the company warned new starters about him. Another who would print off articles on processes and "Best Practices", that the company was not adopting, and leave them on peoples desks to find in the morning. Both had valid points but their methods of tackling these worked to alienate the other members of the team and in both situations resulted in the group adopting a position of sufferance of the individual rather than appreciation of their views.

The Power of Groups

Most of the occasions when I've seen change effectively implemented, it has not been through top-down management initiatives or super-hero inspiration from an individual, but through the shifting of the group mindset to a position of support for the idea. This is not to say that change cannot be initiated by an individual, but this may be best achieved through the injection of an idea at the appropriate time and allowing the group to actually implement the change in a way that they are comfortable with. Some methods that I've used successfully to achieve this have been:-
  • Engage a small group
  • In my earlier days working in an agile process we were using sprint washup meetings to demonstrate features delivered, but not doing regular retrospectives. Working with David Evans and Marta from SQS had shown the value of retrospectives but I didn't feel that adding another sprint end meeting into the team process would be well received. Instead I started running retrospectives for just the testers. Over time we started to show the benefits and built a small enthusiastic core group around the process we started to extend the invitation out to more members. The fact that they were joining an existing group that was positive around the process helped to enthuse people behind the process I think much more than if we'd started with the entire R&D team from the start. Now the whole development team enthusiastically attends a retrospective at the end of the sprint and some of our best improvements have come out of these.
  • Time your suggestions to the identification of problems
  • For a long time at RainStor we worked without a physical scrum board. I felt that it would be a useful thing to have, however the lack of one was not causing us an particular issues and it was difficult to engender enthusiasm to set anything up. Then in one retrospective the team raised up the problems arising through lack of visibility of sprint items as a serious issue for the team. As the team agreed that this was an something that merited attention, I injected the idea of a scrum board. As the suggestion provided a solution to an acknowledged team problem it was taken on with enthusiasm, and the following iteration the whole team worked to set up the board and include it in our daily stand-up meetings.

Anyone who has benefitted from working in a close knit and communicative team will know what a pleasure and privilege this can be. Strongly cohesive teams can, however, be difficult environments in which to implement change as an individual. By working with the team, growing support for changes through smaller groups and in response to specific team problems, the individual can act as the fulcrum for the change, with the team providing the power to shift the balance and achieve results far more quickly and effectively that any individual would have done.

Copyright (c) Adam Knight 2011