Advantages of writing your own harness
You can start as small as you like
Writing a harness for automation can be a daunting task, however one of the major benefits of in house developed test harnesses is that you can start very simply. Many systems have some kind of command line interface (See Rolling with the Punches for some good places to start.) Something as simple as a script to take data input files to drive a command line function and verify an expected result can be a powerful addition to your testing capabilities if used sensibly (I'm not going to discuss the relative risks of automated approaches here). By starting small and proving the benefits you can then use the results as evidence when requesting time/help to expand your efforts and implement something on a larger scale. No matter how small you start, I'd recommend adopting a set of automation principles like the ones we use here to ensure that you start with a structure that can work long term.Driven by your needs
It is unlikely that an out of the box tool is going to integrate seamlessly with your team methods and implementing a tool may involve a level of compromise on your ideal approach to testing your product. Any compromises and workarounds may introduce more risk of the automation not meeting your long term needs or proving more troublesome and error prone than the system being tested. While I have been writing this post (it has taken a while) a couple of posts have been published by Liz Keogh and Elisabeth Hendrickson highlighting the dangers of focussing on tools first. By writing your own tool your product and the team processes and interactions drive your automation rather than the other way around.You control the interfaces and reporting
When creating your own test harnesses you have more control over the information that is reported from the system and the format of it. You have more control in terms of what you want to measure and how and where you want to put this information, such as in a format for loading into a database.Issues are resolved in tool not tests
If you find an issue with compatibility between the test tool and the application, you change the tool. There is no need to compromise on the test or amend your approach to due to inflexibility with a tool that you are not in a position to modify.Quick resolution on problems
If you hit an issue with the test harnesses, you can prioritise a fix and resolve it on your own timescales. You are not beholden to another organisation to resolve issues holding up the test development.Extensible
As you extend the scope of our testing you can extend the harnesses to suit different scenarios. Having control over your own tool allows features to be added on your priority basis. Initially our test harness covered simple query execution and results comparison. Over time it has grown to cover many other scenarios such as- system administration and user management
- bulk repeat data load
- parallel operations on clustered servers
- data schema progression
- memory monitoring and timing
Tools need development anyway
Many of the testing tools that are available still require a high level of scripting or programming in order to implement fully into a development process. If you are going to have to do this anyway then it may be worth considering going the extra mile and creating your own harness as well to provide extra flexibility.Advantages of using third partly tools
There are some key benefits that third party tools do provide which need to be considered in the automation decision.
Not reinvesting the wheel
If you create your own automation harness you do so safe in the knowledge that a great deal of what you are doing has been done before, and probably better. Implementing a tools saves time developing infrastructure items such as scheduling, reporting interfaces and results comparison.Starting feature set
Using third party tools provides a set of functionality out of the box that you can (hopefully) quickly utilise to start testing different aspects of the system. When writing your own harness, functionality is pulled into the system on a priority basis so some lower priority features may be waiting a long time to be implemented.Availability of outside expertise
If it all starts to go wrong, with a third party testing tool then you have the confidence that there are forums, consultants and user groups out there with experiene implementing the exact tools that you are using and have probably encountered many of your issues before. There is no such support network when your harness is developed in-house.Externally Supported and Maintained
A test harness is a piece of software and comes with all of the baggage that software development entails. Tools are developed and maintained by people whose job is to do exactly that. Developing your own tool may be an additional overhead on already busy development/test team.Tools can still be modified
The counterpoint to the last in the list of advantages of writing your own harness, many tools are open source or are highly scriptable to make them more flexible in meeting the needs of the team.Why my circumstances suit harness creation
So what circumstances suit creating your own harness? I can only speak from my own experience but I'd suggest that creating our own harness has worked well in my current role for the following reasons:-
- Command line/programmatial API interfaces Most of our server functionality can be operated through command line and file inputs, plus through programmatical APIs. If I wanted to test through a more complex interface such as web then a third party tool would likely be more suitable for any automation of those interfaces.
- Unique workflow The nature of our interactions with the system mean that "written english" like automation structures would be very difficult to use. ("Given that I've imported this specific 10 million line data file, when I execute this 100 line SQL statement then it yeilds these 1000 results within 10 seconds" doesn't exactly roll off the tongue). Instead we have implemented a layered metadata based approach which allows us to store the intention of each test and step alongside the physical input files and document the "live specification" that our tests provide.
- Large Volume Data The scale of input data and results of our system mean that even specific database test systems such as DBFit do not scale sufficiently to execute tests matching our customer requirements. By writing our own harnesses we can ensure the scalability of results generation and checking.
- Scripting/programming experience in team As I stated in my list of automation principles I don't think that testers should necessarily need coding/scripting knowledge to be able to create tests, however developing your own test harnesses does require this knowledge. This can come either from the testers themselves or from others in the organisation helping. We are lucky to possess the relevant skills in our team.
Whatever approach taken to automation will certainly involve some trade-off between the benefits of using a tool versus an in-house development. If you feel that some automation implementation would be appropriate for your testing context but do not know where to start, doing some small scale automation in house may be the perfect way to get you started. You can demonstrate if it really is for you and discover more about your specific needs to aid with the decision on how to progress. It might just be that you decide that an in-house system is the right approach for you long term.
Copyright (c) Adam Knight 2009-2011
Post a Comment