Friday, 30 March 2012

The Automation Dichotomy

As I write this post I'm in the process of automating a test. The test is to allow me to examine the system behaviour under a specific target load scenario for a potential customer. The scenario is a relatively complicated one and I am encountering a few issues as I progress with the automation. The main problem that I'm facing in automating the test is not the complexity of the target scenario. It is not the limitations in functionality of the harness that I'm using for the automation. It is not limitations in the software under test, although it potentially relates to all of these.

The problem is me. If I am honest with myself, I know that I want this test to run to completion. This is a terrible position from which to start performing any testing activity, automation or not.

A Conflict of Interest


When testers are automating tests they are subject to an interesting dichotomy. On one hand, the end goal of this activity is to execute some test attempt to expose information on the target software. Hence the activity is a testing one aimed at increasing our ability to critically assess the system.

One the other hand, the process of automation is a creative activity in which we are developing new routines and programs, and therefore our natural tendency will be towards trying to get these to run successfully.

The danger that I certainly face when performing the activity of test automation is that I am subject to a positive bias in wanting the tests to run. When I am creating an automated test and a run encounters an issue my natural inclination is not to investigate the underlying cause, but to take the software development approach of finding an alternative path that allows me to reach my end goal of automating my target scenario.

A programmer faced with a failure in their code might furrow their brow and utter the immortal line - "It should work" - often combined with a look of incredulity. In a similar fashion, when faced with an automated test that doesn't run, a tester's first reaction can be one of frustration and disappointment rather than a more appropriate stance of inquisitiveness. In his book "Perfect Software and other illusions about Testing" Jerry Weinberg relates an example of a tester who was so impressed with the automation tool testing her search-page that she hadn't confirmed that the tool was actually checking the search returned anything. This may be an extreme case, but it is very easy to fall into the mindset of placing value on something that appears to run successfully at the expense of making more important observations on the behaviour of the software.

The times that the software is most exposed to the discovery of new issues is when it is subjected to a usage or workflow for the first time. This holds whether in a new customer implementation, an unexpected usage or the execution of a new test. The process of automating tests is a great opportunity to expose new issues in our software. The individual responsible for the automation needs to balance the natural problem solving creative approach of such an activity with a critical testing mindset in order to take advantage of this opportunity. Knowing that I can suffer from this problem, here are a few ideas that I think can help to avoid falling into this trap:-

  • Treat automation as a testing activity

  • Many argue that automation should be done by programmers. This is not the case in my company. I certainly agree that the automation harnesses and tools are best developed by those with programming knowledge, however I do think that abstracting testers away from the automated test design activities is a risky approach. Just as a test will yield the greatest information the first time that it is run, so the activity of automating tests can expose new issues and should be treated as an exploratory activity. In creating the tests we have an opportunity to examine the system in new ways which could be missed if our focus is on getting the test working. Treating the process of creating the automation as a testing activity can help to ensure that the correct mindset is adopted when tackling the task and that issues are not glossed over. My main concern with automation strategies in which the fixtures to drive the functionality are written by the same folks that engineer the software is that both software and fixtures are written from the same mindset. Many bugs do not relate to coding errors but simply by looking at the problem from a different perspective. If any bias towards a specific model of the solution is embedded into the automation as well as the software, this may prevent the tester from using the automation to drive the software in unexpected ways to expose issues.

  • Question the reasons for failures

  • Most initial attempts at automating a test will not run cleanly. Rather than immediately hacking around to get them working, spend some time questioning why the test did not run. Is this a scenario that the user could encounter? If so, was it clear from the system output what the problem was and how to resolve it? If the resolution required key developer knowledge that may not be in the hands of the customer, does this indicate a problem?

  • Look for usability issues

  • IF you are struggling with the complexity of a piece of automation, it could well be the case that the functionality in question is unnecessarily complicated. Review the user facing functionality and discuss with the product owner to see if there is a problem, and examine the potential to simplify the feature.

  • Resist the urge to rerun on failure

  • Clearing down and hitting the run button again "to see if it works this time" could be hiding a multitude of state based issues which could be lost by resubmitting. Even if the software turns out to be OK it is likely that the tests themselves are not as isolated as they should be and you may require some better setup and teardown routines. Resolving this will prevent multiple false negative results cropping up on future runs.

  • Note down issues to follow up as Exploratory Tests

  • It might be that we don't have time to investigate something immediately, it's not ideal but it is a fact of life. Try jotting down issues using a notepad or Rapid Reporter and follow up later in exploratory testing charters once the automation task is completed.

Even as I write this, my test run has just reported "can't create the archive as it already exists" - oh well, I must have forgotten to clean the old run up, I'll just delete the directory and run it again...

Thursday, 22 March 2012

Death of the White Board - A Parable

In recent weeks I've raised two significant bugs that shared a common characteristic. These bugs did not appear in our bug tracking system. In fact they did not relate to a line of code that had even been written. These bugs were raised against models for potential solutions that had been proposed by the programmers in my company as we were discussing these models around a white board. As part of every story that we work on, the testers and programmers will elaborate the details of the story and produce Acceptance Criteria as well as risks and assumptions that we've identified. Generally as part of these discussions, or soon after, we'll discuss the developers' models and ideas for solutions to meet the requirement. This presents the team with a great opportunity for early testing and potential to raise issues before any time has been wasted in coding these issues into the software.

Whilst I feel that this is a very effective approach, saving the company both time and money, the benefits are very hard to measure. I can see this being an issue in the types of organisations who define their testing productivity based purely on metrics around the "artifacts" of software testing such as test case counts, bug statistics and the like. Reading this excellent post by Elisabeth Hendrickson the quote that really stood out for me was "The best way to screw up the usefulness of any process metric is to use it to judge people.". This got me thinking and the result, in true "weird episode of House" fashion, was a short story.

I hope you enjoy it, if not then let me know and I'll promise never to do it again.

Death of the WhiteBoard


Jen sat in uncomfortable silence as the man aross the desk from her examined the papers on his desk. It was clear to Jen that this was the first time that the man, her new senior manager Paul, had looked at the information in front of him. From what Jen could make out the sheet contained her name at the top and a series of figures in tables below. Finally Paul spoke.
"Hmmm. As you know Jennifer we've put in place some new performance measures in the last few months to measure tester productivity. It seems that your bug count is below what we would consider satisfactory"
Jen's worked for a small company who had recently been acquired in a takeover by a multinational organisation called Zefman inc. Paul worked for Zefman and had been brought in as a senior manager to implement their "Industry Standard Best Practices" around the software development processes in Jen's team. This was her first meeting with Paul, a 6 month performance review.
"But I've not had any issues before and we have a great reputation for quality with our customers"
replied Jen,
"anyway I don't always raise bugs in the tracking system as I've got an excellent working relationships with the programmers in my team. Sometimes I raise issues based on their solutions in discussions around a whiteboard before they've even written any code, and those issues don't get counted in my detection rates."
Paul continued, patronisingly,
"Well that approach may have been fine for a small company such as yours, however you're part of a bigger more professional organisation now and we need to ensure that you are adhering to our processes and meeting our minumum standards in your KPIs. The guys on Omega team have a test to live bug detection rate to be proud of, we expect you to achieve the same, as an encouragement we'll give you a performance related bonus based on your Defect Detection Efficiency"
Jen was tempted to argue further about the false incentives and potential for gaming the system with test to live ratios, but she could see Paul was not interested. The other teams that were part of Zefman were playing the game and she could see that Paul was not going to accept any other approach from her. Walking away with a sinking feeling Jen returned to her desk and forlornly fired up the bug tracking system.


Time passed and six months later a somewhat more jaded Jen found herself sat opposite Paul again.
"The thing is Jen"
, he said in a manner inappropriately familiar given that this was the second time that they'd met,
"your bug count is looking better but your team velicity is down, you guys are now not even achieving half of the story points per sprint than the Omega guys"
Jen bit her lip. Zefman had embraced "Agile" based on the prospects of achieving higher productivity and had latched onto the idea of velocity as a means to measure and compare team productivity. Instinct told Jen that there was a time and audience for pointing out that Story points only had relevance in relation to work previously done by that team and comparing them across teams was meaningless. This wasn't it.
"You were achieving a good velocity when I took over. Now you are making me look bad, what happened"
Resisting the urge to scream, Jen gave a measured response,
"Well, if you remember I stated that I used to raise many issues directly with the programmers in discussions around the whiteboard. These discussions were very effective at reducing errors making it into code. Now I'm focussing on getting bugs into the tracker like you asked and so many more issues are not found until code has been written costing time, velocity and therefore money"
At the mention of money Paul's eyes lit up.
"Do you mean to say that by raising issues in whiteboard discussions we can save the company money and increase velocity? In that case it should be encouraged. I'll discuss with senior management how we can incentivise this, and of course we'll have to introduce some means of formal measurement"
Jen came away ambivalent. She was pleased that the value of the way that she had worked previously but the idea of attempting to measure and place incentives around this left a bitter taste.

As a result of their discussion Paul convinced management to implement the "White Board Bug" incentive whereby bugs raised by testers against developer ideas around white boards were actively encouraged. New white boards were installed in every Zefman office. In order to tackle the difficult task of measuring Tester productivity in this area it was decided that in order for an issue to qualify, all attendees in the discussion should agree that it was a valid issue, and that some change was required in the proposed solution as a result. Initially the project appeared successful. Testers and programmers across Zefman engaged in discussions earlier and issues were being found prior to coding that increased velocity around the company, making senior management very happy. News of the scheme even started to spread outside of the organisation as videos of the annual conference talks with titles such as "The White Board Way - a Zefman success" made it into wider circulation.

Jen's fears did not take long to manifest themselves, however. Jen's team continued to work happily much the same way as before, and Jen enjoyed the bonuses she received for the contributions made in the meetings, even if it did feel a little artificial. Elsewhere in the larger organisation things were not going as well. The Omega team had cottoned on to the fact that the programmers had nothing to lose from the incentive so were arranging with the testers to introduce faulty solution models deliberately and then share the bonuses that the tester received for raising the inevitable issues. In an attempt to counter this the senior management imposed an incentive for the developers whereby there bonus allocation would be diminished each time a "White Board Bug" was raised against their solution. This caused even more issues. The programmers were up in arms about the impact of white board issues on their bonuses and actively avoided white board discussions until they had spent hours working through every facet of their solution idea to make it bullet proof. The testers meanwhile would spend hours trying to think of valid problems in the programmer models and refuse to share these unless in a formally recognised white board session.

When whiteboard discussions did take place they could run for hours while the validity of issues was debated, and Jen heard stories of fist fights in the Gamma department. Management reacted by installing video cameras around the white boards and implementing a specialist team "The White Board Panel" to review the videos and arbitrate any team disagreements.

A few months later, inevitably, the incentive was scrapped. Paul left the company under a cloud and cheers rang out in the Omega and Gamma teams when the white van men ripped out the new whiteboards and dumped them in skips in the car park at Zefman HQ. When the vans eventually got round to removing the whiteboards from her office, Jen was not there. Her and her team were in a small room in the basement discussing possible solutions to a new customer requirement ...

... around a flipchart.

ShareThis

Recommended