Wednesday, 25 April 2012

Textual description of firstImageUrl

If this ain't a bug - why does it feel so bad?

I had a mildly embarrassing experience with Twitter the other day. I had tweeted a link to a new blog post that I'd written , as I've done dozens of times before. I then spent a couple of hours in a meeting and returned to my desk to find a message from Mohinder Khosla (who I can heartily recommend as a fantastic source of knowledge and excellent testing person to follow at @mpkhosla) telling me that the link on my tweet targeted a non-existent page. Mohinder was, of course, correct that the link in the tweet was broken. Annoyed with myself and embarrassed that I'd made such a mistake in a social media forum, I thought over how it had happened and soon came to thinking that it might actually have been the result of a bug...

A bug or not?

Having a busy job and three young kids, my time for reading and blogging tends to get squeezed into the late evening. Once I finish a post I tend to tweet a link straight away but then follow up the next morning with a "New post last night ..." tweet for those sane UK folks who are in bed by 12am. For this second tweet I had got into the habit of just copying and pasting the text of the first, and editing it appropriately. With the Tweetdeck desktop client this worked fine because the link shortening in this application results a link in which the display text matches the link target. As a result my use of this application had grown to depend on this relationship. As I found to my cost, the Chrome version of the tweetdeck application that I happened to be trying out that day, did not uphold this relationship and so when I copied the display text, it did not match the original target and the link was broken.

Was this a bug? The obvious answer is no. The relationship between link target and link address is not guaranteed and the application makes no claims as to support the ability to copy and paste links between applications. As someone who works in IT I knew this full well and, if I had thought about it, I would not have made that mistake.

I was not thinking about it, however. I was following a process which was very familiar to me in a way that I'd evolved to give the desired results when using the specific tools in question. I don't want to have to think about what a piece of software is doing when I am using it, and the best software is designed in such a way that I shouldn't have to. In this case I would have had to apply my understanding of the underlying technology to have avoided this mistake. Secondly, I was using the Chrome application based on expectations that I had developed around using the desktop version of the same product. Inconsistencies between the two applications resulted in a failure to meet my expectations and consequently a mistake in my use of the former version. Lastly, and most importantly of all, the behaviour of the Chrome application made me, a long time user of Tweetdeck, feel embarrassed and angry to the extent that I stopped using that version of their product. No matter whether the behaviour was technically correct and conformed to the target design and use cases for that versions, to me personally it was a bug.

A bug is in the eye of the beholder

What constitutes a bug can be one of the most contentious discussions in software development, causing particular stress for relationships such as between tester and programmer, and supplier and customer. When it comes to straightforward programming errors then the definition will hopefully be clear and unequivocal. When we consider a bug in terms of the personal feelings and opinions that it elicits then the waters of what constitutes a bug are a lot more murky.

I would not personally limit the definition of a bug solely to the terms of a coding defect. To me, a bug is a behaviour which elicits a negative emotional response in a positive stakeholder (I use positive stakeholder to exclude those who have a negative relationship with a product, such as a competitor who's emotional response is likely to be inversely propoprtional to our successful behaviours). Once the personal emotional influence is acknowledged then the decisions on what to fix and what behaviours to change become human ones rather than technical ones, raising some interesting implications:-
  • The decisions around fixing the issue can be defined not around technical severity but on how many people will be impacted by the behaviour and how bad will it make them feel. My problem with the links in Tweetdeck might not consitute a bug on the basis of my feelings, however if half the user community felt the same way then a change in behaviour would certainly be justified. If one marathon runner had had their details published on the internet this week, I doubt that the story would have merited the BBC News coverage that it actually received.

  • Two users of the same system may have mutually exclusive opinions on how that product should behave, such that a fix for one will constitute a bug for another. I've certainly encountered this scenario when attending user groups for software products in use by a number of different organisations with contrasting business workflows. In Microsoft Outlook, the fact that the meeting reminders took focus from other applications was a sufficient problem for Microsoft to remove this. For me, and many others, the fact that this prevents reminders from popping up is a major problem, as an icon on my taskbar turning orange is not sufficient to distract me from what I am doing and remind me that I have a meeting to attend. Both behaviours constitute bugs to certain individuals resulting in frustration and anger, and in my case installing another application (the excellent Growl for Windows) to workaround the issue.

  • The impact of the bug may not occur now, but at some point in the future, even after the application has ceased to operate. If the data is ported to another application. I saw a system a few years ago which defined a primary key on its non-normalised transactions table based on forename, surname and DOB - obviously not a good basis for a primary key. The system had not encountered an issue with this up until that point but the nature of the system and the data could have had very serious negative consequences on individuals in the future. Working in the data storage and analysis space I'm all too aware of the frustration that is caused when trying to transfer badly validated data from one system to another.

  • The emotional impact of a bug may not be caused directly by the software itself, to the extent that the victim of the bug are unaware that their emotions are the result of a software behaviour. If a security bug, such as the London Marathon one, releases someones details resulting in fraudulent or undesirable results for that individual then they might be totally unaware of where their details were obtained.

  • A bug that is very hard to justify in any quantifiable terms, might be much more easily justified in emotional ones. Whenever I use the ticketing machines in UK train stations I always feel that they should be faster. I have no supporting evidence for this other than that I see the same emotions in all those who try to use them - frustration, impatience and anger that it takes so long to achieve what should be a simple operation. Attempting to quantify this in terms of transaction times makes it hard to justify a change. When, as I saw today, this results in individuals missing their trains then the emotional justification can be much more powerful.

  • A coding mistake can exist in a product for years and not consitute a bug, as long as no-one at any point, either now or in the future, has a desire that it should be/should have been changed.

  • No matter how much testing you do to ensure you have requirements traceability and functional correctness in relation to the specification, your product can still be full of bugs if the users hate the way that it works.

So I've moved back to using the Tweetdeck desktop application for now. It does mean an extra application running on my laption but at least I can copy and paste my links with impunity. As to whether my problem was caused by user error or was actually a bug - I'll leave that for you to decide - bugs are in the eye of the beholder after all.

Tuesday, 17 April 2012

Textual description of firstImageUrl

Not One Problem - Why We Shouldn't treat Test Automation as a Solution Delivery Project

I recently posted a tweet saying
"You decide to include test automation on your software development project. Congratulations, you now have two software development projects."
Whilst I hold this to be a valid statement, on reading it back I think that it misses the point for a lot of test contexts, including mine. The software development that I work on is not a project, it is an ongoing development. Although there are goals and timescales, these are part of an ongoing iterative process to continuously develop and support a software product. Likewise the test automation efforts that support this development are not a project, this is an important distinction to make.

To be or not to be ... a Project?

A project by definition is a finite activity to run over a fixed period delivering a finite set of specified goals. The classic project model is one of gathering requirements, selecting or developing a solution that best meets those requirements and implementing this solution. Once the solution has been implemented the "users" can then achieve their business goal through the utilization of the features of that product. The temptation when faced with a test automation challenge is to treat this like a traditional business problem which can be addressed by a project based approach. There are many tool vendors out there who market to this model, selling test automation solutions with promises such as "Code Free" designs and "Point and Click" test development once the solution has been implemented. Unfortunately there is a fundamental flaw in the principle of a test automation solution, and that is that test automation does not represent a fixed problem to solve. There are plenty of well defined business problems which can be addressed through a solution delivery project based on a static set of user requirements. The automated testing of an ongoing dynamic software product development is not one of them.

A moving target

The very reason for introducing test automation is to allow us to obtain measurements and run checks to gain confidence in the behavior of a constantly changing system. The advent of agile software development has introduced a higher level of flexibility into software development than was previously seen in many organisations. This furnishes companies with the ability to change focus and adapt to emerging markets quickly with rapid changes to their software solutions. For many organisations, including my own, the target markets and therefore functionality of their products will end up changing significantly over time, with the result that the demands placed on the test automation will change as well.

If I was to write up a list of requirements for a software test automation solution for my product as of 5 years ago, it would read very differently that the same list drawn up today. At that time our software was primarily implemented as a single server database archiving solution with a limited workflow. If at that time we had taken the approach of implementing a rigid test automation solution from the outset then this would have severely limited our scope to adapt with the changes to the software over time to the multi-platform, multi-server data management system that we now test.

Anticipating change

In order to tackle how to approach test automation the key first step is understanding that you are not trying to address one problem. You are addressing a series of problems, each one new and unique and unlikely to have been predictable and the inception of the testing activities. These problems are a inevitable result of working on a changing and evolving software product. Adopting the approach of our automation, not as an up front delivery project but as an ongoing and incremental development activity in its own right, puts us in the driving seat to adapt to these problems as they arise. Here are a few of the changes that we've been able to implement into our test automation over the last few years.
  • Move from a single operating platform to over 40 platform/storage combinations
  • Moving from a static import then querying archiving structure to a more dynamic data management approach
  • Integrating parallel java test harnesses for multi-threaded query testing
  • Allowing the iteration of test suites to scale import and query activity
  • Implementing random execution order into the test packs
  • Adding support for executing test packs across multiple servers
  • Enabling the execution of multiple test packs in parallel against a suite of test servers
  • Adding in support for testing new API interfaces
  • Implement more interactive html based reporting allowing quicker drill-down into failure reasons

This week the team and I are working on some changes to isolate the test archives from each other more cleanly to allow implementation level options to be tested in isolation. Sometimes we do face challenges in the prioritization of automation development efforts alongside new feature testing. As I wrote about in this post on writing your own test harness, what we do have is the flexibility to adapt as new problems arise. As Adam Goucher pointed out when he referenced that post, this is not limited to writing your own harness, the careful choice of customizable automation tools can also furnish this flexibility. In both cases what is required is an approach that test automation is not something that can be defined up front and used for evermore. Instead it is a continuous development effort that sits in parallel with the development of the main product, every wave of change in the latter reflected in the former, and with the success/failure of each very much related to the effective delivery of both.