Wednesday 25 April 2012

If this ain't a bug - why does it feel so bad?


I had a mildly embarrassing experience with Twitter the other day. I had tweeted a link to a new blog post that I'd written , as I've done dozens of times before. I then spent a couple of hours in a meeting and returned to my desk to find a message from Mohinder Khosla (who I can heartily recommend as a fantastic source of knowledge and excellent testing person to follow at @mpkhosla) telling me that the link on my tweet targeted a non-existent page. Mohinder was, of course, correct that the link in the tweet was broken. Annoyed with myself and embarrassed that I'd made such a mistake in a social media forum, I thought over how it had happened and soon came to thinking that it might actually have been the result of a bug...

A bug or not?


Having a busy job and three young kids, my time for reading and blogging tends to get squeezed into the late evening. Once I finish a post I tend to tweet a link straight away but then follow up the next morning with a "New post last night ..." tweet for those sane UK folks who are in bed by 12am. For this second tweet I had got into the habit of just copying and pasting the text of the first, and editing it appropriately. With the Tweetdeck desktop client this worked fine because the link shortening in this application results a link in which the display text matches the link target. As a result my use of this application had grown to depend on this relationship. As I found to my cost, the Chrome version of the tweetdeck application that I happened to be trying out that day, did not uphold this relationship and so when I copied the display text, it did not match the original target and the link was broken.

Was this a bug? The obvious answer is no. The relationship between link target and link address is not guaranteed and the application makes no claims as to support the ability to copy and paste links between applications. As someone who works in IT I knew this full well and, if I had thought about it, I would not have made that mistake.

I was not thinking about it, however. I was following a process which was very familiar to me in a way that I'd evolved to give the desired results when using the specific tools in question. I don't want to have to think about what a piece of software is doing when I am using it, and the best software is designed in such a way that I shouldn't have to. In this case I would have had to apply my understanding of the underlying technology to have avoided this mistake. Secondly, I was using the Chrome application based on expectations that I had developed around using the desktop version of the same product. Inconsistencies between the two applications resulted in a failure to meet my expectations and consequently a mistake in my use of the former version. Lastly, and most importantly of all, the behaviour of the Chrome application made me, a long time user of Tweetdeck, feel embarrassed and angry to the extent that I stopped using that version of their product. No matter whether the behaviour was technically correct and conformed to the target design and use cases for that versions, to me personally it was a bug.

A bug is in the eye of the beholder


What constitutes a bug can be one of the most contentious discussions in software development, causing particular stress for relationships such as between tester and programmer, and supplier and customer. When it comes to straightforward programming errors then the definition will hopefully be clear and unequivocal. When we consider a bug in terms of the personal feelings and opinions that it elicits then the waters of what constitutes a bug are a lot more murky.

I would not personally limit the definition of a bug solely to the terms of a coding defect. To me, a bug is a behaviour which elicits a negative emotional response in a positive stakeholder (I use positive stakeholder to exclude those who have a negative relationship with a product, such as a competitor who's emotional response is likely to be inversely propoprtional to our successful behaviours). Once the personal emotional influence is acknowledged then the decisions on what to fix and what behaviours to change become human ones rather than technical ones, raising some interesting implications:-
  • The decisions around fixing the issue can be defined not around technical severity but on how many people will be impacted by the behaviour and how bad will it make them feel. My problem with the links in Tweetdeck might not consitute a bug on the basis of my feelings, however if half the user community felt the same way then a change in behaviour would certainly be justified. If one marathon runner had had their details published on the internet this week, I doubt that the story would have merited the BBC News coverage that it actually received.

  • Two users of the same system may have mutually exclusive opinions on how that product should behave, such that a fix for one will constitute a bug for another. I've certainly encountered this scenario when attending user groups for software products in use by a number of different organisations with contrasting business workflows. In Microsoft Outlook, the fact that the meeting reminders took focus from other applications was a sufficient problem for Microsoft to remove this. For me, and many others, the fact that this prevents reminders from popping up is a major problem, as an icon on my taskbar turning orange is not sufficient to distract me from what I am doing and remind me that I have a meeting to attend. Both behaviours constitute bugs to certain individuals resulting in frustration and anger, and in my case installing another application (the excellent Growl for Windows) to workaround the issue.

  • The impact of the bug may not occur now, but at some point in the future, even after the application has ceased to operate. If the data is ported to another application. I saw a system a few years ago which defined a primary key on its non-normalised transactions table based on forename, surname and DOB - obviously not a good basis for a primary key. The system had not encountered an issue with this up until that point but the nature of the system and the data could have had very serious negative consequences on individuals in the future. Working in the data storage and analysis space I'm all too aware of the frustration that is caused when trying to transfer badly validated data from one system to another.

  • The emotional impact of a bug may not be caused directly by the software itself, to the extent that the victim of the bug are unaware that their emotions are the result of a software behaviour. If a security bug, such as the London Marathon one, releases someones details resulting in fraudulent or undesirable results for that individual then they might be totally unaware of where their details were obtained.

  • A bug that is very hard to justify in any quantifiable terms, might be much more easily justified in emotional ones. Whenever I use the ticketing machines in UK train stations I always feel that they should be faster. I have no supporting evidence for this other than that I see the same emotions in all those who try to use them - frustration, impatience and anger that it takes so long to achieve what should be a simple operation. Attempting to quantify this in terms of transaction times makes it hard to justify a change. When, as I saw today, this results in individuals missing their trains then the emotional justification can be much more powerful.

  • A coding mistake can exist in a product for years and not consitute a bug, as long as no-one at any point, either now or in the future, has a desire that it should be/should have been changed.

  • No matter how much testing you do to ensure you have requirements traceability and functional correctness in relation to the specification, your product can still be full of bugs if the users hate the way that it works.

So I've moved back to using the Tweetdeck desktop application for now. It does mean an extra application running on my laption but at least I can copy and paste my links with impunity. As to whether my problem was caused by user error or was actually a bug - I'll leave that for you to decide - bugs are in the eye of the beholder after all.
Rob Lambert said...

Hi Adam,

Great post as usual. I once tested a product where the log in screen had a bug which meant each person logging in had to hit the log in button (or return key) twice to log in.

It had been in there for years. So long in fact that no-one ever reported it anymore. The internal users documentation even included this extra click or step in the guide to the product.

So when a product owner made the decision to finally fix this bug we all rejoiced at finally getting rid of the annoying "log in bug" only to find that it really annoyed our customers. They'd worked around it, taught and trained around it and ultimately got used to it that they wanted the "bug" back.

I like your list and your right, it really is a contentious issue for many testers. Really enjoyed reading this post.

Thanks
Rob

Andreas said...

Adam,

First off: enjoyed your post. It reminded me of a (true) story I once heard that was an eye-opener about my work.

The test team, and the business unit both - in a final meeting of some sort - had to list the top 3 of "best things" AND "worst things" about the application/iteration/functionality that was finished.

Turned out that the lists were far apart (not that they were inverted, just that nothing that was on either list of testing was on the business's lists), since they differed on every aspect. What testers found annoying (because they had spent testing - and fretting over - that part rigorously) was not what annoyed the end users, and vice versa. The same applied for the 'good' features. At first, I was flabbergasted - oh, how I enjoy typing that word - because I could not for the life of me understand why they found functionality A okay while Y was obviously a train wreck. Then I came to a similar conclusion as you did.

Cheers,

Andreas

Adam Knight said...

Andreas,

Thanks for commenting, I'm glad that you liked the post. That is a great story - not one that surprises me though. I've found myself surprised in the past about the seemingly small things that will cause the greatest emotion and the apparently more severe issues that end up upsetting no-one. There are far more obvious functional bugs in the tweetdeck application than the behaviour that affected me, yet is was this one that I consider the worst as it caused me the strongest personal emotion, that of embarrassment.

Cheers,

Adam

Adam Knight said...

Thanks Rob - that is a great example!

It never ceases to amaze me the disparity between the technical severity of bugs and the level of emotion that they cause. In previous jobs I've had users, whilst demonstrating an apparently minor cosmetic bug that is making their lives a misery, gloss over application crashes saying "oh it always does that" as if it was expected software behaviour! As with your example - it just goes to show that you can't predict the human impact of issues based on technical severity alone and why trying to understand your customers is such a key element of the testing role.

Thanks for taking the time to comment.

Adam.

Whatsapp Button works on Mobile Device only

Start typing and press Enter to search