Wednesday, 8 April 2015

It's In the Documentation - We're Covered

In my last post "Is it a defect? No but it is a flaw" I examined some of the terminology used in defining defects across our contractual interfaces. I explored in that post some interesting behaviours have arisen as a consequence of this kind of definition. In this post I'll examine what, for me, are some of the most interesting consequences which relate to treatment of the technical documentation of our products.

Let's review a typical statement from a customer Service Level Agreement (SLA).

Defect : a deviation from the documented behaviour

A logical conclusion from this statement is that the presence of a defect in the system is defined on the basis of inconsistencies between product behaviour and product documentation. The relationship between documentation and product therefore moves from one of support and instruction to one of contractual importance. In essence what the SLA wording is doing here is elevating the technical documentation to be the pivotal point on which discussions of customer problems will hinge. The consequences of this are profound, if sometimes amusing.

  • I have had customer support tickets raised with customers asking specifically whether a behaviour is documented. Not whether it was intended, or desirable, but documented. When we replied that it was they agreed to close the ticket - we asked but never found out what their issue was with it.

  • In an extreme case , one company I worked for actually removed all of the documentation, including help, from the application because of a customer contract based on adherence to the docs. They didn't want to run the risk of the out of date documentation being used as leverage against them so they simply removed it all.

It's not in the documentation - we're stuffed

Just as it is impossible to test every possible combination of behaviours in a piece of software, it is equally impossible to document all of the behaviours of a software system of any complexity. This inevitably leads to a set of characteristics of our software that are not documented. This disparity between documented and actual product behaviour constitutes something of an awkward 'grey area' when the contractual SLAs are considered and can make support discussions difficult. How do we define whether or not the system is adhering to the documentation when the behaviour in question is not covered in the documentation?

On the face of it I can understand the desire to tie the software contractually to do what we claim it will, however in practice the way that software is documented, sold and delivered means that this approach is far from ideal.

  • The documentation is delivered with the software.

    For most software the provision of technical documentation is part of the software release process. I've rarely encountered a customer who insists on rigorously reviewing the documentation to check for inconsistencies with their needs before the software is actually delivered.

  • Documentation is referred to on a task basis

    Technical guides are rarely read from cover to cover before people start using the product. Far more often the software will be taken first and the documentation referred to at the point that the user needs help with achieving a specific task. One of the biggest challenges in documenting a big data system, where it is crucial to consider data design up front, is to ensure that the relevant documents are read in time to influence decisions on tasks that affect the data structure, and not when the problems have been encountered.

  • Documentation is not designed to legally capture the system behaviour.

    Good documentation is designed to help the user to achieve their goals. As I've learned from my team and from reading some of the excellent technical documentation blogs out there, such as 'Every page is page one', there is a huge amount of thought and skill in designing documentation to help users achieve their goals by providing them with the relevant information needed to help with each task. The primary purpose of technical documentation is not as a legally worded contract exhaustively detailing every nuance of our software to use as reference over service contract negotiations. It is there, as it should be, to improve the experience of users and stakeholders in the software.

Its in - we're covered

Whilst it is not possible to document every aspect of system behaviour, the importance of the documentation in defining defects can all lead to some adverse practices in our attempts to try.

When a customer raises questions over behaviour this can often result in a hurried search to establish whether it is covered in the product guides. The ensuing sigh of relief is tangible when we discover that it is included and we are able to refer the questioner to the appropriate guide covering the behaviour in question. It is much harder to argue the case that a feature is incorrect if it is clearly stated in the product guides that it works in that way. A system may be riddled with flaws, but if they are documented then we can argue that they are not defects according to our service agreements.

I know from personal experience how easy it is to fall into a pattern of gamification of documentation because of this. It is hard not to strive for a false goal of getting all of our adverse behaviours documented somewhere as release deadlines approach. Whilst there may be intentions behind this - helping the users to avoid pitfalls - we'd be kidding ourselves if we pretended that there wasn't also some element of self preservation involved. The release note in particular can end up as the documentation equivalent of the safety net, where last minute limitations are slotted in to cover ourselves for problems that are too tricky to address in the time remaining.

What not to do

It is a common mistake to introduce negative behaviours into a process by measuring something on the basis of it being easy. Whether measuring testers based on bug counts, or developers on lines of code, it is incredibly easy to encourage activities that focus on achieving measurable quotas at the expense of achieving overall goals. In the same way, placing contractual emphasis on the documentation is in my opinion inappropriate and detracts from its primary purpose.

Whilst I don't claim to have a clearly defined alternative approach, it seems clear to me that this is not the way to specify defects. I'm sure that technical authors everywhere, would breathe a collective sigh of relief if we find another way of defining our defects and free up the documentation to what it should be for - guiding users through understanding their options and achieving their tasks when using our software.


There are some great technical documentation blogs out there. The ones that I read most regularly are:

Monday, 16 March 2015

Is it a Defect? No, but it is a Flaw.

Over the last few weeks I've had cause to spend some time thinking about the nature of software problems and the language that we use to define them.

The driving reason for this is that I've been looking over some old customer support service level agreements (SLAs) as part of our transition of ownership. Occasionally reviewing something familiar for a different purpose can provide new insights into the content of it. As I read the SLAs I was struck by the absolute nature of the terminology used in relation to issues encountered. To paraphrase one such agreement with a former customer:

  • Defect: a deviation from the documented product behaviour
  • Bug: See Defect.

Such statements are commonplace in contractual agreements , both within and between organisations. The text has clearly been structured for the purposes of avoiding ambiguity, and on the face of it this has been achieved. While the terms appear straightforward and unambiguous, the actual wording contains some interesting and possibly concerning implications for software delivery.

Over the next couple of posts I'm going to examine the concept of defects and their definition in contractual type agreeements in light of experiences in testing, customer support and documentation, to see how our behaviour in the field of software has evolved some interesting and adverse characteristics in response to these definitions.

What's in a Name?

The term 'defect' bothers me. The concept of a defect in terms of service level agreements such as the one quoted above have an incredibly narrow scope of applicability. As we can see in the above example, the definition relates specifically to a deviation from the documented behaviour. When compared with the range of ways that I have encountered that negative relationships can arise between software and customer the definition of a defect is restrictive. It simply does not cover the range of problem types that can occur in software use that I encounter on a daily basis and can impact on our contractual relationships.

The concept of a problem being defined as a deviation from the documented design leaves a wealth of potential problems undefined within the scope of such agreements:

  • the design fails to consider some valid scenarios of use
  • a performance issue whereby the software does not meet the customer expectation
  • behaviour that is not explicitly referred to in the documentation
  • an unintuitive interface
  • an inconsistency with expectations based on related products or technologies

All of these things can potentially impact customer perception, yet sit outside of the definition of a defect in the terms of many SLAs.

From my experience running the technical support team of a data storage system I can honestly say that a significant proportion, if not a majority, of tickets that are raised by customers do not relate to defects per-se but rather areas of behaviour, such as the above, that are far more ambiguously defined. My policy historically in these matters has been to strive towards delivering excellent customer service over quibbling around the definition of defects. Even so there are cases where we have to manage the customer expectation due to their specific query shape not having been optimised over tables containing hundreds of billions of rows of data, which can sometimes be difficult.

Even when adopting a more flexible stance, the expectation and even preference from many of our customers can be on a rigid approach due to the fact that that is what their processes are set up for. The rigidity provides the impression of predictability - a desirable characteristic of the support service.

It is this exact rigidity, however, that gives the greatest scope for conflict when problems such as those I mention above arise. At that point it is entirely dependent on the supplier whether to take a flexible stance and provide the assistance the customer needs, or whether to state that the software is behaving as designed, quote the SLA agreement and close their ears.

An internal problem

A rigid definition of what constitutes a defect can also affect internal relationships such as that between developer and tester. There was a time when a significant portion of my working day was spent raising bugs during a waterfall testing phase and seeing them fired back in the bug tracker with the comment 'as designed' added by the programmer.

This "bug tennis over a waterfall" situation can be a soul destroying one for testers, when their efforts in raising genuine concerns over the product are met with the brick wall of a blank refusal to question the design. I'm fortunate, as are many folks working in collaborative teams, to not encounter this situation, but I've encountered it first hand in other roles. Based on prior experience I can understand the stereotypical tester complaint of not having complete specifications up front. When the concept of a defect is defined in relation to adherence to specification then without it, it can be difficult to justify the existence of a problem, as the tester simply can't prove the presence of a 'defect' in absence of a conflicting specification.

When I first introduced group exploratory testing on a middle tier data framework in 2004 the outcome was a collection of new defects but also a series of concerns over the implemented design. We fed these back to the product management team but with no mechanism to get the design changed these concerns went no further. These included issues such as a major inconsistency between the technical knowledge required to configure the user connectivity and the knowledge of the majority of the users.

Thankfully more of us are finding ourselves working in more collaborative teams, Agile or otherwise, where within the team at least there is less need for rigid Defect definitions. In my team the term 'bug' has historically been sufficient to cover all manner of potential problems from design through implemented behaviour and documentation. A bug is as much a placeholder for work as it is a defect in the system. How that approach will need to change as our bug processing comes under the scrutiny of a more traditional organisation structure remains to be seen.

Wherefore art thou defect?

Given the range of situations in which I communicate, even with a fairly informal intra-team definition of bug, I still find the need to be careful with terminology when communicating in other forums. As we can see in the SLA definitions above, 'bug' is synonymous with 'defect' in many contexts, including SLA agreements, and also in the ISQTB glossary of terms. Some, however, define a bug as a superset of defects, while for others it relates to a different entity entirely or an entity in a different stage in the software lifecycle.

Even if we limit ourselves to examination of just defects, what I have observed is that the definitions even within the software community, are inconsistent, sometimes dangerously so.

We've already seen that, in technical support context, a defect can be defined as a difference between behaviour and documentation. This also translates into software testing context, where a defect can be simplistically defined as a difference between actual and expected of a test - for example in testingmentor - a software testing website

While testing when a tester executes the test cases he might observe that the actual test results do not match from the expected results. The variation in the expected and actual results is known as defects.

ISQTB has the following :

defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.

The IT Law Wiki has an interesting definition - not least for its brevity considering the unquestionable importance of defects to software law.

[a] problem or “bug” that if not removed, could cause a program to either produce erroneous results or otherwise fail.[1]

(As an aside - a positive element for me both of these definitions is that they reference the potential for the problem to occur - a key characteristic of software problems that I wrote about in this post.)

This following definition, a popular one in many websites, does consider the possibility that a failure to meet user expectation constitutes a defect, however it does so in the limited context of incorrect or unexpected results.

A Software Defect / Bug is a condition in a software product which does not meet a software requirement (as stated in the requirement specifications) or end-user expectations (which may not be specified but are reasonable). In other words, a defect is an error in coding or logic that causes a program to malfunction or to produce incorrect/unexpected results.

With the variation in these, and many other, definitions it is not suprising that there is some confusion. Witness this conversation on stack exchange, originally asking about the difference between a defect and a bug, for examples of the confusion that supposedly rigid definitions of terms an cause.

Not a Defect, but a Flaw

In response to these definition problems I've found the word 'flaw' to be an extremely useful one. I find myself increasingly using it in situations where defect is overloaded and may have undesirable connotations.

Although the terms defect and flaw have similar definitions they carry very different meanings when applied. The beauty of the word flaw for me is that it has layers and nuances of meaning. It is this very flexibility and lack of formal definition which provides value when tip-toeing over the eggshells of behavioural conversation. Even the Oxford English definition provides three applications:

  • a physical flaw
  • a personality flaw
  • a flaw in a plan or design.

Picture what comes to mind if I described a car as having a defect, as compared to a car having a flaw. To me the former is a fault in a specific vehicle, a badly moulded or cracked component perhaps. The latter could as easily conjure up images of a general problem in the design which affects all cars of the model, from the major design flaw such as the fuel tank problems of the Ford Pinto through to the minor niggles that fill the pages of this discussion - Minor flaws in the cars you love - on piston heads.

The key for me here is that both flaws and defects can be defined in relation to an ideal, a model of perfection. With flaws, however, this ideal does not have to be achievable, and in some cases it is expected that it will not be so. I particularly like, for example, the parallels between the application of the term 'flaw' to software and to people. To say someone had a defect brings to mind physical defects such as heart defects and other conditions. To say someone is flawed carries a very different connotation. Even the most accomplished individuals in all fields, whether politicians, athletes, business people or any walks of life, have flaws. This doesn't mean that they need medical treatment or surgery - it means that they are not perfect, however this is in the context of perfection being neither expected nor attainable. Everyone has flaws, everyone's are different. People may work over time to improve on their flaws, but not doing so doesn't stop them from being successful, wonderful people.

I like to think of software in the same way. The principle of the defect has at its heart the unrealistic notion that perfection is achievable if we could simply find and remove all the defects. There is no better deconstruction of this notion than Jerry Weinberg's excellent - "Perfect Software and Other Illusions About Testing". Even though most acknowledge that it is not possible, the implication in the majority of defect definitions is still there that the only thing between us and software perfection is our inability to find and fix all of the defects.

A Flawed Idea

This principle that the defects are all that separates a piece of software from perfection is one I think we need to move away from. All software is flawed just as people are. Software evolves through human endeavour. Throughout its creation it will have been subject to innumerable human decisions over workarounds, limitations and compromises. Even applications targeted at the same job by different teams will have areas of relative strength and weakness irrespective of their defects. Some designs that work well in some instances may even constitute flaws in others - the phrase 'one of their greatest strengths was also their greatest flaw' is sometimes used to describe people but could equally apply to software, or products of any kind. The simplicity in running one piece of software, for example, may be a flaw in one situation where greater configuration flexibility is required, whereas the rich feature set of a competitor may constitute unnecessary complexity in another.

I tend to use flaw particularly when discussing product behaviours that work as originally designed, but when the design has some limitations of disadvantages in applicability in certain scenarios. This may be a known limitation in capability of running certain shapes of query, a relative weakness in support of one data type over another, or simply a process that is more complicated than it needs to be. The phrase 'one of the flaws of the system is that it doesn't handle storage of large binary objects as well as other data types' carries a whole different meaning if I swap the word defect in there.

If I examine our bug database, what is clear to me is the behaviours captured within it are not just defects, they are flaws. This is a good sign as it means that the testers are looking beyond simple programmatical errors to identify a range of potential problems. For example recently a bug was fixed to return a useful output in our diagnostic capture process. It wasn't a requirement of the original command, it was simply that on using the feature it didn't make a lot of sense not to have it there. Now you could argue for days over whether this was actually a new user story, or a defect through a missed requirement, however this simply reinforces my point. If we concern ourselves too rigidly with sticking to work that conforms to narrow definitions, then some useful and important might be ignored simply through the inability to define them as defects or justify them as new requirements, when in reality they are neither, they are flaws. I've been involved in feature discussions with customers where obvious and useful changes are ignored at the expense of trivial cosmetic changes, as the latter can more easily be categorised as defects and therefore fixes justified. It is saddening to see chances to improve products being overlooked simply because our categorisation of what constitutes a problem in the system is simply too narrow to include them.

It's in the Documentation - We're Covered!

In my next post I'll go on to discuss another worrying implication of defects being a deviation from documentation ...


Many thanks to Shilpa J at for the excellent cartoon - feel free to get in touch with me if you have any ideas on testing cartoons that you'd like - Shilpa is looking for inspiration!

Wikipedia - David - for many Michaelangelo's Daid was flawed - see for some interesting criticisms of the 'flaws' in the statue from its hands to its nose. Few that have seen it (which I am fortunate to have done) would argue, however, that it is a masterpiece.


Wednesday, 4 February 2015

The Evolution of Tools

I mentioned in my last post some exciting conversations that I had in the pipeline with the testing team of my new owners. I had one such conversation last week, in a meeting where some of the team demonstrated a range of testing tools that they had developed in house that were available for us to use.

Over the course of a one hour Web Meeting two very pleasant chaps from San-Diego talked us through three of the tools that they had developed in house for database testing tasks. Particularly interesting for me was the fact that, for a couple of these tools, we had equivalent in-house developed tools and I was interested to look at the similarities and differences between their tools and our equivalent ones.

  • SQL Generation

    I spent a chunk of my time last year looking into random SQL data generation. As a good excuse to teach myself Ruby at the same time, I developed a prototype SQL generation utility that could produce a range of SQL syntaxes based on a model defined in input files. I was therefore particularly interested in looking at SQL generation tools that the team had developed. As the San-Diego team talked through their utility it was clear that, predictably, their tool was at a much higher level of maturity than mine. At the same time, I started to notice some very familiar structures in the configuration files. The way that functions were defined, the parameterisation of specific variables, the definition of data schema all bore close resemblance to the ones in the prototype that I had worked on. I recognised structures which reflected what I had done myself to solve the problems that I'd faced in attempting to generate correct, relevant SQL queries.

  • Test harness

    As we moved on to discussing a more generic test harness the parallels to our own testing harness were even more apparent. The way that workflows, parallel scheduling and multi-machine tasks were defined all had direct parallels in our in-house harnesses. There were some differences in the mechanisms used, however the underlying structural elements were all present and the nature of the solutions overlapped extensively. The location and means of definition of test data, metadata and scheduling mechanisms across servers were similar solutions to challenges that we'd tackled in our own harness as our product evolved in line with our product over the past few years. I found myself standing at the screen gesturing to my colleagues the elements of the harness structure which could be mapped directly to our own.

Convergent evolution

My conference talk at EuroSTAR 2011 is testament to the fact that I am fond of applying evolutionary principles to the development of tools and processes. The fact that these very similar tools had been developed by independent teams in separate organisations reminded me of the idea of homoplacy - a process sometimes known as convergent evolution, in which separate species evolve highly similar characteristics not present in a common ancestor, to fill the same evolutionary roles.

For example many Australian marsupials bear a remarkable resemblance to mammals from other parts of the world despite having evolved completely independently.

There are many other examples, such as the similarities between Cactus and euphorbia, and Mantidfly and Mantis, whereby evolution creates the same or similar solutions to environmental problems.

Much as with evolution in nature, tools and processes evolve based on needs that arise within organisations. Working in similar product markets, it is understandable that our tools have developed independently to meet the same needs. What struck me when comparing my tools with those being demonstrated was the extent of the similarities - there were many familiar structures that had clearly been implemented to address the same challenges, within the context of a specific testing problem, that I or my team had previously encountered.

Time wasted?

The examples here are representative of what I have seen elsewhere - I imagine that it must be a common phenomenon. The 'convergent evolution' of tools must be going on throughout development organisations around the world. Taking SQL generation as an example, through my research on the subject I know that Microsoft, DB2 and MySQL have all developed similar utilities.

One obvious question that arises here is 'is this time wasted?'. If parallel efforts are going towards creating equivalent tools then would it not make sense to collaborate on these? Surely it would be more efficient if these teams combined their resources to create common tools rather than duplicating effort? Or perhaps that's a rather naive perspective.

  • Tools are valuable assets. Microsoft at least have never released their RAGS tool outside of their organisation. I suppose that the theory here is that releasing the tool would provide an advantage to others developing competitive products who haven't invested the time to developing them. A good tool can provide a competitive edge in just the same way as a well designed feature set or a talented team. For an open standard such as SQL, Microsoft have little incentive to release tools that allow others to develop competitive products. By way of an interesting comparison, for Microsoft's own standard of ODBC, they have made tools freely available - perhaps the improved adoption of their own standards provides sufficient incentive to provide free tools in this situation.

  • Joint ventures are prone to failure. As part of my higher education I studied joint ventures between organisations, with a focus on examining the high failure rate of this kind of collaborative project between companies. Based on the examples looked at, they rarely deliver to the satisfaction of both contributors. Some reasons for potential failure include an inability to relinquish full control of the project, and an imbalance between the level of input and the benefit gained between the contributing parties. When it comes to the relatively small and well encapsulated nature of tool development, it makes more sense to take this on in-house to maintain ownership and avoid the pitfalls of joint development.

  • Ownership and control are powerful motivators. Another option for avoiding parallel efforts on tools is if an organisation develops and maintains a tool commercially for others to licence and use. Whilst SQL generation may not have sufficient commercial application to be attractive here, I've seen many scenarios where teams develop their own tools even when there are good commercial alternatives. The incentive behind such an approach can be that a simpler toolset is required and so the cost of expensive commercial options is no justified. Another likely reason is that having control of your tools is a compelling advantage, it certainly is for me. One of the main reasons for developing our own tools has been the knowledge that we can quickly add in new features or interfaces as the need arises, rather than waiting for a commercial supplier with conflicting priorities.

A positive outcome

I came away from the session feeling rather buoyant. One one hand I was pleased that in-house tools development received such a strong focus in my new organisation. The team that we spoke to were clearly very proud of the tools that they had developed, and rightly so. A strong culture of developing the appropriate tools to assist your testing, rather than trying to shoe-horn your testing to fit standard testing tools, is an approach that I have believed in for some time. Whilst the approaches taken between the two teams was somewhat different, what was clear was that we shared a common belief that having the ability and flexibility to develop your own tools was a powerful testing asset.

More importantly, I felt proud of what my team and I had achieved with our tools. With a fraction of the manpower of our new owners we had evolved tools which could be compared favourably with those of a much larger and well resourced organisation. Whilst there was clearly some overlap such that moving to the some of the new tools would make sense over continued development of our own, in some cases the strength of our own tools meant that this was not a foregone conclusion. As I wrote about in my post on the Facebook Effect - we're never quite sure how our work stands up in the market given the positive angle that most folks come from when discussing their work in social media. Given this chance of a more honest and open comparison, I was justifiably pleased with how well the work of my team compared.