Tuesday, 28 April 2015

A Cultural Fit

Ealier this month I attended the third MEWT (Midlands Exploratory Workshop in Testing) Organised by Richard @FriendlyTester Bradshaw, Bill Matthews (@bill_matthews), Simon Knight (@sjpknight) and Facilitated by Vernon @TesterFromLeic Richards - the day brought together 16 testers of various backgrounds and experience working across the Midlands to share talks and discuss ideas in an open environment of trust. As anyone who has read The Facebook Effect will know that I place a huge value on groups such as this where we can share ideas without fear of exposing our faults for public scrutiny. The day did not disappoint - the group dynamic was fantastic. The theme of the day was communication and we discussed a range of subjects all with the common thread of communication, sometimes to a very deep and personal level.

One of the topics that came up during Daniel @TheTestDoctor Billing's hearfelt talk on geeks and personality types, was hiring. It was mentioned that many folks, Rob Lambert at New Voice Media and myself included, have a strong focus on cultural fit when we are hiring. Anna Baik (@testerab) made an salient point speaking from her perspective of being a woman in tech, that we need to be clear when hiring for cultural fit that we're not simply hiring to match a stereotype and using cultural fit as an excuse. Common interests in beer and football do not constitute 'cultural fit'.

I tweeted this sentiment

but as James Bach (@jamesmarcusbach) rightly pointed out - the tweet lacked the group context, and Anne-Marie Charrett (@Charrett) suggested is important enough to expand upon so I'll attempt to do so here.

I am happy to say that I place a strong emphasis on cultural fit when I am hiring, When I say this I have absolute clarity on what I mean, and what I don't mean. Anna's comments provided an excellent reminder that others may interpret this in a completely different light such that one person's cultural fit may be another's excuse to restrict their hiring to favour certain individuals, men, or people in a certain age bracket, for example. Let me make it as clear as I can.

What cultural fit is

When I say that I place a strong emphasis on cultural fit when hiring - I mean that I look for people who, in joining my company will not undermine the work culture that we have worked to establish. Our cultural principles are the cornerstone of our successful ongoing product development and we work very hard to ensure that new team members will contribute to and build on them. Some of the characteristics that I will look for that I think are key to this are

  • A desire to collaborate and a focus on the importance of team working
  • Tactful appreciation of the importance of a focussed working environment without too many distractions
  • A willingness to take on a variety of tasks if necessary to achieve the teams goals
  • A focus on personal skill and expertise over adherence to standards or certification
  • A focus on delivering working software over delivery of a successful process
  • An appreciation of the individuals filling the range of roles required to produce software as having an equal contribution to that overall effort
  • A focus on solving problems over apportioning blame under the guise of establishing root cause

Some of these are rooted in our agile approach, some are more distinct to our own style of working. Not all of these things would necessarily form the cultural basis of all sucessful teams. Each team is different, but in general our office tends to be quiet with high levels of focussed working - other offices may rely more on camaraderie and a 'buzz' of activity to maintain their team dynamic. I once attended an excellent talk by Tom Howlett describing how he achieved great success with a completely virtual team formed of highly focussed individuals all working from home. I would expect the dynamic of that team and the resulting recruitment focus could be very different to mine and still deliver great success.

What I do believe is that, for whatever team dynamic we are aiming for, it is possible to recruit a high variety of people in terms of personal interests, gender, age and background whilst still focusing on preserving your cultural ideals.

What cultural fit is not

Years ago, when I was working for a different company, I was helping my manager in recruiting. At the end of an interview we were discussing the candidate - he asked me whether I thought that the candidate might not be 'a good fit'. Without him saying it, or even possibly admitting it to himself, I knew what he was implying. We had a team of white men and he was worried about how the fact that the candidate was a woman of Indian origin would affect the existing team. Our team was a comfortable enclave of white maleness who discussed football on Monday mornings and went to the pub on Friday lunchtimes. My response was clear - we would not consider the candidates gender or ethnicity as a basis for their suitability for the role, because

  1. As far as I was aware it was illegal and
  2. I was not prepared to work for a company that did so and would have to leave

The manager's comments demonstrate how easy it is to confuse cultural fit with hiring to stereotype. The implication was that someone from a background that was a closer match to the existing team would constitute a 'better fit' with the team. As it happened on that occasion I did not feel that the lady was the best candidate, mainly due to what I saw as our inability to support and train her at a very early stage in her career. I stated my belief that we would be letting her down by hiring her, as we would have been for anyone at a similar stage in their career, as our culture was simply not supportive enough at the time. ( It goes without saying that the manager was totally wrong in their concern as, after taking responsibility for hiring, I went on to build out the team with a mix of genders and backgrounds and the team dynamic only grew in strength as a result. )

Our team habits and common interests are nothing to with our work culture as I mean it when I target 'cultural fit'. Our culture was that we were a competent, collaborative and highly professional small team who used daily face to face communication to self organise. If hiring a new candidate meant that we had to adapt our team or personal habits slightly to welcome and accommodate them as a member of that team and that culture, then that was our responsibility to do so.

Hiring for cultural fit is not about finding people who conform to a stereotype that matches the people that already work in your organisation to improve the likelihood of them 'fitting in'. It is about finding people who want to work in a style that is consistent with the values of your team, however those values should not be such that they exclude capable candidates based on their gender, age or background.

It is hard

Hiring with a focus on a good cultural fit is difficult. One reason for this that I've encountered is that it is much harder for recruiters to understand a set of cultural ideals than it is a set of technical skills and certifications. Even the good recruiters that I work with will naturally focus on technical skills that allow easy CV searching over finding someone with the personality to integrate well with our team. I can picture the frustration on the other end of the email or phone when a candidate that looks great in terms of technical fit is rejected because I think that they will disrupt our team dynamic.

You could argue that I and others like me are being too fussy in focusing so much on the 'softer' skills. After all at the end of the day as long as someone can get the job done then surely that is enough. For some organisations that may be the case, however I have seen the effects that bringing someone who is a poor fit into a team can have. I have seen a team which once enjoyed open and professional discussion withdraw into themselves because of the new recruit who takes a slightly too aggressive approach on getting their point across. I've seen the previously confident tester grimacing as they have to deal with the new programmer who, despite the false asertions to the contrary, doesn't really respect the testing role and treats testers as second class citizens.

You could also argue that perhaps our team culture is such that it naturally excludes individuals from certain backgrounds. This is a difficult one as certainly the values in some organisations will run contrary to those that we operate on. Given that many organisations operate across regional and national borders I think this is more an issue of experience and history rather than background. I take the approach of clearly explaining in the interview process what makes us tick as a team, and how that may be frustrating if accustomed to a different way of working. If a candidate is happy to accept these differences, can appreciate the potential benefits and engross themselves in a new environment then I'm more than happy to have them on board.

On discussing the subject with Rob Lambert he made a great observation that perhaps it is the use of the phrase 'cultural fit' that could be overloaded.

"We've started saying "team values" now - as the word cultural tends to mean "ethnicity" or "gender" to many people - and this is not what we mean."

I think this is a great point and I'll probably try to use 'values' rather than 'culture' from now on to avoid potential for misinterpretation. For me, and I'm sure many others, hiring consistently with our team values is a critical element in maintaining the team dynamics that allow us to deliver great work in a supportive environment. What it is not is an excuse to target our hiring around a stereotypical norm at the exclusion of the wealth of individuals from a variety of backgrounds who would augment and enrich our culture if they are simply given the opportunity.

Wednesday, 8 April 2015

It's In the Documentation - We're Covered

In my last post "Is it a defect? No but it is a flaw" I examined some of the terminology used in defining defects across our contractual interfaces. I explored in that post some interesting behaviours have arisen as a consequence of this kind of definition. In this post I'll examine what, for me, are some of the most interesting consequences which relate to treatment of the technical documentation of our products.

Let's review a typical statement from a customer Service Level Agreement (SLA).

Defect : a deviation from the documented behaviour

A logical conclusion from this statement is that the presence of a defect in the system is defined on the basis of inconsistencies between product behaviour and product documentation. The relationship between documentation and product therefore moves from one of support and instruction to one of contractual importance. In essence what the SLA wording is doing here is elevating the technical documentation to be the pivotal point on which discussions of customer problems will hinge. The consequences of this are profound, if sometimes amusing.

  • I have had customer support tickets raised with customers asking specifically whether a behaviour is documented. Not whether it was intended, or desirable, but documented. When we replied that it was they agreed to close the ticket - we asked but never found out what their issue was with it.

  • In an extreme case , one company I worked for actually removed all of the documentation, including help, from the application because of a customer contract based on adherence to the docs. They didn't want to run the risk of the out of date documentation being used as leverage against them so they simply removed it all.

It's not in the documentation - we're stuffed

Just as it is impossible to test every possible combination of behaviours in a piece of software, it is equally impossible to document all of the behaviours of a software system of any complexity. This inevitably leads to a set of characteristics of our software that are not documented. This disparity between documented and actual product behaviour constitutes something of an awkward 'grey area' when the contractual SLAs are considered and can make support discussions difficult. How do we define whether or not the system is adhering to the documentation when the behaviour in question is not covered in the documentation?

On the face of it I can understand the desire to tie the software contractually to do what we claim it will, however in practice the way that software is documented, sold and delivered means that this approach is far from ideal.

  • The documentation is delivered with the software.

    For most software the provision of technical documentation is part of the software release process. I've rarely encountered a customer who insists on rigorously reviewing the documentation to check for inconsistencies with their needs before the software is actually delivered.

  • Documentation is referred to on a task basis

    Technical guides are rarely read from cover to cover before people start using the product. Far more often the software will be taken first and the documentation referred to at the point that the user needs help with achieving a specific task. One of the biggest challenges in documenting a big data system, where it is crucial to consider data design up front, is to ensure that the relevant documents are read in time to influence decisions on tasks that affect the data structure, and not when the problems have been encountered.

  • Documentation is not designed to legally capture the system behaviour.

    Good documentation is designed to help the user to achieve their goals. As I've learned from my team and from reading some of the excellent technical documentation blogs out there, such as 'Every page is page one', there is a huge amount of thought and skill in designing documentation to help users achieve their goals by providing them with the relevant information needed to help with each task. The primary purpose of technical documentation is not as a legally worded contract exhaustively detailing every nuance of our software to use as reference over service contract negotiations. It is there, as it should be, to improve the experience of users and stakeholders in the software.

Its in - we're covered

Whilst it is not possible to document every aspect of system behaviour, the importance of the documentation in defining defects can all lead to some adverse practices in our attempts to try.

When a customer raises questions over behaviour this can often result in a hurried search to establish whether it is covered in the product guides. The ensuing sigh of relief is tangible when we discover that it is included and we are able to refer the questioner to the appropriate guide covering the behaviour in question. It is much harder to argue the case that a feature is incorrect if it is clearly stated in the product guides that it works in that way. A system may be riddled with flaws, but if they are documented then we can argue that they are not defects according to our service agreements.

I know from personal experience how easy it is to fall into a pattern of gamification of documentation because of this. It is hard not to strive for a false goal of getting all of our adverse behaviours documented somewhere as release deadlines approach. Whilst there may be intentions behind this - helping the users to avoid pitfalls - we'd be kidding ourselves if we pretended that there wasn't also some element of self preservation involved. The release note in particular can end up as the documentation equivalent of the safety net, where last minute limitations are slotted in to cover ourselves for problems that are too tricky to address in the time remaining.

What not to do

It is a common mistake to introduce negative behaviours into a process by measuring something on the basis of it being easy. Whether measuring testers based on bug counts, or developers on lines of code, it is incredibly easy to encourage activities that focus on achieving measurable quotas at the expense of achieving overall goals. In the same way, placing contractual emphasis on the documentation is in my opinion inappropriate and detracts from its primary purpose.

Whilst I don't claim to have a clearly defined alternative approach, it seems clear to me that this is not the way to specify defects. I'm sure that technical authors everywhere, would breathe a collective sigh of relief if we find another way of defining our defects and free up the documentation to what it should be for - guiding users through understanding their options and achieving their tasks when using our software.


There are some great technical documentation blogs out there. The ones that I read most regularly are:

Monday, 16 March 2015

Is it a Defect? No, but it is a Flaw.

Over the last few weeks I've had cause to spend some time thinking about the nature of software problems and the language that we use to define them.

The driving reason for this is that I've been looking over some old customer support service level agreements (SLAs) as part of our transition of ownership. Occasionally reviewing something familiar for a different purpose can provide new insights into the content of it. As I read the SLAs I was struck by the absolute nature of the terminology used in relation to issues encountered. To paraphrase one such agreement with a former customer:

  • Defect: a deviation from the documented product behaviour
  • Bug: See Defect.

Such statements are commonplace in contractual agreements , both within and between organisations. The text has clearly been structured for the purposes of avoiding ambiguity, and on the face of it this has been achieved. While the terms appear straightforward and unambiguous, the actual wording contains some interesting and possibly concerning implications for software delivery.

Over the next couple of posts I'm going to examine the concept of defects and their definition in contractual type agreeements in light of experiences in testing, customer support and documentation, to see how our behaviour in the field of software has evolved some interesting and adverse characteristics in response to these definitions.

What's in a Name?

The term 'defect' bothers me. The concept of a defect in terms of service level agreements such as the one quoted above have an incredibly narrow scope of applicability. As we can see in the above example, the definition relates specifically to a deviation from the documented behaviour. When compared with the range of ways that I have encountered that negative relationships can arise between software and customer the definition of a defect is restrictive. It simply does not cover the range of problem types that can occur in software use that I encounter on a daily basis and can impact on our contractual relationships.

The concept of a problem being defined as a deviation from the documented design leaves a wealth of potential problems undefined within the scope of such agreements:

  • the design fails to consider some valid scenarios of use
  • a performance issue whereby the software does not meet the customer expectation
  • behaviour that is not explicitly referred to in the documentation
  • an unintuitive interface
  • an inconsistency with expectations based on related products or technologies

All of these things can potentially impact customer perception, yet sit outside of the definition of a defect in the terms of many SLAs.

From my experience running the technical support team of a data storage system I can honestly say that a significant proportion, if not a majority, of tickets that are raised by customers do not relate to defects per-se but rather areas of behaviour, such as the above, that are far more ambiguously defined. My policy historically in these matters has been to strive towards delivering excellent customer service over quibbling around the definition of defects. Even so there are cases where we have to manage the customer expectation due to their specific query shape not having been optimised over tables containing hundreds of billions of rows of data, which can sometimes be difficult.

Even when adopting a more flexible stance, the expectation and even preference from many of our customers can be on a rigid approach due to the fact that that is what their processes are set up for. The rigidity provides the impression of predictability - a desirable characteristic of the support service.

It is this exact rigidity, however, that gives the greatest scope for conflict when problems such as those I mention above arise. At that point it is entirely dependent on the supplier whether to take a flexible stance and provide the assistance the customer needs, or whether to state that the software is behaving as designed, quote the SLA agreement and close their ears.

An internal problem

A rigid definition of what constitutes a defect can also affect internal relationships such as that between developer and tester. There was a time when a significant portion of my working day was spent raising bugs during a waterfall testing phase and seeing them fired back in the bug tracker with the comment 'as designed' added by the programmer.

This "bug tennis over a waterfall" situation can be a soul destroying one for testers, when their efforts in raising genuine concerns over the product are met with the brick wall of a blank refusal to question the design. I'm fortunate, as are many folks working in collaborative teams, to not encounter this situation, but I've encountered it first hand in other roles. Based on prior experience I can understand the stereotypical tester complaint of not having complete specifications up front. When the concept of a defect is defined in relation to adherence to specification then without it, it can be difficult to justify the existence of a problem, as the tester simply can't prove the presence of a 'defect' in absence of a conflicting specification.

When I first introduced group exploratory testing on a middle tier data framework in 2004 the outcome was a collection of new defects but also a series of concerns over the implemented design. We fed these back to the product management team but with no mechanism to get the design changed these concerns went no further. These included issues such as a major inconsistency between the technical knowledge required to configure the user connectivity and the knowledge of the majority of the users.

Thankfully more of us are finding ourselves working in more collaborative teams, Agile or otherwise, where within the team at least there is less need for rigid Defect definitions. In my team the term 'bug' has historically been sufficient to cover all manner of potential problems from design through implemented behaviour and documentation. A bug is as much a placeholder for work as it is a defect in the system. How that approach will need to change as our bug processing comes under the scrutiny of a more traditional organisation structure remains to be seen.

Wherefore art thou defect?

Given the range of situations in which I communicate, even with a fairly informal intra-team definition of bug, I still find the need to be careful with terminology when communicating in other forums. As we can see in the SLA definitions above, 'bug' is synonymous with 'defect' in many contexts, including SLA agreements, and also in the ISQTB glossary of terms. Some, however, define a bug as a superset of defects, while for others it relates to a different entity entirely or an entity in a different stage in the software lifecycle.

Even if we limit ourselves to examination of just defects, what I have observed is that the definitions even within the software community, are inconsistent, sometimes dangerously so.

We've already seen that, in technical support context, a defect can be defined as a difference between behaviour and documentation. This also translates into software testing context, where a defect can be simplistically defined as a difference between actual and expected of a test - for example in testingmentor - a software testing website

While testing when a tester executes the test cases he might observe that the actual test results do not match from the expected results. The variation in the expected and actual results is known as defects.

ISQTB has the following :

defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.

The IT Law Wiki has an interesting definition - not least for its brevity considering the unquestionable importance of defects to software law.

[a] problem or “bug” that if not removed, could cause a program to either produce erroneous results or otherwise fail.[1]

(As an aside - a positive element for me both of these definitions is that they reference the potential for the problem to occur - a key characteristic of software problems that I wrote about in this post.)

This following definition, a popular one in many websites, does consider the possibility that a failure to meet user expectation constitutes a defect, however it does so in the limited context of incorrect or unexpected results.

A Software Defect / Bug is a condition in a software product which does not meet a software requirement (as stated in the requirement specifications) or end-user expectations (which may not be specified but are reasonable). In other words, a defect is an error in coding or logic that causes a program to malfunction or to produce incorrect/unexpected results.

With the variation in these, and many other, definitions it is not suprising that there is some confusion. Witness this conversation on stack exchange, originally asking about the difference between a defect and a bug, for examples of the confusion that supposedly rigid definitions of terms an cause.

Not a Defect, but a Flaw

In response to these definition problems I've found the word 'flaw' to be an extremely useful one. I find myself increasingly using it in situations where defect is overloaded and may have undesirable connotations.

Although the terms defect and flaw have similar definitions they carry very different meanings when applied. The beauty of the word flaw for me is that it has layers and nuances of meaning. It is this very flexibility and lack of formal definition which provides value when tip-toeing over the eggshells of behavioural conversation. Even the Oxford English definition provides three applications:

  • a physical flaw
  • a personality flaw
  • a flaw in a plan or design.

Picture what comes to mind if I described a car as having a defect, as compared to a car having a flaw. To me the former is a fault in a specific vehicle, a badly moulded or cracked component perhaps. The latter could as easily conjure up images of a general problem in the design which affects all cars of the model, from the major design flaw such as the fuel tank problems of the Ford Pinto through to the minor niggles that fill the pages of this discussion - Minor flaws in the cars you love - on piston heads.

The key for me here is that both flaws and defects can be defined in relation to an ideal, a model of perfection. With flaws, however, this ideal does not have to be achievable, and in some cases it is expected that it will not be so. I particularly like, for example, the parallels between the application of the term 'flaw' to software and to people. To say someone had a defect brings to mind physical defects such as heart defects and other conditions. To say someone is flawed carries a very different connotation. Even the most accomplished individuals in all fields, whether politicians, athletes, business people or any walks of life, have flaws. This doesn't mean that they need medical treatment or surgery - it means that they are not perfect, however this is in the context of perfection being neither expected nor attainable. Everyone has flaws, everyone's are different. People may work over time to improve on their flaws, but not doing so doesn't stop them from being successful, wonderful people.

I like to think of software in the same way. The principle of the defect has at its heart the unrealistic notion that perfection is achievable if we could simply find and remove all the defects. There is no better deconstruction of this notion than Jerry Weinberg's excellent - "Perfect Software and Other Illusions About Testing". Even though most acknowledge that it is not possible, the implication in the majority of defect definitions is still there that the only thing between us and software perfection is our inability to find and fix all of the defects.

A Flawed Idea

This principle that the defects are all that separates a piece of software from perfection is one I think we need to move away from. All software is flawed just as people are. Software evolves through human endeavour. Throughout its creation it will have been subject to innumerable human decisions over workarounds, limitations and compromises. Even applications targeted at the same job by different teams will have areas of relative strength and weakness irrespective of their defects. Some designs that work well in some instances may even constitute flaws in others - the phrase 'one of their greatest strengths was also their greatest flaw' is sometimes used to describe people but could equally apply to software, or products of any kind. The simplicity in running one piece of software, for example, may be a flaw in one situation where greater configuration flexibility is required, whereas the rich feature set of a competitor may constitute unnecessary complexity in another.

I tend to use flaw particularly when discussing product behaviours that work as originally designed, but when the design has some limitations of disadvantages in applicability in certain scenarios. This may be a known limitation in capability of running certain shapes of query, a relative weakness in support of one data type over another, or simply a process that is more complicated than it needs to be. The phrase 'one of the flaws of the system is that it doesn't handle storage of large binary objects as well as other data types' carries a whole different meaning if I swap the word defect in there.

If I examine our bug database, what is clear to me is the behaviours captured within it are not just defects, they are flaws. This is a good sign as it means that the testers are looking beyond simple programmatical errors to identify a range of potential problems. For example recently a bug was fixed to return a useful output in our diagnostic capture process. It wasn't a requirement of the original command, it was simply that on using the feature it didn't make a lot of sense not to have it there. Now you could argue for days over whether this was actually a new user story, or a defect through a missed requirement, however this simply reinforces my point. If we concern ourselves too rigidly with sticking to work that conforms to narrow definitions, then some useful and important might be ignored simply through the inability to define them as defects or justify them as new requirements, when in reality they are neither, they are flaws. I've been involved in feature discussions with customers where obvious and useful changes are ignored at the expense of trivial cosmetic changes, as the latter can more easily be categorised as defects and therefore fixes justified. It is saddening to see chances to improve products being overlooked simply because our categorisation of what constitutes a problem in the system is simply too narrow to include them.

It's in the Documentation - We're Covered!

In my next post I'll go on to discuss another worrying implication of defects being a deviation from documentation ...


Many thanks to Shilpa J at cartoonistshilpa.com for the excellent cartoon - feel free to get in touch with me if you have any ideas on testing cartoons that you'd like - Shilpa is looking for inspiration!

Wikipedia - David - for many Michaelangelo's Daid was flawed - see http://www.globusjourneys.com/travel-stories/italy/michelangelos-david/ for some interesting criticisms of the 'flaws' in the statue from its hands to its nose. Few that have seen it (which I am fortunate to have done) would argue, however, that it is a masterpiece.