Sunday, 20 November 2016

Rewrites and Underestimation

In my opinion, the word 'rewrite' is one that should not exist in software. The word is most commonly used to describe the process of recreating functionality or behaviour of an existing piece of software through the process of writing a new one. On the face of it this presents a straightforward task - does the previous incarnation of the software provide us with the perfect specification for the development and testing of the new capability? Unfortunately things are rarely that simple. Use of the term rewrite implies such a level of simplicity in the activity that belies the difficulty involved in such an undertaking. In my experience rewrites are some of the most underestimated of all development projects.

Why rewrite?

So what would possess us to attempt to recreate a piece of software? Surely if we have something that does the job then we don't need to rebuild it? I've encountered two main reasons for performing rewrites - either to support the same use with improved technology/architecture, or to allow the same feature set to be delivered into a new context. In both of these cases there is an inherent assumption that recreating the same feature set will yield the value that you are looking for, which is by no means guaranteed.

Another false assumption in these situations is that development will inevitably be quicker because we've 'done it before'. The fact that we want to rewrite implies a certain level of success in the existing capability in terms of adoption and use. Despite the effort involved on both the initial build, and also in the ensuing months and years of maintenance and improvement, to deliver that success we seem to have an incredible knack of forgetting what it has taken to get to where we are and consistently underestimate rewrite projects.

How do I underestimate thee? Let me count the ways

Let's look at some of the many ways that these underestimations come about

  • Underestimating how smart you were before
  • The attitude of those approaching or proposing a rewrite is usually along the lines of insisting that we won't make the same mistakes that were made in the original development. This can be particularly amusing when it is the same individuals approaching the rewrite that created the original. How easy it is to forget that we were actually pretty smart when we created the original software, and the decisions that we made were done for genuine and valid reasons. I've lost count of the number of times I've looked back at work I created in the past somewhat sceptically and ended up quietly impressed with myself for the quality of it under scrutiny. We shouldn't assume that anything we create now will inevitably be better than what went before. If we don't give ourselves enough credit for the work done first time around then we risk simply repeating the same mistakes, only worse because we have given ourselves even less time to develop this time around.

  • Underestimating the power of incremental improvements
  • It's amazing how prone we are to forgetting how much a piece of software has improved over time relative to the initial release. One of my earliest tasks at RainStor was to test a new version of our data import tool. We already had a data importer that was suitable for our range of existing supported types. The reasoning behind the rewrite was to allow us to more easily extend to add more types to the supported range. The work had been performed by the chief scientist in the company, a man without whom "there would be no company" according to my introduction on my first day. As a new recruit to the organisation, testing this work was understandably a daunting prospect. What I discovered repeatedly through testing it was the performance of the importer was nowhere near what we were seeing for the previous one. Ultimately in attempting the rewrite the focus had been on improvements relating to architecture and maintenance, yet an important customer characteristic of performance had been sacrificed. It had been assumed that a better architecture would be more performant, however a huge number of iterative improvements had gone into improving the performance of the previous version over time, to the effect that the newly written version struggled to deliver equivalent performance.

  • Underestimating the growth in complexity
  • Just as quality and value will incrementally grow in a product over time, similarly complexity also builds. The simple logic structures that we started with (we were, after all, pretty smart) may have become significantly more complicated as new edge cases and scenarios arose that needed to be supported. On a recent development rewrite I worked on, I thought I had a good understanding of the user/roles structure after a spike task to research that exact area of the old system had been completed. As the development progressed it became apparent that my understanding was limited to the simple original roles structure first created in the product. Subsequent to that a number of much more nuanced rules had been added to the system which made the roles structure far richer and more complex than I understood it to be.

  • Underestimating how bespoke it is
  • It is tempting to look at an existing system and think that the logic can be 're-used' in a new application. This can be dangerous if the differences in application context aren't considered. Even very generically designed systems can grow and mould over time to the shape of the business process in which they are applied through maintenance, support and feature development requested by the users.

  • Underestimating how much people are working around existing behaviour
  • As I wrote in my piece "The workaround denier" the ways that people use a system are often very different to how we want or expect them to be. If we don't understand the workarounds that folks are using to achieve their ends with our products then we risk removing important capabilities. One one system I worked on, some users were working around restrictions on the data available under an individual account by having multiple accounts and logging in and out of the system to view different data linked to each user. When, as part of a rewrite, we provided them with a single sign-on capability this actually hindered them as they no longer had control of which account to log in with.

  • Underestimating external changes
  • Even if the users of a product haven't changed, the environment they are working in and the constraints of it will have. I was caught out not long ago on a project working with an external standards body. The previous software had achieved an accreditation with a standards organisation. As we'd essentially copied the logic of the original we mistakenly assumed that we would not encounter any issues in gaining the same accreditation. Unfortunately the body were somewhat stricter in their application of the accreditation rules and we were required to perform significant extra development in order to fulfil their new exacting criteria. Our old software may not have moved on, but the accreditation process had for new software.

  • Underestimating how many bugs you've fixed
  • When creating a new piece of software we will accept that there will be issues apparent due to working on a new code base and that we will improve these over time. It's strange how when creating new code to rewrite existing software we somehow expect those issues to be absent. It's an unwritten assumption that if you've created a product and fixed the bugs in it, that you'll be able to rewrite the same capability with new code without introducing any new bugs.

A fresh start

The chance to re-develop a product is a golden opportunity to take a fresh start, yet often this opportunity is missed because we've left ourselves far too little time. What can we do to avoid the pitfalls of dramatic underestimation in rewrites? It is tempting here to product a long list of things that I think would help in these situations, but ultimately I think that it boils down to just one:-

Don't call it a rewrite

It's that simple. At no point should we be 'rewriting' software. By all means we should refresh, iterate, replace or any other term that implies that we're creating a new solution and not simply trying to bypass the need to think about the new software we are creating. As soon as we use the term rewrite we imply the ability to switch off the need to understand and address the problems at hand, and just roll out the same features parrot fashion. We assume that because we've had some success in that domain that we don't need to revisit the problems, only the solutions we've already come up with. This is dangerous thinking and why I'd like to wipe the term 'rewrite' from the software development dictionary.

Yes, a pre-existing system may provide all of the following:-

  • A window to learning
  • A valuable testing oracle
  • A source of useful performance metrics
  • A communication tool to understand a user community
  • A basis for discussion

What it is not, is a specification. Once we accept that, then our projects to create new iterations of our software stand a much better chance of success.

Image: Chris Blakeley

Tuesday, 27 September 2016

The Risk Questionnaire

What does it mean that something is "tested". The phrase "tested" troubles me in a similar way to "done" in the sense that it implies that some status has been achieved, some binary switch has been flicked such that a piece of work has achieved some threshold status that up until that point had not been the case.

Of course, there is no magic switch that renders the feature "tested" or the user story "done", we will simply reach a point where the activity performed against that item gives us sufficient confidence in it to move on to other things. My biggest problem with "tested" is that it can mean very different things to different people, often within the same organisation. When two different people in the same organisation refer to something as tested then that each can be taking a very different understanding of what is implied, particularly the exposure to risk involved.

A common understanding

Working for a long time in the same organisation affords one with the luxury of time to reach a common understanding across the various business and technical roles of what "tested" means. To the mature team, tested hopefully means testing has been performed to a level that we've established as a development team that the business is happy with in terms of the time and cost of carrying out the testing and the level of bug fixing that we're likely to need. This level will differ from company to company, with a critical driver being the level of acceptable business risk. I suggested in my post "The Tester, The Business and Risk Perception" that businesses, like individuals, will likely operate at a determined level of risk and the overall activities within that business will gravitate towards that level irrespective of the actions of individual testers.

In my experience an understanding evolves as a team or department matures around what the expectation is on testing and how much time is to be spent on it. If a team spends too much time on testing then it is questioned by management, if the team are pushed to do too little testing then they push back, or quality suffers and the customer feedback prompts review. Eventually in response to these inputs, an equilibrium is reached. But what of new companies or teams? How do we identify an appropriate testing balance between rigour and pace, when a team is either newly created or adopting a new approach such as agile? Is it possible to fast-track this learning process and calibrate ourselves and our activities more quickly around the risk profile of the business

Finding an equilibrium

I was recently asking myself just such a question when facing the challenge of establishing a test strategy for my new company. The company had grown quickly and also recently adopted agile methods, and so inevitable had not yet found its equilibrium in terms of an acceptable level of risk and testing. This caused some confusion across the development teams as they wasn't a consistent understanding of the level that was expected of them in their development and testing activities.

It would have been easy to come in and immediately push to increase the level of testing, however I felt that it was important first to establish amongst the business leaders understanding of the general attitude to acceptable risk. The approach that I took yielded some very interesting results:-

Learning from IFAs

Independent financial advisers (IFAs) need to understand risk. When an individual in the UK attends a financial adviser then they may well be asked to complete something like this questionnaire:-

The use of these questionnaires is common practice amongst financial advisers and institutions to assess the risk appetite of their clients and recommend a pension/investment strategy accordingly. The principle is simple, the customer is presented with a series of statements associated with their attitude to finances, with the answers to these questions provided through a set of standard options reflecting different risk levels.

Example questions might be ones like this

Assume you had an initial investment portfolio worth £100,000. If, due to market conditions, your portfolio fell would you:

  • a) Sell all of the investments. You do not intend to take risks.
  • b) Sell to cut your losses and reinvest into more secure investment sectors.
  • c) Hold the investment and sell nothing, expecting performance to improve.
  • d) Invest more funds to lower your average investment price.

If you could increase your chances of a return by taking a higher risk would you

  • a) Take more risk with all of my money
  • b) Take more risk with half of my money
  • c) Take more risk with quarter of my money
  • d) Not take the risk

The resulting range of answers provides the adviser with an overall impression of the individual's risk level, and they can build an investment plan accordingly. I recently went to an IFA to consolidate my pensions and took just such a questionnaire, coming out at a rather dull 6/10 (apparently the vast majority of people sit between 4 and 7).

I felt that this idea of a questionnaire was a useful simple technique which could be validly used for assessing the risk appetite of individuals across a development company. I wanted to establish a starting position that would allow me to progress the conversation around an acceptable level of business risk and establish whether or not there was consistency of opinion across the company on our approach to risk, and this looked like a good option.

The questionnaire

I created a risk questionnaire that we could ask members of the leadership team to complete. The audience was selected to be a range across development and client services, and the CEO. For the questionnaire I focussed around 4 primary areas

  1. Business Risk
  2. Development Risk
  3. Perceived Status
  4. Time Spent on Testing Activities

1. Business Risk

The business risk statements were focussed on the interaction between the software process and the customers. These were open to all to answer, but primarily targeted at the non-development leadership roles. The aim here was to establish an understanding of the priorities in terms of customer deliverables and the conflicting priorities of time, cost and rigour.

Rate the following questions on 1 - 5 from 1=Strongly agree 5=Strongly disagree

  1. On time delivery is more important than taking longer to deliver a higher quality product

  2. I am happy to accept the need for later effort in maintaining a product if we can deliver that product at a lower up-front cost

  3. Our customers would knowingly accept a reduced level of rigour in development compared to other products in order to keep the cost of the software down

  4. Putting the software in front of customers and responding to the issues they encounter is a cost effective way to prioritise fixing software problems

  5. The cost of fixing issues in production software is now reduced to the point that this is an economically viable approach

  6. Our product context is one in which we can adopt a relatively low level of rigour compared to other business facing software development organisations

  7. I would be reluctant to see an entire sprint given over entirely to testing and bug fixing unless this was driven by issues encountered by the customer

In review of the results we identified that some of these questions were open to interpretation. For example, in question 4. I was not clear as to whether this referred to giving customers early visibility prior to release, through demos or betas, or whether it meant pushing early to production and letting them find the issues in live use (I had intended the latter). Having similar questions worded in slightly different ways, such as question 5, helped to identify whether there was any misunderstanding there, however if doing a similar exercise again I would look more carefully for possible ambiguity.

2. Development Risk

Development risk questions focussed on the risks inherent in our development activity. The idea was to get an understanding of the expectation around the development process and the feeling towards developer only testing. Again these were open to all to answer, but did not shy away from slightly more involved development terminology and concepts.

Rate the following questions on 1 - 5 from 1=Strongly agree to 5=Strongly disagree

  1. The effective application of developer/unit testing can eliminate the need for further devoted testing activity

  2. Appropriate software design can eliminate the need for devoted performance and stability testing

  3. Adding further development skills in our agile teams provides more value in our context than devoted testers

  4. The testing of our products does not require specialist testing knowledge and could be performed by individuals with limited training in software testing

  5. I would be reluctant to schedule specific testing tasks on a team’s backlog without any associated development

In creating the questions I did try to avoid leading questions around any known problems or perceived shortfalls in the current status. At the same time, clearly the questions needed to be suitable for the nature of the business - questions suitable for on a long release-cycle big data database would not have been appropriate here.

3. Perceived Status

This was a really interesting one. I pitched two very straightforward questions around company status.

  1. With 1 being lowest and 5 highest rate how you think the company currently stands in its typical level of rigour in software quality and testing?

  2. With 1 being lowest and 5 highest rate how you think the company should stand in its typical level of rigour in software quality and testing?

The first of these doesn't give an answer that reveals anything about the level of acceptable risk, however what it does do is put the answers that do into context. Knowing how people feel about the current status and how this compares to where they want to be gives a strong indication of whether changes to increase development/testing rigour will be accepted and supported.

4. Testing Commitment

This section didn't work quite so well. I was aiming to get an understanding of how much time people felt should be spent on testing activities as part of a typical development.

Rate the following in terms of time; 1 = Less than 1 hour; 2 = 2 - 4 hours; 3 = 0.5 - 1 days ; 4 = 1-2 days; 5 = more than 2 days

  1. How much time out of a typical user story development taking 1 person-week in total should be given over to the creation of unit tests?

  2. How much time out of a typical user story development taking 1 person-week in total should be given over to automated acceptance testing?

  3. How much time out of a typical user story development taking 1 person-week in total should be given over to human exploratory testing?

One respondent didn't answer these as he felt that, even in the same organisation, different work items were too different to provide a 'typical' answer. My feeling was that there was a fairly typical shape for the developments undertaken that we could use to calibrate teams around how much testing to do, however I could see his point and understood the reluctance to answer here

Another issue here was that I provided timescales for user stories anchored by my experience of development of complex data systems. In this context "typical" user stories would be much shorter than 1 week in duration and therefore there was a contradiction built into the questions. Nevertheless the answers were informative and provided useful information to help in constructing a strategy.

Presenting the Figures

All testers are wary of metrics. It is a natural suspicion borne of seeing too many occasions of glowing bug statistics masking serious quality problems. Presenting the figures in a visually informative and digestible way was key to getting the benefits of the analysis here. I used Tableau Public to create some useful visualisations. The most effective being a simple min/max/average figure for each question. This format allowed me to highlight not only the average response but also the range of responses on each question.

(I've altered peoples names and scores of the responses here for obvious reasons, however tried to keep the outputs representative of the range of results and patterns that I encountered)

Business Risk:

With the business risk it was the ranges that were most interesting. Some questions would yield a wide range of opinion across respondents in their answers, whereas others would be much more focussed. Clearly in some areas specific individuals were prepared to consider a higher risk approach than others, something that hadn't been necessarily highlighted previously and possibly the cause of some uncertainty and pressure within the business. What was apparent in the real results was a general desire to reduce the risk levels in development and an acceptance of needing to increase rigour.

Development Risk

Most interesting on the development risk front was that, as I've shown here, there was 100% consensus on the need for specialist testing skills, however the organisations strategy to that point had been not to have specialist testers. Whilst testing skills doesn't necessarily require testers, the phrase "specialist testing skills" does imply a level of testing focus beyond a team solely consisting of developers.

Company Perception

The "Company Perception" demonstrated most clearly the desire to increase the level of rigour in development, with the desired level of testing clearly above what was perceived to be the current status in a similar way to the results shown here.

Starting from the Right Place

As I wrote in my post "The Living Test Strategy", a test strategy in iterative software development is not based around documents, but embodied in the individuals that make up the development teams, and the responsibilities that those individuals are given. The role of defining test strategy is then not in writing documents, but in communicating to those teams and individuals what their responsibilities and expectations are. Some fundamental decisions need to be made in order to establish a starting framework for these responsibilities. Questions such as:-

  • Are we going to have specialist testers?
  • If so will this be in every team?
  • What level of acceptance test automation is appropriate for the risk appetite of the business?

Need to be answered to establish the initial testing responsibilities and hiring needs of the team.

Using a risk questionnaire to profile the business leadership has given me an invaluable insight into the risk appetite stepping into a new company. In addition to giving an overall understanding of where the company sits in its acceptance of risk, the approach has also highlighted where there is a lack of consensus over testing issues that might need further investigation as to why.

As an experiment I would definitely regard it as a success. In my context, as someone stepping into a new agile organisation and looking at test strategy, it is particularly useful. I can see other uses as well, though. Whether your goal is to understand your new company, or to simply review where your current company stands, or even to expose conflicting opinions on testing and risk across the business, a risk questionnaire may just be the tool to help you achieve it.


A good guide to financial risk profiling and the requirements of it can be found here:

Another good example of a risk profiling questionnaire that you can take online:

Thursday, 21 July 2016

Making Luck

This week I've accepted a permanent role at River, the company that I've been consulting with since February. The title of the role is immaterial - what matters is that I'll be helping both Product Owners and Testers to develop themselves and their functions within the company.

I'd like to share the story about how I came to working in River. I apologise if it comes across as self-indulgent - I think there are some valuable lessons there for those focussing on their careers in demonstrating the value of the effort that we put into our professional development over time.

An Eventful Day

The events of my day on 18th January ran something like this:-

  • 8:00am - at train station as normal ready for work
  • 8:05 am - receive calendar invitation on my phone for "all company" meeting that morning. Predict redundancies.
  • 8:30am - get to work.
  • 10:00am - discover am being made redundant along with a number of colleagues
  • 11:15 am - in pub
  • 1:00pm - in Cafe eating disappointing sausage sandwich in pathetic attempt to soak up copious amounts of beer before going home
  • Afternoon with family sobering up
  • 7:00pm - At Cheltenham Geek Nights. Cheltenham Geek Nights is an development community event run in Cheltenham by Tom Howlett (@diaryofscrum). I'd become aware of it through communicating with Tom previously when he approached me to do a talk on Testing Big Data in Agile. The speaker on this occasion was David Evans. I had already been looking forward to taking the opportunity to catch up with David having benefited from his expertise in the past and maintained an amicable relationship since. After the events of the day I was particularly interested in his insight on the roles and team structures he was seeing develop through his work consulting in the fields of Agile Testing and Specification by Example.
  • 9:30pm - back in pub
  • 11:00pm - taxi home. Speaking to Tom and David that evening was the perfect antidote to my bad news and I ended the day on an optimistic note.
  • 5:00am - awake with hangover and worrying about future

During the next few days I was genuinely overwhelmed with the number of people who got in touch with me. Some contacted me just to wish me good luck, or to ask me what had happened. Others provided suggestions of possible roles or options available to me, and some even came in with direct offers of work. I can't overstate how valuable this communication was to me. Whether or not I followed up on every suggestion, the fact that so many people got in touch was hugely reassuring in an uncertain time and to those folks who got in touch - I can't thank you enough.

One of the options that I did follow up on was when Tom suggested a possible contract Product Owner role at a company he was consulting at. It turned out that he was looking to create a team of Product Owners to help to alleviate some of the challenges that they were facing in their Agile development adoption. This was exactly the kind of option that I was looking for - something that would allow me to get back to work quickly, in a very different organisation and provide some time for me to decide on a next long term move.

Landed on Your Feet

Over the ensuing weeks and months I've found that the nature of challenges presented in a very different work context have been refreshing. The experiences of working with such an amazing, close knit team at RainStor provide a constant reminder of how effective teams can be if you

  • Structure them with the right combinations of skills and experience levels
  • Allow them sufficient stability of members and technologies to confidently establish and meet release forecasts
  • Provide sufficient trust combined with flexibility over the delivered solution to fully engage the team in the creative process Having this knowledge is a powerful tool in looking where to focus to help add value in a new company.

When I've told people about my new position in a local organisation, a number have used a phrase like

'You've landed on your feet'.

This is an interesting one, as it implies a strong element of luck. Of course I would agree that the timing was somewhat fortuitous, however let's look at some of the events that led up to the situation:-

  • The contract role was suggested to me by Tom based on a conversation at a community event that I attended
  • The reason that I knew about the event was because of this blog and talks I'd done in the Agile/Testing communities had prompted Tom to ask me to speak at the event in the past
  • The reason that I specifically attended the event that day was because there were people there who I already knew from the software community
  • The reason that Tom felt confident in suggesting the role to me was because he had read my work, spoken to me personally, engaged me to do a talk and was confident in my abilities

This option, and other suggestions or offers that folks so kindly extended, made me really thankful for the time that I'd taken to invest in being an active member of my professional community.

The Slight Edge

I recently started reading the Slight Edge by Jeff Olsen. Whilst I'm not big on self help books, this one had been recommended to me and the concept interested me. One of the principles that the author presents in his book is the idea that it is not one-off Herculean effort but repeated incremental effort over time yields long term success. Many trends in modern society revolve around the concept of instant gratification: the lottery win, the talent competition, the lean start up overnight success. What Olsen suggests is that, far more reliable success comes from incremental effort to improve, applied over time. It comes from sticking with something even if it doesn't yield immediate results.

I've been writing a blog now for 6 years, and hit my 100th post last year, and I'd like to think that the effort that I've put in to blogging, attending meetups, making connections and generally working to improve myself through sharing ideas with others made that possible. If I hadn't invested that time over those years , then I wouldn't have interacted with so many different folks and would not have built a network of contacts and a body of work that allowed other professionals to place their faith in me.

Writing a blog is not easy, particularly when you have a busy job and a young family. For example during the first months of writing a software testing blog I had no idea whether anyone at all was reading it - it was 8 months and 10 posts before I got my first tweet share. In later years, particularly during the 'creativity vacuum' that comes when your small company is taken over by a large multinational, I have a couple of times considered stopping writing. I really started the blog from a desire for validating approaches rather than networking, but it was through redundancy that the realisation came as to just how valuable the effort was in building a great network of contacts to provide opportunities and guidance when needed. It is never nice losing a job involuntarily, yet ironically it was at exactly such a time that the extra effort in committing to something beyond the day job really showed its value.

I'm of the opinion that it is better to commit to one thing and stick with it than take on 5 things and give all of them up weeks later. I've given a specific example of writing here, however there are innumerable other areas where success comes not through moments of inspiration but through perseverance:

  • Keeping on top of emails
  • Keeping a backlog or to-do list up to date
  • Maintaining a sprint/status board
  • Writing unit tests
  • Testing with the right user roles not just admin
  • Adding testability/supportability properties to code
  • Having regular catch ups with team members
  • Diet
  • Exercise
  • Not having a cigarette (10 years so far for me)

The takeaway is that commitment matters here - you don't see the value, or get the power of incremental improvement, unless you stick at something. One of the great powers of the internet, blogs and conferences is that anyone with a great idea can contribute massively to their professional community. Looking at those people who enjoy universal respect amongst the groups that I communicate with, it is the ones that are persistently contributing, always putting in effort and maintaining their commitment that attract the greatest admiration.