Sunday, 19 February 2017

The Innovation Silo

Who is responsible for product innovation? Is it an individual, a specific role, a devoted team, or is it everyone? I can tell you now that unless the answer is the last of these options then you could well have a situation that undermines the motivation and performance of your development group.

The Innovation Silo

A couple of weeks ago I was discussing with a colleague the challenges of innovation. We'd both encountered in previous roles the situation where one individual or group was given a free role on innovation. These people went by different titles, either as individuals or teams : Chief Scientist, Architect, Research Team. The result was the same: those given the freedom to research and innovate would perform those activities in isolation from those working on the main product streams, operating in what was effectively an 'innovation silo'. The focus of the innovator's activities would be to come up with advances in technology or capability. Their output would be aimed at senior management, with the goal of demonstrating to them a 'working' capability of an idea that they had developed and obtain approval to push it through to production.

The Motivation Sink

Arrangements like this were great ... if you happened to be the ones in the innovator roles. These people enjoyed a high level of job satisfaction and operated with a low fear of failure as their work was rarely exposed directly to live use, instead it fell on the regular development team to take these initiatives and carry them through into production.

In addition to inevitable practical issues of handing over prototype code, (which I've decided is a big enough topic to merit a post of its own shortly), the problem was that this arrangement caused a huge amount of frustration for the individuals that weren't part of the innovation activity. The freedom that one individual or team enjoyed came at the expense of many other developers and testers working hard to deliver the commercial obligations of the company. The people who were responsible for the day to day delivery of release schedules and customer deadlines would rarely have the time or the freedom to try out new ideas, unless done in their personal time. The closest many would come to innovation would be in struggling to understand, implement and test designs (if they were lucky) or code (if they weren't) of new products or features which had been created by someone else.

The inevitable outcome of this type of activity on the development and testing teams is demotivation. Fundamentally what we are saying to these people is - "we don't trust you to come up with good ideas". Not only is the exciting work of innovation and research out of reach to the rest of the development group, but to add insult to injury they are then asked to take the output of that innovation and perform the necessary yet uninspiring activities that are so crucial in software, and deliver these in an atmosphere of unrealistic expectation and high fear of failure. I've never encountered the situation where a developer was ecstatic to be working on someone else's code, however at least when that code has been working as part of a live system there's an element of respect in it. When the author of the code has never had to expose it to formal testing or the rigours of production use then that respect is absent and resentment can result.

The arrangement places testers in a frustrating situation as well. Being isolated from the process of innovation results in a separation from the goal of the development. In order to perform effective testing a tester needs to be on board not only with the behaviour of a solution but also to have a keen grasp of the problem that is being targeted. When product research is performed in isolation from the testing group then testers are forced to work within the solution domain. Any benefits of having the critical thinking of a tester integrated in the design process to question decisions and refine early designs is lost. Even if the tester does have valid concerns it can be politically difficult to raise these in the face of support from influential technical staff backed by senior management. Instead the tester politically and practically is compelled to limit their frame of reference to what functional understanding they can gain from behaviour of the developed solution. Their role is diminished to a confirmatory one of verifying existing behaviour rather than a wider scope of questioning value.

An open innovation

I'm hoping that I've just been rather unfortunate to see this type of relationship in more that one of my past jobs - I'd be interested to know in the comments here whether anyone reading this has encountered a similar situation. It it a common scenario for opportunities for innovation to be restricted to a chosen few?

I'm sure that many companies out there have very open and inclusive approaches to innovation, and I'm keen to help make sure that my current company is one of those - so what are the alternatives to innovation silos? In a recent product workshop I ran, some ideas that we identified to avoid this kind of situation included:-

  • Getting everyone in the company on a user feedback forum to share their innovation ideas and vote internally for their favourites
  • Allowing people time to pitch their ideas and provide the support for small cross-functional teams to be created to develop them together.
  • On a similar vein one of the developers at River pitched an interesting idea recently of having 'ideation sessions' where at regular intervals different cross functional groups were given the chance to try out an idea and develop it.
  • Having innovation sessions at whole company days to get everyone involved at the same time to generate ideas.
  • Developing and trialling new capabilities into our own internal version of our engagement software

Whatever form it takes, I think the important thing here is that everyone should be able to contribute and feel part of innovation activities. At the very least everyone should feel included in the process and able to actually raise their concerns over any new approaches being adopted.

As I stressed in my post on 'sharing the vision' there's a huge amount of value in simply having all of your team members engaged with a vision and able to identify connections and opportunities that help to move towards it. You don't have to be the smartest person in the company, with a job title of 'Scientist' or 'Architect' to come up with an idea that advances your capabilities. Naturally some individuals are more inclined to innovation and will generate more ideas than others, however at the same time many great advances come from simply making connections, such as applying an existing approach to a different problem, which anyone can do. The less we think of innovation as part of a role, and more part of an organisational culture, the more likely we are to get everyone driving and supporting our innovations rather than resenting them.


Tuesday, 31 January 2017

Sharing the Vision

When you work in the field of employee engagement , as I've done for the past year at River, it's hard not to pick up a thing or two about the subject. Luckily for me this was already an area of interest - as I wrote in this post I've always placed a high value on the motivation and personal development of the individuals in my teams, so working in the field of employee engagement is a great opportunity to extend my own knowledge in an area I care about.

Whilst researching the subject of engagement, one strong driver of people's commitment to their employers that I've seen referenced as crucial across a number of studies is when they share in a vision of the company's goals and culture. This sounds like a simple thing to achieve, however many organisations struggle to do exactly that.

The importance of vision

Studies such as this one of employee engagement and motivation, such as this one have identified the feeling that you are a contributing part of a larger vision for your organisation as one of the strongest factors in engagement and motivation. In fact some studies have concluded that a feeling of being part of a wider vision is the single most important factor in employee motivation.

But what do we mean when we talk about a vision? Well what I'm not referring to its a facile written statement, posted on the office wall - although I'm aware some organisations seek to compensate for an absence of genuine vision with just such a weak imitation.

This is not to say that vision statements themselves cannot be inspirational - this list of vision statements for nonprofits organisations contains some wonderful examples (though you have to admit the creation of an inspirational vision statement must be somewhat less challenging when ones organisation has such inspirational purposes). For organisations with less altruistic goals the encapsulation of the vision into a one line statement can feel contrived, and I personally find written vision statements in isolation from company practice or principle both trite and unsatisfactory.

A vision is, and needs to be, more than a statement. Just as in agile development we describe a user story as a place-holder for a conversation, so a vision statement needs to act as a place-holder to summarise the common values, goals and behaviours that characterise the employees of an organisation. A company vision is more like a shared consciousness of where we want to be, and how we want to get there and has to be not just understood but embodied by the individuals working for that organisation.

Missing the vision

Given the importance of a vision it's surprisingly easy to overlook when hiring teams and structuring work. This HBR article quotes 95 percent of employees being unaware of the company strategy. In development specifically I've seen situations where there simply isn't a high perceived value in sharing corporate goals with development teams. Perhaps it is the often quiet, focused nature of roles such as development and testing that results in the perception of people in these roles as resources that can be moved around interchanged, needing only their immediate inputs and outputs to be defined to deliver effective work. It is easy to target peoples' efforts on tackling the feature developments, bug fixes and releases that the make up the day to day activities, without considering any value in their understanding the wider organisational vision.

Why would we worry about missing the vision? As we tackle the tactical work that is prioritised around our immediate needs there will be opportunities to advance ourselves towards our longer term goals. Whether or not we take these opportunities depends heavily on whether the individuals working on those activities identify them as important. I've seen very successful cases of teams identifying opportunities. For example a team working on a customer website project recently identified the addition of a new user role in the programme as an opportunity to start introducing a content management capability that was one of the long term strategic aims of that programme.

On the flipside, opportunities can be missed if those involved don't have context of a wider purpose. I've also encountered projects that deliver value in the project context in question, but miss the opportunity to deliver value on a more central strategic level for the organisation of the work had been approached slightly differently. If the whole team is not on board with the strategic goals of the company then opportunities can be missed.

Vision workshop

Small companies, such as the ones I've worked in for the last 12 years, often have a distinct advantage when it comes to sharing a vision in that they are likely to have a much more accessible CEO or leadership team than larger organisations. Having a leader that can interact directly with employees is one of the key strengths of smaller organisations and I believe can provide competitive advantage in the motivation and engagement that it can drive. Software development and IT teams in particular can be operational silos, separated from the roles with a more direct business connection. Having access to top level leadership who can convey the importance of the work that is being achieved across a company is a huge benefit to these roles.

Recently I organised a workshop day with the main aim of sharing the vision of the CEO with the new product owners recently brought into the company and to allow those POs the opportunity to cement their relationships with other key people in the company. An introductory session from the CEO on the history of the company justified the day before we'd even hit lunchtime. I'd not previously appreciated the direct historical link from the very first engagement rewards company to the modern company that we were now part of. Learning the history of River and how this has shaped the programmes and capabilities now being offered was both interesting and important knowledge for the attendees to understand.

In further sessions we explored our capabilities across various customer programmes and examined how these fitted with the theories and models of engagement which are relevant to our business domain, as well as looking at options for how to validate new ideas and innovations through the research and development lifecycle.

Whilst some elements of the day were more successful than others, for example the sessions probably generated too many high level ideas at the expense of practical implementable suggestions, overall the value of having the Product Owners in a room with the experience of the CEO and creative and UX folks discussing our products was invaluable in instilling in them a vision of the company and where we wanted to be.

making your own vision

Not everyone has the autonomy to arrange events such as the one I just described. Even those that do sometimes don't have the luxury of an inspirational leader or a company situation that they find engaging or motivating. The good news here is that in my experience it is still possible to engage people with a vision in the form of a group cause that people can get behind.

At one point in a previous role the development department were struggling with motivation. A significant change was being imposed externally which as a group we had little control over or belief in, but risked a damaging impact on the quality of our work and our pride in it. It was a challenging time where engagement was in short supply. Whilst it was never going to be an inspiring project, I focussed on establishing a collective cause for the development team. Our goal was to take ownership of the situation and do the best job that we could of providing a quality delivery. As a group we renamed the project to introduce some team ownership of the situation. We established the work we were able to do in the short term to understand and improve the status of it and defined and communicated appropriate approaches to refactoring, testing and documenting the changes (the testing was based around my mind map for testability).

I believe that the work done in the time available on that delivery was exemplary. As a group, having a shared vision of what we were working towards really helped us to gel together, when it would have been easy to descend into melancholy and denial of the situation.

Just do it

"If you are working on something exciting that you really care about, you don't have to be pushed. The vision pulls you". Steve Jobs.

Even without inspirational leadership of the nature of Steve Jobs it is possible to create a vision within a department or team that motivates and inspires. This could be working towards an aspirational goal or simply collectively tackling adversity in a constructive way. Just as there are always going to be times when things aren't going as well as we'd like and a clear vision can help us in persevering, so it can be that when times are good a vision helps individuals to make connections and create opportunities to move us further, faster.

Having a higher purpose in terms of a group or organisational vision is such an important motivator that, if you don't have one, and you don't have access to someone who does, maybe it's time you made your own.


Sunday, 20 November 2016

Rewrites and Underestimation

In my opinion, the word 'rewrite' is one that should not exist in software. The word is most commonly used to describe the process of recreating functionality or behaviour of an existing piece of software through the process of writing a new one. On the face of it this presents a straightforward task - does the previous incarnation of the software provide us with the perfect specification for the development and testing of the new capability? Unfortunately things are rarely that simple. Use of the term rewrite implies such a level of simplicity in the activity that belies the difficulty involved in such an undertaking. In my experience rewrites are some of the most underestimated of all development projects.

Why rewrite?

So what would possess us to attempt to recreate a piece of software? Surely if we have something that does the job then we don't need to rebuild it? I've encountered two main reasons for performing rewrites - either to support the same use with improved technology/architecture, or to allow the same feature set to be delivered into a new context. In both of these cases there is an inherent assumption that recreating the same feature set will yield the value that you are looking for, which is by no means guaranteed.

Another false assumption in these situations is that development will inevitably be quicker because we've 'done it before'. The fact that we want to rewrite implies a certain level of success in the existing capability in terms of adoption and use. Despite the effort involved on both the initial build, and also in the ensuing months and years of maintenance and improvement, to deliver that success we seem to have an incredible knack of forgetting what it has taken to get to where we are and consistently underestimate rewrite projects.

How do I underestimate thee? Let me count the ways

Let's look at some of the many ways that these underestimations come about

  • Underestimating how smart you were before
  • The attitude of those approaching or proposing a rewrite is usually along the lines of insisting that we won't make the same mistakes that were made in the original development. This can be particularly amusing when it is the same individuals approaching the rewrite that created the original. How easy it is to forget that we were actually pretty smart when we created the original software, and the decisions that we made were done for genuine and valid reasons. I've lost count of the number of times I've looked back at work I created in the past somewhat sceptically and ended up quietly impressed with myself for the quality of it under scrutiny. We shouldn't assume that anything we create now will inevitably be better than what went before. If we don't give ourselves enough credit for the work done first time around then we risk simply repeating the same mistakes, only worse because we have given ourselves even less time to develop this time around.

  • Underestimating the power of incremental improvements
  • It's amazing how prone we are to forgetting how much a piece of software has improved over time relative to the initial release. One of my earliest tasks at RainStor was to test a new version of our data import tool. We already had a data importer that was suitable for our range of existing supported types. The reasoning behind the rewrite was to allow us to more easily extend to add more types to the supported range. The work had been performed by the chief scientist in the company, a man without whom "there would be no company" according to my introduction on my first day. As a new recruit to the organisation, testing this work was understandably a daunting prospect. What I discovered repeatedly through testing it was the performance of the importer was nowhere near what we were seeing for the previous one. Ultimately in attempting the rewrite the focus had been on improvements relating to architecture and maintenance, yet an important customer characteristic of performance had been sacrificed. It had been assumed that a better architecture would be more performant, however a huge number of iterative improvements had gone into improving the performance of the previous version over time, to the effect that the newly written version struggled to deliver equivalent performance.

  • Underestimating the growth in complexity
  • Just as quality and value will incrementally grow in a product over time, similarly complexity also builds. The simple logic structures that we started with (we were, after all, pretty smart) may have become significantly more complicated as new edge cases and scenarios arose that needed to be supported. On a recent development rewrite I worked on, I thought I had a good understanding of the user/roles structure after a spike task to research that exact area of the old system had been completed. As the development progressed it became apparent that my understanding was limited to the simple original roles structure first created in the product. Subsequent to that a number of much more nuanced rules had been added to the system which made the roles structure far richer and more complex than I understood it to be.

  • Underestimating how bespoke it is
  • It is tempting to look at an existing system and think that the logic can be 're-used' in a new application. This can be dangerous if the differences in application context aren't considered. Even very generically designed systems can grow and mould over time to the shape of the business process in which they are applied through maintenance, support and feature development requested by the users.

  • Underestimating how much people are working around existing behaviour
  • As I wrote in my piece "The workaround denier" the ways that people use a system are often very different to how we want or expect them to be. If we don't understand the workarounds that folks are using to achieve their ends with our products then we risk removing important capabilities. One one system I worked on, some users were working around restrictions on the data available under an individual account by having multiple accounts and logging in and out of the system to view different data linked to each user. When, as part of a rewrite, we provided them with a single sign-on capability this actually hindered them as they no longer had control of which account to log in with.

  • Underestimating external changes
  • Even if the users of a product haven't changed, the environment they are working in and the constraints of it will have. I was caught out not long ago on a project working with an external standards body. The previous software had achieved an accreditation with a standards organisation. As we'd essentially copied the logic of the original we mistakenly assumed that we would not encounter any issues in gaining the same accreditation. Unfortunately the body were somewhat stricter in their application of the accreditation rules and we were required to perform significant extra development in order to fulfil their new exacting criteria. Our old software may not have moved on, but the accreditation process had for new software.

  • Underestimating how many bugs you've fixed
  • When creating a new piece of software we will accept that there will be issues apparent due to working on a new code base and that we will improve these over time. It's strange how when creating new code to rewrite existing software we somehow expect those issues to be absent. It's an unwritten assumption that if you've created a product and fixed the bugs in it, that you'll be able to rewrite the same capability with new code without introducing any new bugs.

A fresh start

The chance to re-develop a product is a golden opportunity to take a fresh start, yet often this opportunity is missed because we've left ourselves far too little time. What can we do to avoid the pitfalls of dramatic underestimation in rewrites? It is tempting here to product a long list of things that I think would help in these situations, but ultimately I think that it boils down to just one:-

Don't call it a rewrite

It's that simple. At no point should we be 'rewriting' software. By all means we should refresh, iterate, replace or any other term that implies that we're creating a new solution and not simply trying to bypass the need to think about the new software we are creating. As soon as we use the term rewrite we imply the ability to switch off the need to understand and address the problems at hand, and just roll out the same features parrot fashion. We assume that because we've had some success in that domain that we don't need to revisit the problems, only the solutions we've already come up with. This is dangerous thinking and why I'd like to wipe the term 'rewrite' from the software development dictionary.

Yes, a pre-existing system may provide all of the following:-

  • A window to learning
  • A valuable testing oracle
  • A source of useful performance metrics
  • A communication tool to understand a user community
  • A basis for discussion

What it is not, is a specification. Once we accept that, then our projects to create new iterations of our software stand a much better chance of success.

Image: Chris Blakeley