Monday 27 February 2023

Is UX the new waterfall?

‘It’s a real problem’ my host admitted as we walked down to the entrance lobby. I was in the main office of a major social media company in London a few years ago after completing some user research with their team. The topic we were discussing was how to embed UX discovery and design processes into agile teams without losing the benefits of agility. He didn't have a good answer. Instead he acknowledged that it was a big challenge that even the tech giants struggle with.

Where we came from

Some reading this may not have experienced working on a good old fashioned waterfall development project. They will have missed the protracted business analysis process and the joy of repeated document reviews, version edits and sign off that preceded any development.

When agile came along it provided a new found freedom for software product teams. A shift to conversations over extensive documentation brought developers closer to their users. Small, frequent releases allowed teams to focus on current problems with relevant solutions.

At around the same time that software product teams were discovering agility, a shift was also taking place in the world of websites - an increased focus on User Experience. The growth of e-commerce meant that websites had become major revenue drivers in their own right. A refined user experience became vital to harnessing the opportunities of commercial websites and marketplaces. Even the smallest uncertainties in the user journey could impact key metrics like basket completion.
As this Nielsen Norman summary of the history of UX explains - the "Web Revolution" through the '90s and '00s was a major driver in UX due to the changing role of software in the buyer journey.

on the web, the sequence is user-experience first, payment second. Making UX the gatekeeper of the money vastly increased executives’ motivation to invest in their UX teams

Two worlds collide

While often discussed collectively, looking back it's clear that the rise in popularity of agile and UX were driven by quite different motivations. At the risk of oversimplifying -

  • Agile adoption flourished in situations when the software was the product
  • UX methods were popularised in situations where the software sold the product

With the rise in popularity of SaaS what we have seen is an interesting coming together of these two forces under the banner of 'Product led growth'. Now the software both is the product and sells the product. B2B enterprise tools historically sacrificed polished design for an extensive feature set. Users of modern B2B SaaS products are still expecting rich and powerful functionality, but a consumer grade web/app experience is needed to drive adoption and license expansion.

In this shift lies the crux of a challenge to modern agility. SaaS product development involves the bringing together of two worlds that seem to be in opposition

  • Agile product teams who want to solve the highest priorities of their users incrementally on a sprint-by-sprint basis
  • UX research and UI design teams who want to follow a user-centred research process through multiple stages of prototype through to a final design

I've seen two responses to this tension.

Reverting to waterfall

The first approach I've seen is where user research and design are tackled as a pre-development project. A common driver for this can be the use of UX agencies through a reluctance or inability to hire permanent expertise. The shape isn't limited to external teams though, and in house design teams can also tend towards this shape if they adopt agency working patterns to serve internal customers.

I've seen this approach throw up a number of challenges

  • Data complexities take deep understanding. Anyone that's worked with a complex digital product will know how important it is to understand the data structures that support it. Design projects aimed at rethinking the user experience can fail on implementation if the interactions don't allow for underlying complexity.
  • User needs emerge organically. While it's certainly possible to lay out an initial design principal, a lot of learning comes from seeing users interact with a product over time. Observing how the software is used helps to drive small but impactful improvements.
  • It's hard to convey context. Even with an extensive Figma design and handover documentation a lot of context is lost when a design is 'handed over'. This makes it very hard for developers to refine and adjust if any element of the design isn't practical to implement.
  • Missing the art of the possible. Ironically when design and engineering are separated the result can be a lack of real ground-breaking innovation. External UX redesign projects I've seen have tended to limit their focus on reworking navigation and onboarding journeys. This should come as no surprise given that these are the foundations to success in the ecommerce arena. Genuine software product innovation needs input from those who understand the whole solution and what may be technically possible.

This all seems hauntingly familiar. By adopting modern design methods as a stage in a process we've undermined any of the benefits that agility was providing. Development teams aren't engaged with solving problems and are instead reduced to feature delivery teams. User stories go from being placeholders for a valuable conversation to little more than a novel structure for paragraph headers in a specification document.

A better way

There is an alternative to splicing UX design and agile together into 'waterfall mkII'. It involves developers, testers, product and UX designers working together as part of a cross-functional team. Discovery includes not only UX activities like user interviews and wireframing but also technical discovery such as research spikes and functional prototyping. While they don't need to participate in every process, each member gets visibility and input at every stage. While discovery and solutions ideation is still done ahead of development, the whole team are engaged with the end-to-end process.

A popular name for this approach is ‘dual track agile’. While not a term I necessarily promote, it kind of makes a lot of sense as the name conveys that there are still multiple processes going on (though I think there are more than two). It is unrealistic to expect to discover, prototype, refine and deliver complete solutions under the banner of a single user story in a sprint. Even multi-functional teams still need to split the work into separate 'tracks' to allow time and find the right cadence for each activity.

Discussing the practicalities of adopting such an approach is a topic for another post. I will say this is a difficult balance to achieve but when working well it is possible to obtain the benefits of both agility and great design:

  • Discovery at every stage making everyone accountable for ensuring the solutions provide the outcomes being targeted
  • Visibility for developers into discovery activities even if getting them fully involved in customer proves impractical
  • Technical expertise focussing on analytics helping to ensure user behaviour analytics is well supported in the product
  • An engineering team involved in solution ideation to provide informed input on viability before time and effort is put into impractical designs
  • Availability of designers through the process - for me this is essential to explain their thinking and provide input if the original idea proves technically impractical

Don't underestimate the problem

Writing this post has given me a clearer appreciation of why UX design in agile is such a challenge. When considering two modern schools of thought that both have a focus on user understanding at heart it's easy to assume they should be compatible. What we need to appreciate is the popularity of each approach has had different motivations. We can't assume we can combine expertise and methods that have quite different origins and retain all of the benefits. Nurturing a healthy whole-team process of discovery, innovation and delivery is a challenge, but the rewards are worthwhile and the journey to get there is for me, in my current role at least,one that I relish

References

Main photo by Jonatan Pie on Unsplash

Whatsapp Button works on Mobile Device only

Start typing and press Enter to search