This week I gave a talk at the Birmingham STC meetup entitled "What Testers can Learn from Marketing". The subject of testing and marketing was something that I had been intending to write or present on for a while, having gained an insight into marketing operations in my time testing a marketing analytics database. Committing myself to giving a talk on it was a good, albeit somewhat high pressure, way of providing an incentive to do this.
One the evening of the meetup, as I was preparing to give the talk, Vernon Richards (the @testerfromleic) came up to me and said "You're talk is either going to be really good or really short, as when I think of what we can learn from marketing I think - nothing at all!" (apologies to Vernon if I've misquoted him here.) I was hoping to encounter a little of this type of opinion in the room - is there really stuff that testers can learn from the people who put junk mail on our doormats, spam our inboxes and subject us to Kerry Katona on our televisions (if you are not familiar with the name, please replace with the name of any irritating minor celebrity from TV commercials in your own country)?
As I went on to explain - actually yes there is a great deal...
I want to learn
Marketers' work has a huge amount of crossover with Software Testing, and increasingly so as the models of testing change to meet the demands of emerging testing environments and contexts.
I typically feel the urge to run for the hills when someone pulls up a Wikipedia definition in a presentation, however I have to confess that on visiting the Wikipedia towards the end of preparing the talk, I found the following two statements on The Wikipedia Marketing page compelling and worthy of inclusion:-
“Marketing is the link between a society’s material requirements and its economic patterns of response”
There is an interesting, albeit indirect, parallel here – Marketing is an interface between requirements and response to requirements. A critical element in the success of a Marketing operation is understanding what people want and how they respond as a result, just as understanding what customers want and their resulting expectations and behaviours is a fundamental part of being a software tester.
“Marketing can be looked at as an organizational function and a set of processes for creating, delivering and communicating value to customers”
So both marketing and testing roles involve the understanding of what people want, and what constitutes value to them. Both involve the understanding of responses to those requirements and the attempts to meet them, and most importantly both are communication roles in relaying information in response to those requirements to allow value decisions to be made.
Looking at some of the specific activities that marketing is involved in I highlighted some characteristics of marketing activities which should be more than familiar to testers:-
- Market Research is an exploratory activity researching new markets and opportunities
- Marketing is a communication role providing information to inform decisions
- Marketing is a testing role – testing campaign effectiveness to establish differences and significance of response
- Marketing is a checking role – ensuring campaigns are delivered correctly
- Marketing is a learning role – taking information from previous tests and runs to improve future efforts
I think that all of these verbs are equally at home in discussion of a testing role as they are of Marketing, with one major difference…
Marketing has a lot more money behind it.
Marketing is Exploring
Market Research is an exploratory activity. It is the process of investigating information gathered both from and on markets to identify opportunities and threats that the company instigating the activity may want to position itself to best respond. The market research process is commonly defined as being a six stage process:-
- Step 1 - Articulate the research problem and objectives
- Step 2 - Develop the overall research plan
- Step 3 – Collect the data or information
- Step 4 – Analyze the data or information
- Step 5 – Present or disseminate the findings
- Step 6 – Use the findings to make the decision
There are a huge number of parallels here with the process of testing. Whilst a little more formal and staged than the approach that I prefer to take in a testing project, we can see that each step in the process has direct relevance to testing.
Articulate the problem. - How often do we have a testing problem where we are expected to wade into the testing without articulating the scope of the problem or the objectives of our testing activity? When I asked if anyone present had been asked to "just test something", without clarifying the scope or goal of that testing, the show of hands in the room of individuals suggested that it was something that affects most testers at one time or another.
Ensuring that the objectives of testing are well defined up front helps us to maintain focus on what it is that we are trying to achieve and avoid distractions. One of the recommendations that I read for Market Research professionals was to re-iterate the original problem at the end of the research activity to ensure that you communicated the purpose of your work.
Step 2 - developing a plan and Step 3 - collecting information are pretty much fundamental to any research activity. Whilst important in themselves and worthy of discussion, I personally felt that steps 4 onward were more interesting from the perspective of comparison and learning.
Analyze the Data - Marketers have a wealth of techniques at their disposal to analyse data collected from market research. What they are particularly interested in is identifying and distinguishing causal relationships – does having a dog make you more likely to buy a certain brand of toilet cleaner? The use of statistical modelling tools such as multi-variant regression modelling allows marketers to identify which variables have the most influence over behaviours that they are interested in. Whilst many of the testing problems that we face are more obviously deterministic – "if I enter this text and click this button then the application crashes", for certain behaviours, where the causes are less easily determined, these techniques could be something that we make use of. For example - if analysing failure rates in large installations involving high levels of concurrent activity it may be that the factors contributing to failures are unclear. The extensive use by marketers of regression modelling is something that I think will be seen increasingly in the testing industry as our patterns of testing change in relation to the systems we work on.
As we move towards the later stages the process changes in focus such that reporting and communication become the key elements.
Present the findings – mymarketresearch.com has some interesting recommendations regarding presenting the findings of a research project:-
When it comes time to presenting your results, remember to present insights, answers andrecommendations, not just charts and tables. If you put a chart in the report, ask yourself “what does this mean and what are the implications?” Adding this additional critical thinking to your final report will make your research more actionable and meaningful and will set you apart from other researchers.
This has huge relevance to testing, what they are saying here is that, to set yourself apart as a talented and valuable Market Researcher you will go beyond the collection of data and presenting of figures. Making use of critical thinking and reporting insights along with data from running a research project when presenting the findings will increase the researchers value to the business decision makers. I believe that this is a principle that applies equally to the reporting of the findings of software testing activities, particularly in the context of helping to deliver the next step.
Make Decisions - The final step in the process involves using the findings to make a decision. If we refer to the old favourite from - “How to measure anything” by Douglas Hubbard – all information gathered in business should be done so with the purpose of informing a decision. The structure of the 6 step process helps to maintain a focus on this goal as the final step in the process. Again this is something that testers would be well advised to bear in mind. The execution of tests and collection of data has little relevance if not helping to inform a decisions. More importantly, if the data that we are collecting is the wrong data to inform that decision, such as counting a percentage of automated test cases that have passed to decide whether to release rather than examining results and implications individually, then we could be doing more harm than good.
Whilst simplistic, I believe that one of the main strengths of this model is that it maintains a focus on the end goal of making decisions, such that it is maintained as the ultimate target of the previous stages.
Accept decisions - mymarketresearch.com also has some relevant advice relating to the fact that you may not always get the decision that you expect.
Remember that market research is one input to a business decision (usually a strong input), but not the only factor.
Whilst you have collected and presented critical information to help to inform decisions, there will be other factors in play which will also influence those decisions and the logical decision based on the results of testing alone may not be the one that is made.
Marketers as Testers
Given the cost of planning and executing marketing campaigns, there is naturally a high level of interest from the campaign organisers in ensuring that the campaign ‘creatives’ (the media deliverable of a marketing campaign) are tested to establish their effectiveness. Campaign Management therefore involves a huge amount of testing, and marketers have come up with some interesting solutions to what essentially break down into a series of testing problems.
When looking to make changes to a marketing message or deliverable, it can be a major risk to make the change wholesale to the entire campaign list. One example given on a website I was reading was if you attempt to introduce personalization and it goes wrong it can have an opposite effect to that desired, making the recipient feel even less valued. Having been on the end of a “Dear [CustomerName],” email I know how much negative feeling can be generated from mistakes such as this.
It is not just mistakes that Marketers test for, even subtle changes in campaign deliveries can have a postitive or negative impact on the effectiveness on that campaign that can result in big differences to resulting sales. Instead of risking a negative impact hitting the entire campaign list, typically the marketer will perform what is known as A/B segmentation of the list to identify a reduced set of individuals to receive the new/changed campaign.
Segmentation actually has two related but distinct terms in marketing. Market segmentation is the process of splitting a market, typically on socio-demographic factors, in order to identify those segments most likely to respond to specific marketing approaches. Campaign list A/B segmentation, which is what I am referring to here, is a testing activity where the list of people targeted for a campaign are split to measure different responses to different marketing activities.
The marketer will create a ‘test cell’ of individuals who will receive a different target to the ‘control’ group. These two groups can then be compared for significant differences in response to see if the test of the changed campaign was more effective than the control.
What the marketer is essentially doing when performing these tests against the ‘test cell’ is tackling the problem of 'Testing in Production'. The data in play is live data and therefore the risks involved with potentially costly mistakes are high. These processes of testing are responses in order to allow the testing of different campaign formats, or correct delivery, whilst attempting to reduce the impact that mistakes could make on a live customer base such as a list of email subscribers. These approaches are directly comparable with any application or software that releases new code to a subset of selected users in order to elicit feedback and ‘test’ the response before releasing to the full user base and risking an embarrassing rollback. Testers operating in these kind of systems are very well placed to use marketing techniques to establish significance of these changes, not only in relation to new bugs but potentially in terms of other negative relationship indicators such as account closure or reduction in use.
Marketers as Checkers
With the number of individuals typically targeted in a marketing campaign it is rarely practical to test every individual to ensure that they received the ‘creative’. Instead the marketer may opt to test a subset of the list , or they may decide to include a set of predefined records who are not real campaign targets but individuals specifically included to test the receipt of the marketing creative, known as a set of seeds.
As well as being a response to the problem of using production data for testing, this approach of using seeds is also a response to the challenges of ‘Testing Big Data’ – something that I am very familiar with. The cost of checking every single recipient for successful receipt of the creative would be prohibitively expensive, just as checking every data row in a data warehouse or big data storage system would not be practical. Instead the marketer creates a series of dummy respondents exhibiting the required characteristics which can then be included in with the live campaign list to test the correct execution of the workflow.
I had a conversation on big data testing with a number of testers last year, one of whom was working on a data warehouse ETL process, and this is the exact approach that he took to test the ETL processes. He would design test data records which would exercise the various decision rules of the ETL process, and include these in the batch live data payloads. It was then a much less costly job to check that these seed records had undergone the correct data transformations than looking at the huge volume of records in the live payload.
Closing the Loop
Marketers use the phrase, “Closing the Loop” to the process of analysing previous campaigns and activities to improve future efforts. According to Tom Breuer in his paper How to evaluate campaign response
Closing the loop' is key in state-of-the-art database marketing. It means testing, measuring–tweaking campaigns.
Marketers spend a huge amount of time examining the results of previous campaigns to establish the most effective targets and ‘creatives’ to yield the most response. Breuer identifies two costs associated with running campaigns. There is the “complexity cost” of creating the deliverable, and also the “opportunity cost” of testing against control groups which are not marketed to yet still have to be tested. He states that using control groups inevitably yield lower responses than groups which have been targeted (although I don’t think that he had considered the possibility of Kerry Katona being involved). A key to making the most of the marketing investment is to make the most of learning from previous campaigns to identify the specific customers who are most likely to respond, so marketers will again perform a wealth of data mining and analytics on previous campaign data to identify the targets that are going to give them the greatest benefit relative to the investments of marketing to them.
Whilst continuous improvement and continuous learning are core principles for the majority of testers that I relate to, this kind of algorithmic learning is something different that has very specific applicability in the testing arena. I’ve recently been investigating the use of stochastic test generation for SQL testing. This approach involves the random generation and execution of test queries, using other databases, or other versions of our own system, as test oracles to identify inconsistencies. This type of testing activity is very much a ‘closing the loop’ type operation. A model is created for test generation, then the outputs from running that model are used to see how many issues the tests identified. Given that the time taken in executing the queries, and examining the outputs to factor out false negatives, is the major cost in this exercise then analysing the output results to improve the chances of finding issues in the next run is intuitively an approach that makes sense. This allows the tester to tweak the model in order to focus the next iteration of tests on the model structures that expose the most problems.
A natural progression once this cyclical process is established would be the introduction of algorithmic learning programs to identify the models that are most effective at exposing issues. The ability to mine historical data in order to guide future tests, particularly in an automated randomised testing activity, is a compelling capability.
I should make my position clear on this kind of approach - I think that it can be hugely powerful in the right context, such as applications with a well defined input model like an SQL parser, and only then as an augmentation to human exploratory testing. As these approaches increase in popularity and move into the testing mainstream, then the body of knowledge available in the Marketing community around iteratively testing, tuning, learning and measuring the significance of responses is going to prove invaluable.
Wrapping up
I received some excellent feedback on the talk and the subject matter, I hope that the material in the form of this post proves as interesting. I acknowledge that with the talk and this post I have only scratched the very surface of the marketing profession and what it can teach us. In the subsequent discussions around learning from other professions, the meetup group discussed other fields that we could look to learn from, and also where we should be cautious. I was glad that many contributions in that discussion echoed my own feelings. For me the primary principles that I maintain when looking to other fields for ideas are these:
- Testing is a research, exploration and communication role far more than a rubber stamp quality checking operation and I believe that the professions that we look to learn from should be reflective of this. We shouldn't always look to manufacturing as the only industry from which we can learn. There are a wealth of other fields which pursue more relevant areas to the role of software engineering, and testing specifically. A little imagination and lateral thinking goes a long way in finding a rich field of research from which to learn to better ones craft.
- More importantly - we should look to learn and from other industries and professions and adapt, not attempt to mimic them. Software engineering and software testing stand alone as having very specific challenges and opportunities. Whilst other fields have a huge amount to teach us we should approach this in the sense of selecting the most useful and appropriate elements instead of resorting to mimicry.
Further Reading
A useful glossary of Marketing Terms http://www.campaigner.com/resource-center/discuss-and-share/glossary.aspx
Basic introduction to the 6 stage Market Research Process http://marketresearch.about.com/od/Market_Research_Basics
A good general website on Market Research http://www.mymarketresearch.com
Some useful further reading on approaches to forecasting from Judgemental Models to identification of Causal Relationships http://www.marketingprofs.com/Tutorials/Forecast/index.asp
Tom Breuer's Paper on How to evaluate campaign response http://www.palgrave-journals.com/jt/journal/v15/n2/full/5750036a.html
A useful introduction to marketing data analysis techniques using Excel https://www.iacquire.com/blog/quantitative-data-analysis-techniques-for-data-driven-marketing-2
Slides from the talk can be found on my talks page
Images: http://www.flickr.com/photos/x1brett/5054129673 (Modified); http://www.flickr.com/photos/maccast/306309853/sizes/o/
Post a Comment