Optimizing retail promotions by predicting customer lifetime value effects
“Buy one, get one free… Don’t miss this special offer: click here for 2 months premium membership.” Promotions. Retail is hard to imagine without them. Spending on promotions in Europe has doubled over the past 10 years, with nearly 30% of purchases being on promotion. This increased promotional pressure is a result of different factors. Increased competition from e-commerce and changing consumers expectations, to name a few. As such, the ability to design winning promotions with positive sales and margin impact is more important than ever. Unfortunately, many retailers get stuck at basic levels. Determining post-ROI evaluations based on solid uplift measurement – that is, compared to a baseline forecast – is more the exception than the rule. A more accurate view on value creation requires getting more granular, looking at promotion effects on a customer level. For instance, the lifetime value creation from newly attracted customers can be significant. With increasing shares of sales sold on promotion, accurate ROI measurement becomes a crucial capability to win the game. We identified up to 11 different measurable effects of promotions by using a customer lifetime value (CLV) approach. Insights into all these value drivers gives retailers the handrails to shift their promotion calendar to the highest performing promotions, and cut the underperformers. But that is only the beginning. Having accurate post-ROI as a foundation, there is a level up: optimizing future promotions (pre-ROI). This can be a real game changer but requires learning from myriad past promotions, extract winning patterns, and use those insights to design a winning mix for a new promotion. We develop an AI solution – called AIR-Pro – which can do this at scale. It combines human experience with machine learning algorithms to predict future promotion ROI. One European supermarket has adopted this solution in their promotion process and increased profitability of promotions by as much as 20% to 40%.
I. The game is on: getting more value out of promotion pressure
Promotional sales are an increasingly important lever for retail sales growth
Brick-and-mortar retailers, including supermarkets, discounters, drugstores, and mass merchandisers, rely on promotions for 10 to 45 percent of their total revenues¹. And the importance of promotions is growing. Since the financial crisis in 2008, a combination of increased price consciousness and changing consumer needs have triggered price wars between retailers who are struggling to attract and keep customers². The continued rise of e-commerce has created a more level playing field, in which consumers can more easily compare price and features. Next to superior customer service and experience, (personalized) promotions have become a more important tool to sway customers over. Discounts and promotions are at an all-time high, often comprising the single-biggest cost bucket for many retailers².
1 BCG 2015 – How Retailers Can Improve Promotion Effectiveness
2 Harvard Business Review 2018 – For Better Retail Promotions, Ask These Questions
Most retailers lack accurate ROI management and leave money on the table
Our work with European supermarkets revealed that more than 20% of promotions were loss making, if all (long term) effects on customer behavior were properly accounted for. More general, three-quarters of consumer product companies are struggling to grow both revenue and profitability simultaneously. Pricing and promotion are critical levers to address this challenge. But companies require a more fact-based understanding of what actually drives a successful promotion and manage those levers better³. Most retailers have at best some rudimentary ROI measurement which tracks the sales uplift compared to baseline levels, leading to an incomplete evaluation of past promotions. Optimizing future promotions is even more difficult without detailed insights into underlying drivers of ROI. This is a notoriously difficult challenge for retailers⁴. The complexity is that promotions trigger many different consumer behaviors that all mix into the bottom line. For instance, high promotional sales can come at the expense of lower sales for substitute products. Hoarding could negate the promotional sales uplift partially or even entirely. On the positive side, promotions can attract new customers, or drive return visits by building a brand image around certain promotional categories. Knowing all the effects of a promotion is a must to all retailers who want to be on top of their promotion game.
3 EY 2016 – Can smarter pricing and promotion reduce the emphasis on discounting?
4 McKinsey 2017: Perspectives on retail and consumer goods
Deep promotion ROI measurement: 11 effects
To approximate the true effectiveness of a promotion, the different underlying drivers of sales – both negative and positive – have to be isolated. Let us look at an example: Coca Cola 1.5L bottles. 2 for 1 (buy one, get one free). Figure 1 shows the ROI waterfall containing all 11 effects.
Example promotion: Coca Cola 1.5L: 2 for 1
- Total sales Coca Cola 1.5L during promotion period
- Sales Coca Cola 1.5L during promotion period without promotion (baseline)
- Δ Sales substitutes (Coca 1L, cans, Pepsi Coke, Fanta etc.) during promotion period
- Extra sales (rest of basket) from buyers that came to the store especially for Coca Cola promotion: Existing Customers
- Extra sales (rest of basket) from buyers that came to the store especially for Coca Cola promotion: New Customers
- Δ Sales Coca Cola 1.5L because promotional buyers build a long-term stash
- Δ Margin: cost of waste because of over ordering
- Extra sales Coca Cola 1.5L because promotional buyers continued buying Coca Cola but did not before the promotion
- Δ Revenue/margin because customers switch to brand Coca Cola from other Coke brands
- Sales from new customers from returning visits after the promotion period
- Total cost of promotion (promotion material, media support etc.)
Net Value effect = Sum of all calculated effects
Figure 1 quickly shows how different effects determine the net value creation of the promotion, in this case Coca Cola. After subtracting baseline sales of Coca Cola (driver 2), the analysis shows that this Coca Cola promotion attracted new customers who bought more than just Coca Cola (driver 5). Some of these new customers will become loyal, creating return sales after the promotion period. This also seems to be a big driver for Coca Cola ROI (driver 10).
This example shows how the different drivers, all representing different effects on sales and margins, determine the success and ROI of a promotion. Having these insights will help retailers accurately evaluate promotions. Only looking at sales uplifts (driver 1 and 2) in this case would give an underestimation of the ROI. Furthermore, they provide a first step towards optimization: designing future promotions in such a way that they are expected to create maximum value.
Optimizing future promotions: not a piece of cake
Optimization of future promotions can be done in two ways. The most obvious route is to develop a solid understanding of ROI of past promotions, as explained above, and use those insights to prioritize and continue high performing promotions and stop doing non-value adding promotions. This assumes that you repeat past success and stop past failures. It does not tell you how to design new promotions. The second, more advanced approach is to understand how different parameters of a promotion influence performance. These parameters can include many factors: product, store, discount, pricing mechanism, seasonal timing, day of the week, etc. Knowing the influence of all these drivers would allow to predict the expected ROI of a new promotion. That insight can be used to design winning promotions by tuning all the parameters to maximize ROI.
In short, we have a measurement and a prediction challenge. How to measure all the 11 effects of a promotion across products and customers? How to measure whether a promotion attracted new customers, and how much value that created? How to measure the loyalty effect of those customers, i.e. their repeat buys? How to predict all these effects for a new promotion, which has not been done before? And how to do this accurately across thousands of different products while accounting for changing market circumstances and varying customer behavior? This is where AI comes in.
II. Using AI to change the game: optimize ROI by predicting customer lifetime value effects
Clearly, the challenge of measuring and predicting promotion effects is complex and multidimensional. Hard to address manually. Retailers need scalable solutions that work in all situations. In other words, a generic approach which is accurate in thousands of very specific cases: for specific products, for specific stores, for specific promotion designs under constantly changing circumstances! We developed a scalable AI solution that meets these objectives by combining 3 key innovations:
Innovation 1: Post-ROI measurement of 11 effects based on CLV
The basis of any promotion optimization is to understand the actual combined value creation of all intermingling effects. Our approach starts with traditional methods for comparing baseline sales with promotion sales to determine cannibalization, substitution and hoarding effects. But we take it a step further by analyzing promotion effects on a customer level. All these effects have been described in the first section. To combine all the 11 different effects into one ROI measure for the promotion, each individual promo effect has been quantified in terms of Customer Lifetime Value (CLV). The CLV impact of a promotion on each of the 11 effects is determined as follows:
Linking transactions to individual customers requires a unique identifier. This can be done via loyalty cards, but also via other means such as unique fingerprints of electronic payment details.
To calculate the ROI, a detailed view on both costs and margins is required. The creation of a comprehensive and detailed margin waterfall is not trivial for retail. It should include an accurate allocation of all costs, discounts and supplier contributions to an individual sold item.
Using CLV as a value metric implies taking a time view on customer purchases. For instance to determine the loyalty effect of new customers that were attracted by a promotion.
Innovation 2: Self-learning Pre-ROI optimization
To optimize promotions, a prediction is required for the expected performance across all promotion effects at a very granular level: for a specific product, for a specific promotion and for a specific promotion period. To do this well, a combination of different algorithms is used that learn from past promotions to make an ROI prediction for a new promotion.
Determining the expected value creation and ROI of a new promotion works as follows:
Determine historic consumer demand for past promotions. Lost sales can distort the actual promotion demand that was created by past promotions. In order to arrive at a fair and accurate prediction for future promotions, an accurate estimate of past demand is needed. To do this, a lost sales algorithm estimates how much more promotion sales could have been created at store level, for stores that experienced stock out during the past promotion. These lost sales are added to the actual promotion sales to arrive at an estimated demand for each past promotion.
The next step is to search through all past promotions and identify promotions that best matches the design parameters of the new promotion. A matching algorithm determines a matching score for each past promotion based on 15 different parameters such as recency, price, price mechanism, merchandizing, etc. The past promotions with the highest matching score are taken as a starting point.
The third step is to adjust the historic demand of the selected matches to a prediction of future sales based on the new promotion design. For instance, the new promotion could have a different price discount. A machine learning algorithm is developed to determine the impact of all promotion parameters on expected sales, based on learnings from all past promotions across products and stores. Based on these input parameters, the algorithm predicts the delta demand the new promotion will generate compared to the historic matches. Combining the historic demand with the expected delta demand we end up with the expected promotional sales
Besides the promotional sales, the most crucial (negative) driver for promotion success is the cannibalization effect, since discount is given to customers who would have bought the product anyway. To determine the size of this effect a baseline model is created predicting the regular sales per product on a weekly or even daily level. This prediction also takes into account various influencing factors such as seasonality, vacations and public holidays.
In the previous steps we determined the most dominant drivers that determine the direct sales uplift. To arrive at the total promotional effect, the remaining 9 effects from the past promotion are applied proportionally to the new promotion. In other words, these are assumed not to be influenced significantly by the promo design parameters, except for product type.
The last step is to convert the predictions to an ROI measure by combining all effects and determining the related revenue and margin. This leads to the expected revenue value effect and expected margin value effect. Balancing this against the promotional costs (e.g. for commercials and leaflets) we end up with the expected ROI, also called the pre-ROI.
Innovation 3: Model Management as unique human-machine collaboration
The third innovation behind our scalable solution for promotion optimization is to combine the strengths of algorithms with human experience and knowledge. Both algorithms and humans are fallible. We have taken an approach that allows both to work in tandem, exploiting the unique strengths of each. Algorithms are unbeatable when it comes to extracting patterns and relationships from data at almost unlimited scale. They can produce fact-based predictions for future results across thousands of products almost instantly. But they cannot predict what they cannot measure. For instance, low sales from a past promotion because of logistical issues are difficult to take into account if they do not show up in the data, or if the algorithm has not been designed to account for these circumstances. Moreover, the accuracy of predictions depends on the extent to which future promotions resemble promotion dynamics from the past. Humans can identify outliers and unusual situations quite easily. They have a broader context and often possess information which is not captured by algorithms, such as an upcoming strike next week. Our solution has been built to exploit the synergies between human and machine.
Model Management: improve performance of algorithms through human input
The prediction accuracy of all underlying algorithms is constantly monitored. This information is used to identify situations or segments for which performance can be improved. In other cases, accuracy is poor because of sheer variance of the promotion results. In other words: if there is no stable pattern, results become hard to predict. This happens for instance for outliers due to external factors, such as extreme weather, logistical hiccups, etc. Business users have the ability to identify these outliers, enabling the algorithms to discard them as training data. This prevents training data from being polluted by non-representative patterns. Gradual improvements of algorithms has become a joint collaboration between business users and data scientists.
Human-machine collaboration is not only vital for improving algorithms, but also for improving human performance. This symbiotic collaboration pans out in different ways. First, the promo matching algorithm gives a score as indication for its confidence which is translated into a green-amber-red indicator. Approximately 80% of all predictions has a green indicator, meaning they have high accuracy, and can be adopted without much consideration. This is crucial, since the prediction output is used to order quantities and determine stock levels. Amber predictions require careful consideration by an experienced category manager who can adjust the expected sales quantities. Red predictions generally mean that there is no to very little past promotion data to learn from. This leaves the category manager to adopt traditional gut-based methods for estimating sales uplifts.
Analysis of predicted sales quantities versus the adjustments by users reveals an additional feedback loop. It can either provide more insight into improvement potential for the algorithms. Alternatively, it can provide a feedback loop to the category manager that well intended adjustments are on average actually worse than just going by the algorithms’ predictions. This in turn helps to further establish confidence in the technology.
III. Building an AI-powered promotion capability with AIR-Pro
One solution to optimize promotions across the value chain
The approach to using AI to optimize future promotions by prediction CLV impact across 11 different promotion effects has been embedded in a user-friendly application called AIR-Pro: AI for Retail Promotion Optimizer. This application facilitates all workflow steps to optimize promotions, and connects to the retailers’ systems for promotion management and supply chain management. Various stakeholders and departments involved in the sales and operations planning process benefit from the enhanced intelligence.
As a Promotion Manager you can:
- Learn from previously successful and unsuccessful promotions.
- Optimize exposure of promotions by choosing the right marketing format.
As a Category Manager you can
- Negotiate the right deal with the supplier based on facts.
- Optimize individual promotions by choosing the right price, marketing format, products and other characteristics.
As Logistics department employee you can
- Purchase the right amount of stock and
- Divide it over your stores to minimize lost sales and over ordering for your stores.
As a Store manager you can:
- Make sure promotions are visible and attractive in store.
- Get more customers to the store because of the right promotions.
- Improve the availability of promoted products.
For example, figure 3 shows a workflow in which a supply chain manager can choose to adjust quantities based on the Green-Amber-Red prediction accuracy indicator.
AIR-Pro helped European supermarket to boost promotional ROI by 20%
One European grocery retailer has fully adopted AIR-Pro for optimizing their promotion decisions and sales forecasts. Based on the post-ROI insights from AIR-Pro, they have defined six different segments that define the success of a promotion based on net value creation (across all 11 effects). See figure 4. Promotions are segmented on net contribution to revenue (y-axis) and margin (x-axis).
The grocer used these segments to take concrete measures for improving their usual promotional calendar. For example, if a promotion is a ‘margin killer’, this means that the promotion realized only a small uplift in revenue and at the same time showed a negative margin value effect. For this type of promotions the category manager needs to understand what drives this disappointing effect and decide to either not play the promotion anymore, or adjust it to improve performance. This could mean a price increase or negotiations with the supplier, but also a different set of SKU’s for this promotion or some extra call to action in the brochure, a commercial or in the store itself. On the other side, ‘diamonds’ and ‘winners’ are the type of promotions that should be treasured, since they have a high impact on the total margin. A logical step would be to look for opportunities to repeat these promotions more often. A second step is to understand what drives the success of these promotions and replicate this to other promotions.
The supermarket used the post-ROI insights across the segments to do a top-down rationalization of their promotion portfolio: reducing promotions in poor performing segments and increase winning promotions. Furthermore, they adopted the pre-ROI optimization approach for all promotions. The optimization step has been fully established as a new common practice. Category Managers and Promotion Managers have gradually come to understand and collaborate with the prediction algorithms of AIR-Pro. Human and machine continuously challenge one another, jointly arriving at the best answers.
The results are undeniable. Figure 5 shows the impact on various KPIs from using AIR-Pro to optimize promotions.
Changes in KPIs between comparable periods with and without extensive use of data analytics
Food retail example
NB. Objectives (and therewith effects) differ per retailer and country. Insights enable retailer to steer on promotion KPI’s that requires highest uplift.