When we need to measure marketing campaigns, most of us have cracked online campaign tracking – we have a cookie or a user ID that we can use to clearly track user behaviour through to conversions, and develop robust datasets. But out in the offline world, there are powerful channels like TV and radio that marketers typically treat like a black box where cash goes in and very little data comes out. Long-term brand effects are similarly tricky to analyse.

But the good news is, you can measure anything if you have the right tools – and in these cases, you won’t even have to train an AI. All of the following solutions use tried-and-true statistical analyses to unlock insights you probably didn’t think you could access.

1. Immediate Response (TV or Radio Campaigns)

For channels and use cases where customers tend to use a device in parallel to seeing or hearing an ad, and where you know the exact timings of marketing activities, you can measure immediate response. With TV and radio, for example, you have data on when each individual spot airs, as well as traffic and user data from your web or mobile site or app. You can compare the airing times to user response immediately following the ad using a real-time statistical algorithm.

The KPIs: Web and mobile site visits, app usage or installs.

The method: Measure peaks in visits and installs that are significantly higher than normal, day-to-day traffic fluctuations, if they happen directly after an airing. Attribute this traffic to those individual airings.

What to watch out for: Make sure to filter the traffic so it’s limited to sources that have a high likelihood of coming from that specific campaign. This requires domain knowledge on how users arrive on your site or app. Also, traffic fluctuates for reasons other than marketing campaigns. Having a threshold in place that changes dynamically based on these fluctuations is crucial in deciding whether subtle changes are due to a spot airing or just noise.

2. Short-Term Response (Radio or Billboard Campaigns)

As well as immediately responding to marketing activities, customers can also respond with a delay of a few hours or days after they’ve seen an ad. This is the case for radio or billboard campaigns, where users might visit your website or use the app once they’re back at home, or they have some free time on the weekend. Their responses won’t show up in the above method for immediate response measurement, so we need a more refined model.

The KPIs: Web or mobile site visits, app usage or installs, or even brick-and-mortar store visits.

The method: With Bayesian modeling, you can analyse the time-series data of customer responses when you have some prior knowledge of what other factors might affect the campaigns. It needs to cover all possible effects, saturating trends, the change-points of the trend, multiple patterns of seasonality, and additional external events like holidays and product launches.

What to watch out for: This is a labour-intensive method that needs a lot of experience in evaluating models, as well as domain knowledge (e.g. knowledge of the channels and the business model). Without this, it will be difficult to define the best parameters and make sure you find the most accurate model within a reasonable amount of time.

3. Long-Term Brand Impact

Marketing campaigns don’t just drive users to your site or store in a one-off event – over time, they’ll contribute to the long-term value of your brand. Different channels may even have a cumulative effect on each other’s effectiveness, which makes this a challenging phenomenon to measure. To do this, you need a combination of different methods.

The KPIs: Long-term trends in brand awareness and affinity, but also conversions (e.g. purchases and app activity). Here you can focus more on further-down-the-funnel KPIs and higher-value conversions, which can then be used to analyse and optimise campaigns on the basis of customer lifetime value.

The method: A tried-and-true way to measure long-term brand impact is with market research. Here you can monitor consumers’ long-term awareness of your brand and their affinity to it within the target group or a general population. For a slightly more complex method, but one that provides deep insights, you can again use Bayesian models to capture all possible sources for trends and changes. First, clean your time series data of all repeating seasonal patterns or other events that possibly impact conversions, then check for long-term trends along change points, correlating them with known marketing activities.

What to watch out for: In addition to the short-term response, the model you use needs to explain long-term halo effects, where one channel will cumulatively increase the effectiveness of other channels. For example, TV advertising will also have an impact on display advertising and retargeting, which needs to be taken into account.

4. Cross-Media Campaigns

It’s rare that ads are running on just one channel at a given time – for example, you might supplement a TV campaign with radio, billboard, and display ads. To get a complete picture of all these activities, you’ll want to understand the impact of each of these different channels. Once again, Bayesian statistics comes to the rescue.  

The KPIs: The individual KPIs might vary per channel, but in this method you’ll look closely at the strength of the cross-correlation of all your marketing channels and then their individual correlation to visits and conversions.

The method: Similar to the above method, we can use a Bayesian regression analysis to understand the impact of each channel. By using hierarchical models (e.g. comparing historical data of typical sales patterns for a vertical with your own sales patterns), you can handle more complex models and take many different channels and factors into account. Again, you’ll need to include additional factors that have an impact on visits and conversions into the parameters of your model, such as holidays or weather patterns.

Hierarchical Bayesian models can deliver deep insights even with small datasets, which are typical in these contexts where you may have just one point of data per channel per day. Additionally, Bayesian statistics provides a measure of uncertainty for the model itself, to benchmark optimisation of the marketing channels. Recent work in this field has shown great results with this kind of modeling.

What to watch out for: Challenges in this sort of analysis are the limited amount of data, channels that are strongly correlated (like if a marketing manager books Facebook and display advertising at the same time), and having to identify the many individual factors that might have an influence on user behaviour.

What Else Can I Measure?

Long answer: There are so many tried-and-true statistical models at our disposal that can show us (hopefully strong!) correlations between our advertising activities and user behaviour. Bayesian statistics gives us a powerful method to measure the effects of any marketing activity on any user behaviour, so long as we have data – even when the datasets are small, and even when campaigns overlap. Also, this kind of analysis gives us additional information about how closely the model fits the actual data collected. This tells you how much you can rely on the results, so you know how confident you can be in making business decisions. The caveat is, you need deep domain knowledge of the industry, business model, marketing channels, and local markets to make sure your model is taking everything into account.

Short answer: You can measure (nearly) everything – you just need domain knowledge and the right tools.

Perhaps even moreso than even big data or blockchain, AI is fast becoming the buzzword on everyone’s lips. Machine learning has been a promising field for years, but with the astonishing success of deep learning techniques, we’re rapidly being propelled into an automated future. But can AI withstand the hype? What’s standing between us and successful large-scale adoption of ML and AI techniques?

Ahead of Data Natives 2018 on the 22nd & 23rd November, I talked to four key speakers in the AI space about the future of AI, and what transformations AI will (and won’t!) bring about.

On What the Public Needs to Know About AI

“AI needs data. If you have enough data, an AI can learn anything. This is stated by the universal function approximator theorem. Even a single hidden layer neural network can represent any mathematical function. Those functions are highly sought-after, since the allow us to control and predict nearly everything, ranging from a detailed psychological profile of an internet user to a complete model of the physical world. The individuals and corporations that own these models have power in their hands similar to a nuclear bomb. So let’s create clean energy from AI.”

Romeo Kienzler, Chief Data Scientist, IBM Watson IoT; read our full interview with Romeo here.

On AI’s Ability to Enhance Human Life

“AI is going to fundamentally change how we operate in our work lives, and our private lives. I see strong parallels to how personal computers revolutionized these spaces, freeing us from manual, repetitive tasks and allowing us to focus on more creative and strategy-intensive topics. AI will do this to an even greater degree, because it can actually support us in extremely difficult tasks, enhancing our most human features, intelligence and creativity. AI will become an omnipresent personal assistant in every aspect of our lives, maybe even developing real personalities. But this is maybe 10 years out still.”

Dr. Heiko Schmidle, Lead Data Scientist, DCMN

On Busting Through the AI Hype

“As most professionals know, there are two types of AI – the general AI and the narrow AI. Over time, it will probably be hard to draw a clear line between the two, but the today’s so called “narrow AI” is, in my view, just well-trained algorithms and, therefore, it should be actually referred to as such. Having said that, the term “AI” will certainly continue circulating around with respect to such algorithms at the very least for the marketing purpose. Nevetherless, I believe that the trained algorithms will become better and better exponentially. So, there are certainly many exciting innovations to be expected to emerge within just a couple of years. I, for one, am particularly curious about the merge of the machine learning and blockchain technologies, which can already be observed today. There are still not many companies that combine these two areas, but the potential of this, so to speak, collaboration is exciting and quite promising.”

Igor Drobiazko, CTO, elastic.io

On Combining Data Streams for Stronger Insights

“More data sets will be analyzed together for better insight. You’ll see bundles combining web, social, and Internet of Things (IoT) data in various combinations to see how different ideas from various areas of the online world weave together and influence one another. We’ll see, for example, how social media usage directly influences reliance on particular integrated IoT devices and vice versa. Eventually, the divide between different genres of data will erode and we’ll deal with data as a single entity.”

Justin Wyman, VP, Socialgist

Romeo, Heiko, Igor and Justin will all be speaking at Data Natives– the data-driven conference of the future, hosted in Dataconomy’s hometown of Berlin. On the 22nd & 23rd November, 110 speakers and 1,600 attendees will come together to explore the tech of tomorrow, including AI, big data, blockchain, and more. As well as two days of inspiring talks, Data Natives will also bring informative workshops, satellite events, art installations and food to our data-driven community, promising an immersive experience in the tech of tomorrow.