A Step-by-Step Guide to Incrementality Testing for CTV Advertising

A Step-by-Step Guide to Incrementality Testing for CTV Advertising

Learn how to determine if your CTV advertising campaign is producing ROI

Sophisticated digital advertisers use incrementality testing to help isolate unique contributions for their many digital marketing channels.   As these marketers move into CTV as the next growth channel, incrementally testing is especially important because CTV is unique in the digital advertising domain because consumers cannot click on TVs.    Incrementality testing is a foundational way for marketers to go "beyond last clicks attribution", which of course is a requirement when there is no last click.

From defining your audience to calculating incrementality, this article will show you how to apply this methodology to the fastest-growing video advertising channel in the world.

What Is Incrementality Testing?

Incrementality testing in advertising is an approach to ad performance measurement that allows marketers to determine the exact level of contribution — or “lift” — that a channel’s advertising has on audience behavior. The approach revolves around using two groups: the test group and the control group. The test group is exposed to the advertisement. The control group is not.

If the exposed group performed better, then the test has shown a positive incremental lift. If the control group outperformed the exposed group, then the test has shown a negative incremental lift. If there is no difference in the results, the test is deemed neutral. A difference between the results is only consequential if that gap is statistically significant.  

Incrementality testing is a type of A/B testing, but instead of determining which is preferable between two alternative advertisements, incrementality testing determines the value of advertising at all. In other words, incrementality testing is about understanding whether advertising on a given channel works — and exactly how much it works.

How to Run Incrementality Testing for CTV Advertising

Step 1: Create a Hypothesis and Establish Performance Metrics

It’s surprising how often advertisers can jump into any testing process before they’ve clearly established what it is they’re trying to test or how they’re going to assess success. With incrementality testing, you’re going to uncover whether an advertisement works better than no advertisement. This means you could run a test using messaging and creative you believe is effective, but you could also run a test with placebo or dummy ads, which would measure the fundamental value of the channel.

On top of generating your hypothesis, you also want to agree on the performance metrics you’ll be using to evaluate that hypothesis. These are usually metrics like purchases, visits, average order value (AOV), conversion rate (CVR), and visit rate.

Step 2: Define Your Testing Audience

Next you need to build a testing audience derived from your overall target audience. This testing audience will contain the control and the exposed group. When you’re defining this subset, you want to account for any variables that could have an impact on the outcome of your test. If, for example, you’re advertising a luxury product, it would likely be a good idea to hold income constant across the groups. Without accounting for these variables, the results could be erroneous — as would be any actions taken as a consequence.

For advertisers looking for precision audience control during testing, tvScientific’s CTV advertising and attribution platform provides numerous targeting parameters. These include age, gender, education, income, household, zip code, brand affinity, online purchases, offline purchases, viewing habits, device, business vertical, and fifteen thousand more segments. This level of granularity provides advertisers with complete control over their incrementality testing process.

Step 3: Determine Timeframe and Ad Content, and Run Your Test

While the timeframe of the test depends on the volume of interactions — not any specific amount of time — it’s accepted best practice to run your experiment for a minimum of a week. Generally speaking, you should also consider running your test during a relatively normal time of the year. Holidays, elections, and any other major event could impact your audience’s state of mind in unpredictable ways and undermine the validity of your results.

Now, you need to develop ad creative based on the hypothesis you’ve set out to test. If you’re looking to measure the raw, incremental value of CTV advertising, you need to run placebo creative during your test. This creative needs to be irrelevant to your business, whether it’s a placebo or a public service announcement (PSA) ad. By running this kind of test campaign, you’ll be able to get an accurate handle on the baseline value of CTV advertising.

With these done, it’s go time: Flight your ad to your test group while suppressing the control group’s exposure.

Step 4: Analyze Test Data and Calculate Incrementality

Once your test is finished, collect all the campaign data from your two groups and review them side by side. From visits to CVR, analyze all your performance metrics and use them to calculate the incremental lift, as an overall percentage, that the test group saw versus the control group. Using this percentage, you can then calculate the incremental cost per acquisition (iCPA), conversion value, and return on ad spend (iROAS). With these figures in hand, you now have an exact read on the value CTV advertising brings to your business.

Finding a Platform that Supports Incrementality Testing

While the impact of traditional TV ads has always been difficult for advertisers to pin down, the arrival of CTV advertising means those days are behind us. Now, advertisers can measure the value of their ads with the kind of precision that had only been possible on digital channels. They no longer need to wonder if their TV ad campaigns are working: All they need to do is run incrementality testing on their CTV advertising platform to find out.

Advertisers looking for a CTV advertising platform that provides a comprehensive testing toolkit should consider tvScientific. The world’s first CTV advertising and attribution platform, tvScientific provides a robust, built-in incrementality testing feature. This feature not only allows advertisers to run ad hoc tests to assess channel value, but it also runs continually to adjust for the lack of ad exposure data via walled garden environments. The only platform on the market with this always-on feature, tvScientific was designed to ensure advertisers are consistently getting value out of their CTV campaigns — a commitment it also backs up with a fraud-free guarantee.

Reach out to the team to learn more, or get started with your first CTV incrementality test!