The marketer's guide to holiday testing

Holiday testing means running controlled experiments, usually A/B tests, on your seasonal campaigns so you can put your budget behind the version that performs best. The smartest time to do most of it is before the peak, when mistakes are cheap and there is still time to act on what you learn.

The holiday season raises the stakes for marketers. Budgets are bigger, inboxes and ad auctions are more crowded, and B2B teams often run end-of-year offers aimed at buyers with budget left to spend. Testing replaces guesswork about what will work with evidence.

What types of holiday tests can you run?

Two types cover most needs.

A/B testing

An A/B test, also called a split test, compares two versions of something, A and B, by showing each to a random part of your audience. You then measure which version does better on the metric you chose.

A/B tests suit single, clear questions: which subject line, which offer, which call-to-action button, which landing page headline. If the two versions differ in one element, you know that element caused the difference. For more, see our guide to split testing.

You can also A/B test bigger choices, such as two campaign concepts or two audiences. Just accept that you will know which version won, not exactly why.

Multivariate testing

A multivariate test changes several elements at once, such as headline, image and button, and compares every combination. It shows which combination works best and how the elements affect each other.

The catch is traffic. Every extra element multiplies the number of versions, and each version needs enough visitors for a reliable result. Multivariate testing is usually only practical on high-traffic pages.

When should you run holiday tests?

Run most tests in the weeks before your peak. There are three reasons:

1. There is time to act. A winning subject line style or offer can be rolled out across the whole season.
2. Mistakes cost less. A losing variant during your busiest week costs real revenue.
3. Peak traffic is unusual. During the holidays people buy with more urgency and hunt for deals. What wins on a peak day may not reflect normal behaviour, and the reverse is true too.

During the peak, keep to a few large, low-risk tests, such as two versions of your main offer email. After the season, retest any finding you want to keep using, such as a pricing message, in a normal period.

How do you build a holiday testing plan?

1. Set a goal and a hypothesis

Decide what the test should improve: sales, demo bookings, trial sign-ups or revenue per email. Then write a hypothesis, for example: "A dollar-amount discount in the subject line will get more clicks than a percentage discount." A hypothesis keeps the test focused and makes the result easier to use.

2. Choose one variable

Common holiday test variables:

• Subject lines and preview text
• The offer itself: percentage off, amount off, free shipping, a bonus or an extended trial
• Send day and time
• Call-to-action wording and placement
• Landing page headline or layout
• Ad creative and audience

Choose the variable most likely to move your goal.

3. Work out how much data you need

This is where most tests go wrong. The sample size you need depends on three things: your current conversion rate, the smallest improvement you want to detect, and how sure you want to be. Small differences need large samples.

For example, if 2% of recipients currently buy and you want to reliably detect a rise to 2.5%, the standard two-proportion calculation at 95% confidence and 80% power needs roughly 13,800 recipients per version. Detecting a bigger jump, from 2% to 3%, needs about 3,800 per version. If your list or traffic is smaller than that, test bigger differences, such as two very different offers, rather than small wording changes.

Use a sample size calculator before you launch, not after.

4. Split randomly and run the full test

Send each version to a random group, at the same time, so the only difference is the variable. Decide in advance how long the test runs, and don't stop it the moment one version pulls ahead: early leads often disappear. For website and ad tests, run whole weeks where you can, because behaviour on weekdays and weekends differs.

5. Measure the right metric

Pick the metric that matches your goal and judge the test on it.

• Email subject lines: use clicks or conversions, not opens. Apple Mail's Protect Mail Activity setting loads email content in the background, so senders cannot tell when a message was really opened.
• Calls to action and offers: use click-through and conversion rate.
• Ads and landing pages: use conversions and cost per conversion.

Also watch guardrail metrics, such as unsubscribes, spam complaints and refunds. A subject line that wins on clicks but doubles complaints is not a winner.

6. Pick the winner and write it down

Roll out the winning version, then record what you tested, the result and the sample size. Next year's holiday plan starts from this log.

Which tools can you use?

• Email platforms. Most email marketing tools have built-in A/B testing for subject lines and content, and can send the winner to the rest of the list automatically.
• Google Ads. Custom experiments test changes on a copy of a campaign that shares the original's traffic and budget, split evenly between control and test. They are available for Search, Display, Video and Hotel campaigns, and only one experiment can run at a time.
• Website testing tools. Google Optimize, which many teams used for free website tests, was shut down on 30 September 2023. If your old testing process relied on it, you need another tool.

How do you protect deliverability during the holidays?

Holiday sending volumes rise, and so does the risk of landing in spam. Google requires anyone sending 5,000 or more messages a day to Gmail accounts to authenticate with SPF, DKIM and DMARC, offer one-click unsubscribe in marketing messages and keep the spam rate reported in Postmaster Tools below 0.30%. Check these before your biggest sends, and don't email old, unengaged contacts for the first time in months on your busiest day.

Quick tips for better holiday tests

• Start early. More time means more tests and more data.
• Plan the calendar. List every test, its hypothesis and its end date before the season starts.
• Keep it simple. One or two well-planned tests beat ten underpowered ones.
• Be patient. Let tests reach their planned sample size.
• Act on the result. A test only pays off if the winner gets rolled out.

For more on preparing a seasonal plan, read how to start marketing a seasonal business and our guide to conversion rate optimization.