Before A Beats B, Check Who Saw Each Page

A small team wants to know whether a shorter form converts better. It duplicates the landing page, cuts the form from seven fields to four, and points a new ad set at the new version while the old ad set keeps sending people to the original. Two weeks later the short form is ahead. The team moves all its spend to it, and the leads that follow look nothing like the test.

Nothing was wrong with either page. The test compared two audiences that happened to land on two pages. The new ad set had its own targeting, its own learning period and its own placements, so the people who saw version B were never the people who saw version A.

That is the question to settle before you buy an optimization tool or download our AI landing page optimization playbook: is your traffic steady enough for a comparison between two pages to mean anything? AI tools can move visitors between versions faster than a person can. That makes the question more pressing, for reasons I'll get to below.

A test compares visitors as much as pages

A conversion rate belongs to a page and a visitor together. Someone who searched for your exact product name behaves differently from someone who scrolled past a video, whatever page you show them. A fair comparison needs both pages to see the same kinds of visitors, in the same proportions, over the same days. Random assignment behind a single URL does that for you. Separate ad sets, separate campaigns and before-and-after comparisons don't.

Before-and-after is the most common version and the easiest to fool yourself with. Change the page on the first of the month and compare it to last month, and you have also changed the season, your competitors' ad spend, whatever your sales team did that week and whatever the ad platforms changed in their auctions. The page is one variable among many, and the report gives it all the credit.

Why adaptive tools make the mix matter more

Tools that allocate traffic adaptively, often described as multi-armed bandits, send more visitors to whichever version is ahead at the moment. That is their appeal: fewer visitors spent on a loser. It also means each version's share of traffic changes over time. If the kind of traffic you receive changes over time as well, say weekday search visitors give way to weekend social visitors, then the version the tool happened to favor on your best days collects credit that belongs to the days.

Some tools try to correct for this, and it is fair to ask any vendor how theirs does. My point is simpler. A tool can only be as fair as the traffic you feed it. If budgets, audiences and creative change every few days, freeze what you can while a test runs, or accept that the result describes that particular stretch of weeks and nothing more.

Five checks before you trust a winner

1. One URL, random assignment. Both versions should live behind the same link, with the tool deciding who sees which. If you are testing with separate ads, you are testing the ads.
2. The same sources on both sides. Break the result down by traffic source and by device. If one version got most of the mobile social visitors and the other got most of the desktop search visitors, the headline number isn't about the page.
3. A conversion you would pay for. Count the lead that reaches your CRM with a working email address, not the button click or the thank-you page view. A shorter form can collect more submissions and fewer usable ones, and only the CRM can tell you which happened.
4. Whole weeks. Run tests across full weeks so each version sees weekdays and weekends. Stop early only if you wrote down in advance what result would make you stop.
5. A written guess. Before the test starts, write down which version you expect to win and why. When the result surprises you, that note is the difference between learning something and simply picking the bigger number.

When your traffic won't hold still

Plenty of small accounts can't freeze their budgets for two weeks, and some landing pages will never collect enough visitors for a clean comparison. That doesn't mean you stop improving the page. It means you change the method.

• Make one larger change at a time, such as a different offer or a different form, instead of several small ones you can't tell apart.
• Fix what a five-second look reveals before testing anything: a slow page, a form that breaks on a phone, a headline that doesn't match the ad that sent the visitor.
• Judge the change on lead quality in your CRM over a longer period, instead of declaring a winner after a few days of clicks.
• Keep a dated log of every change to the page and to the campaigns pointing at it, so a jump in the numbers can be matched to something that actually happened.

The guide has a section on the minimum traffic AI optimization needs and on alternatives for pages that don't reach it. If your landing page gets a modest number of visitors, read that section first. It may save you from paying for a tool that needs more data than you have.

Write every test up on one page

The cheapest improvement to any testing program is a one-page note per test, written before it starts and finished when it ends. It turns a result into something the next person can check, and it makes the five checks above a habit instead of a post-mortem. Mine would have these lines:

• Change. What differs between the versions, in one sentence.
• Guess. Which version you expect to win, and the reason.
• Dates. Start and end, both on the same weekday.
• Traffic. Which campaigns and sources were live, and anything that changed during the test.
• Conversion. The exact event counted, and where it was counted.
• Result by source and device. Not only the total.
• Decision. What you shipped, and what you would test next.

A stack of these notes is worth more than any single result in it. After a few months you'll see which kinds of change tend to matter for your visitors and which never move anything, and that knowledge stays with the company when the person who ran the tests moves on.

Where to go from here

Our free guide, AI Landing Page Optimization: The Compounding Conversion Playbook, covers the mechanics of adaptive traffic allocation, which page elements to test first, a comparison of Unbounce Smart Traffic, VWO, Mutiny and Intellimize for B2B teams, and a testing calendar. Before you read it, run the five checks on the last test your team called a win. If the winner survives them, you are ready for faster tools. If it doesn't, you've found the first thing to fix, and fixing it costs nothing but a meeting with whoever runs your ads.

← Back to all articles