A/B Testing

Quick definition

A/B testing is a controlled experiment in which two versions of something, such as a web page, email, ad or button, are shown to similar groups of people at the same time, to see which version performs better against a chosen goal.

How an A/B test works

  • Choose one goal metric, such as click-through rate or sign-ups.
  • Change one thing, such as the headline, image or button wording. Version A is the original (the control) and version B has the change.
  • Split the audience randomly so both groups are comparable.
  • Run both versions at the same time for a period decided in advance.
  • Compare the results against the goal, then decide which version to keep.

Getting trustworthy results

  • Do not stop early because one version happens to be ahead. Early differences often disappear.
  • Change one element at a time, so any difference can be traced to it.
  • Make sure the split is random and both versions run over the same period.
  • Check whether a difference is large enough, given the amount of data, to be more than chance. A small gap on little data is often noise.
  • Write down the hypothesis and the result, whichever version wins.

A/B testing and multivariate testing

A/B testing compares whole versions. Multivariate testing varies several elements at once to see how they combine, and needs substantially more traffic to give reliable results.

Example

An email's subject line is tested. Version A reads “Your spring checklist” and version B reads “5 spring tasks to do this weekend”. Each goes to a random 10% of the list, and the version with the higher open rate is then sent to the remaining 80%.

Related marketing terms

Related KPIs

Related tools

Campaign URL Builder

Marketing

To tell test variants apart in analytics, give each variant's link its own utm_content (or campaign) value. The builder names and tags the links; it does not split traffic or run the test.