Learn How a Split Sample Test Affects Market Research

Team reviewing hybrid market research data.
Jupiterimages/ Stockbyte Collection/ GettyImages

Split sample testing, also referred to as split-run testing or A/B testing, is a term used to describe an experiment with two variants or options. In marketing, this form of research is often used to test audience reactions to visuals or campaigns and determine which option generates the most activity.

One of the most common uses is with web design. Two forms of the website are created, with some isolated differences.

The sites are monitored to determine which version gets the most activity, such as the most impressions or highest levels of engagement. Split sample testing can help companies identify which site is most effective.

Recent History of Split Sample Testing

Split sample testing is relatively new. Google first used A/B testing in 2000 to evaluate the ideal number of search results on a page. Improvements were made based on the test and continuously developed. Now, Google runs over 7,000 A/B tests every year.  

How to Design a Split Sample Test

To create your own split sample test, follow the below steps:

  • Create a plan: Identify your goals, such as improving link clicks or more page views
  • Pick a variable: Choose variables, such as one element on the site that will appear horizontal on the "A" site and vertical on the "B" site. 
  • Analyze results: Using Google Analytics or other tracking tools, review the results of the test, comparing engagement levels or views on each site. Depending on the results, you can see if one version performed better than the other, or if the results were relatively static. 
  • Implement changes: Make changes based on the first test and redo the test to confirm the results. 

Common Mistakes

Many people go into split sample testing with too many variables, ruining the power of the experiment. If you have too many differences, you cannot identify the factors that improve consumer behavior.

To drill down to what differences make an impact, you need to isolate components one at a time. This process takes longer, but it will give you a more accurate picture of how to increase conversion rates or drive consumers to action. 

Many researchers also do not run the test long enough. You likely will need about one week to get an accurate depiction of customer behavior. Otherwise, you risk your results being not statistically significant, muddling the outcome. 

You also need to ensure a wide sample base with different genders, races, and ages to get an idea of how each version performs. If you target a very specific audience, you will want to include a large number within that group to give you useful information. 

Split sample testing is a useful tool in making changes to your marketing approach. However, it's important to keep in mind that A/B testing should be done often, not just once and forgotten. As technologies and perceptions change, attitudes towards your design and products can change too. Running tests continuously will allow you to evolve along with your target market. Testing and improving is a constant process that must be repeated to stay up to date.