A/B testing Definition


A/B testing is a tool that can be used to improve different parts of a website.

What Is A/B testing?

A/B testing is a tool for evaluating and improving various aspects of your website. Simply put, an A/B test means that two different versions of the website (A and B) are set against each other. By setting and measuring specific goals for what the website should deliver and then test the different versions on randomly selected visitors, it is possible to see which variant gives the best effect.

You can, for example, test the website’s function, design or content and A/B testing is therefore a powerful tool, regardless of whether you work as a developer, editor or designer.

With tools like Google Optimize, you also do not need any advanced programming skills to get started with simple tests.

What is A/B testing?

With an A/B test, you can test which of two versions (A and B) performs best and get data that can help you make decisions. You go from guessing to knowing how users actually behave.

Say you want to increase sales on your site. You have several different thoughts and ideas about how you think you can improve your conversion, but you do not know which of all your ideas gives the best effect.

Is the buy button clear enough? Should the button be red or green? Is the title clear enough? Should you have one big picture or several small ones?

Instead of guessing, you can test two versions and see which one performs best. The one that gives the best conversion is the one you use when the test is over.

In fact, small margins or changes can sometimes be crucial. The data you collect through A/B tests can give you valuable insights that enable you to make informed decisions.

This in turn contributes to increased opportunities to achieve the goals of the investment, for example to get more people to convert.

What can you test using A/B testing?

You can do the tests concerning most elements on your website.

A common mistake, however, is that you want too much and are in a bit of a hurry. Therefore, be sure to test only one thing at a time.

Here are some examples of what you can try:

  • Landing page headings, which directly affect the conversion.
  • Forms – Visitors often feel that the forms are too long, so do not demand too much. Another mistake can be that the form blends in too well on the page. Make sure it attracts the visitor’s attention by testing clearer texts or an interesting image.
  • Design – change colors, clarify value words in bold and make your buttons clear and large enough to be able to click on in mobile.
  • Subject lines in e-mails – have a direct impact on open rate. If the email looks uninteresting already in your inbox, you will lose clicks.
  • Content – you can change headings and other wording. Swap images or add links.
  • Structure and layout – you can change the order of blocks or elements, delete fields that may not be needed or add explanatory texts.

How does an A/B test work?

In short, you take two solutions to the same problem and set them against each other. The winner is the version you publish.

To more clearly explain how an A/B test works, I have divided it into 4 different steps:

1. Hypothesis – How would it get better?

To pit two versions against each other, first develop a hypothesis. A hypothesis is based on an actual problem and which you think will lead to the effects you want.

“Users miss the buy button because it blends in with the rest of the design. This prevents users from converting. ”

“If we delete a field in the contact form, it will lead to 25% more leads”

2. Solution – We think this will get better!

Once you have defined your hypothesis, you make a change that tests the hypothesis. A change could be that you change the color of the buy button or delete a field in a form.

You now have two versions, an original as the website looks today and a trial version with some form of change.

The important thing when testing is not to make too many or too many changes at once. Then it becomes difficult to see what it was that gave or did not give effect.

At the same time, you may not see any noticeable difference in the result if you make too little change between the original and the trial version.

Or you try the wrong things.

3. Test – Collect data.

The new version is set up in parallel with the old one and then half of the visitors are allowed to see the new version and the other half the old one.

Visitors do not notice any of the test, which means that you avoid any external influences.

Keep in mind that you need a relatively large number of visitors and conversions to make safe decisions. If you have high traffic and many conversions on your website, it can be quick to collect data, while it takes longer for websites with few visits.

It is difficult to state an exact figure on how many tests should be performed to ensure a result.

If the result is crystal clear, you dare to make a decision earlier, but if there is a subtle difference, more time may be needed.

4. Analysis – What does the data show?

The variant that gives the best results wins.

Then you refine the winning variant and test again. Remember that for each test you learn something new and that you learn from your mistakes.

Dare to make mistakes and redo!

What is needed for a good A/B test?

Some things are easy to A/B test, such as a different color, text, order of elements, while functionality or major changes in the visuals require an agency’s skills.

It simply pays to possess knowledge of HTML, CSS and Javascript when you want to do various A / B tests.


In addition to this you need:

  • time – so that you see that the result has been consistent. A good piece of advice is to run your A / B tests for 30 days, corresponding to a business cycle for most companies, regardless of traffic volume. This way you get a representative selection.
  • patience – make one change at a time, so you can clearly see what is causing any changes in the conversion.
  • traffic – preferably 100 conversions per version or more to get statistical significance.
  • clarity – the changes must be perceived by the visitors included in the test.

Advantages of A/B testing

  • You test on real visitors and get hard data that supports one or the other option.
  • Your visitors do not notice the test, their behavior is not colored by the fact that they are “test persons”, which may be the case in, for example, focus groups or in-depth interviews.
  • The results can give you new insights into how your visitors behave and what works and does not work. This allows you to make more informed decisions and optimize your website.

Disadvantages of A/B testing

  • A/B tests take time. You must have a hypothesis about an improvement and then develop two versions to test. Sometimes there is hardly any time to develop a single alternative.
  • In order to be able to draw certain conclusions, a relatively large number of visitors and transactions are required. With too little data, there is a great risk of drawing hasty conclusions.
  • Even if you have designed a good test, it does not automatically mean that the results are useful. The test result may not show any significant difference between A and B.

Strategy for A/B testing

One of the most important things about A/B testing is getting started. After a few simple, trying tests, a test plan is made for the coming quarters.

The plan is based on hypotheses based on experience, own analyzes, internal workshops or the help of an agency. Then you test the hypotheses that you think may have the greatest impact on the conversion but are easiest to test.

When some A/B tests have been run, you review the statistics and modify your test plan. In this way, it becomes easier to perform tests that achieve better results and then implement the improvements on the website.

If you start at the other end, you often get a big uphill climb with few rewards and a slow development. This hampers the opportunity to bring employees and the organization to work.

Possible risks with A/B testing

If you are careless, the A/B tests can create bugs on your site. These can be easily avoided if you use software such as Optimizely or Visual Website Optimizer, which allows you to preview changes before putting them live.

Another aspect is not to A/B test changes you do not plan to implement, such as removing headlines that are important for search engine optimization (SEO).

Some quick tips on A/B testing

  • Buying behavior is often affected by external factors, such as the time of year. Take that into account.
  • implement the winning version in the A/B test 10-20 times a year, it has very little significance that some versions did not give an increase in practice.
  • do not settle – continue A/B testing changes before implementing them, keep optimizing your site! It pays off over time.

Conclusion

In conclusion, it can be said that A/B testing can be a valuable tool for optimizing your website and a way to help you make informed decisions.

It is also important to see the web as something alive and constantly evolving.

But just because you got a successful result and implemented the winning version from an A/B test, does not mean that the work is finished.

Just as your customers’ needs change over time, your website should continue to evolve. By iteratively testing, improving and refining elements, you can continuously optimize your website.

Sources

https://www.researchgate.net/profile/Ron-Kohavi/publication/316116834_Online_Controlled_Experiments_and_AB_Testing/links/59b7583b458515c212b3cd46/Online-Controlled-Experiments-and-A-B-Testing.pdf

https://dl.acm.org/doi/abs/10.1145/2783258.2788602

https://dl.acm.org/doi/abs/10.1145/2736277.2741081

https://books.google.se/books?hl=en&lr=&id=VfVvAAAAQBAJ&oi=fnd&pg=PT6&dq=A/B+testing&ots=T0TREi2_i9&sig=nFm_qbp0UqlR7O9VtAW7Hh0xeQ8&redir_esc=y#v=onepage&q=A%2FB%20testing&f=false

http://proceedings.mlr.press/v35/kaufmann14.html

https://books.google.se/books?hl=en&lr=&id=eM-PDgAAQBAJ&oi=fnd&pg=PR9&dq=A/B+testing&ots=AKwgP37lEq&sig=Kk-SMN2LOi-aapB2joBemcv9mh4&redir_esc=y#v=onepage&q=A%2FB%20testing&f=false

https://arxiv.org/abs/1512.04922

https://ieeexplore.ieee.org/abstract/document/8240912

Kevin

This article has been reviewed by our editorial board and has been approved for publication in accordance with our editorial policies.

Recent Posts