The experts' best tests

Hear the tricks of the trade from CRO experts in The Optimize CRO Series.

Finding out the test results from other companies is like opening a treasure box of insights! In this episode of the Optimize CRO series, we ask the CRO experts what their best tests have been. Dive in, and get inspiration for your next A/B test.

Dr Karl Blanks and Ben Jesson, founders of Conversion Rate Experts

Dr Karl Blanks and Ben Jesson, founders of Conversion Rate Experts and authors of Making Websites Win
United States

“Using the process described in the previous article about frameworks, we created a new landing page for the FinTech company goHenry. Here are just some of the proven techniques we used:

  1. We had a long story to tell, so we created a long page. The mobile visitors were intrigued but had many strong objections—as you would expect when selling a new financial product. A long page allowed us to systematically address all of the key objections, starting with the strongest. Then, as each objection was addressed, we gave the prospect an opportunity to proceed through the funnel by sprinkling the call to action throughout the page. This effectively made the page as short as possible, but as long as necessary. This was particularly important on mobile. Because there’s no such thing as a too-long page—only a too-boring one. So we had to ensure that the page was engaging from top to bottom.
  2. We used visual ways to present the information. Our research revealed that mobile visitors landed on the website with only a vague idea of what was being offered. Coupled with their strong objections, we were dealing with seriously volatile visitors, liable to leave the website at any time. To get our message across quickly, we presented the information in the form of images and diagrams.
  3. We built trust and credibility—fast. Selling a new financial product is hard. Buyers understandably have issues with trust and credibility. We needed something that would wipe out trust issues fast. The answer lay in the fact that goHenry is backed by Visa, one of the world’s most well-known and trusted financial institutions. By positioning Visa much more prominently as part of the proposition, showing it even more prominently than the goHenry brand, we reassured visitors that goHenry was associated with a brand they already trusted.

The new page test increased sign-ups by 78%. That’s from a single test. Overall, in a single year, we helped goHenry to grow by 470%.”

Gino Arendsz, Growth and E-commerce Manager at Helloprint

Gino Arendsz, Growth and E-commerce Manager at Helloprint
Netherlands

“Based on earlier experiments we learned that recommending products from within the cart had a positive effect on transactions. This made us wonder if there are other ways of helping customers adding more products to their carts and convert better?

We came up with an add-to-cart notification–if you add something to your cart, we will tell you with a little pop-up at the top right corner that we’ve added the product to your cart instead of actually going there (having a redirect). It’s quite a common practice in Ecommerce, but we did not have it in place. It ran on all devices and across all of our domains.

The experiment resulted in a conversion rate uplift of 3.02% with a 12-month business value of €693.000 (Bayesian confidence of 92.2%). This experiment also resulted in a 2.19% uplift in average order value, another indicator that this helped our customers adding more products to their cart without giving in on, for example, discounts.”

See screenshots of the test from Helloprint in the A/B test results library in the Optimize Resource Hub.

Lorenzo Carreri, Optimization and Growth at CXL Agency

Lorenzo Carreri, Optimization and Growth at CXL Agency
United Kingdom

“One of my clients is an educational institution. An interesting insight that came up from different data points during our initial research process was that first time visitors were not sure how much their main course cost. And the only place where the price was mentioned was at the checkout page, at which point the drop-off rate was of course quite high.

So we decided to run a test where we basically started talking about the price of the product at the beginning of the funnel, along with some benefits on why the user should buy the product.

After 28 days the treatment was a clear winner with over 11% uplift in conversions (95% confidence level) which corresponded to an incremental 7 figures in revenue.

The test ran on Desktop only as I like to run tests on different devices separately, because the mindset of the user is different. Considering this big success, once the enrollment for the course opens again in a few months, we will run the same type of test on Mobile and Tablet.

I now keep a Post it on my monitor with the results of this test as a reminder of the importance of research during the testing ideation process.”

James Flory, Senior Experimentation Strategist at WiderFunnel

James Flory, Senior Experimentation Strategist at WiderFunnel
United States

“We ran such a simple experiment on a client’s website—a well-known media and financial investing company. The website features tons of free content, but they also offer premium investing services. In the experiment, we placed a primary call to action in the top navigation bar to direct users to the paid services they could sign up for. It resulted in a site wide 11.2% increase in transactions. Huge!

Although it was a simple idea, it was a case study in the value of an external partner’s perspective and harkens back to putting yourself in your users’ shoes. Our initial analysis had shown how hard it could be to find this organization’s paid services, and because the internal team was so deep in their world they hadn’t come to the same conclusion.

Read more about this experiment (and find screenshots) here.”

Melanie Bowles, Specialist in Optimization and Insights at InfoTrust

Melanie Bowles, Specialist in Optimization and Insights at InfoTrust
United States

“An A/B test that resulted in great results was for a company that sells products that appeal to “makers”. They had amazing “how-to” video content that was heavily viewed in a gallery but visitors often left the site after viewing them. So we created a version of the product detail page that included related videos that inspired users with project ideas for the product they were viewing. This resulted in great improvements in product detail page engagement, add to cart, transactions and revenue.”

Patrik Matell, Conversion Optimization Specialist at Conversionista

Patrik Matell, Conversion Optimization Specialist at Conversionista
Sweden

“When working with a subscription based website, we were doing testing on payment option based on lifetime value data. We were limited in test runtime and we needed a strong signal, so it was decided to do a redesign aimed at getting the users signing up for a larger package as well as using automatic payments. We knew we would potentially lower overall conversion a bit in favour of increased lifetime value revenue. But the result was both worse, and in the end, better than we expected.

Total conversion was down 18%. But the average order value was substantially higher, and a majority of the conversion were using the preferred payment mode. After inputting the data into the predetermined lifetime value models, the variation was going to earn them 20% more than the original, and we could breathe a sigh of relief.”

Extra treat: These were the experts’ most surprising test results

Dr Karl Blanks and Ben Jesson, founders of Conversion Rate Experts

Dr Karl Blanks and Ben Jesson, founders of Conversion Rate Experts and authors of Making Websites Win
United States

“We discovered the following phenomenon with one of our first clients. We designed a new page that was clearly much better. It felt like we had “cracked the walnut with a sledgehammer.” We had usability tested it, and the users were emphatic that it was much more persuasive. The page contained mountains of proof and the product’s benefits were explained better. Then we ran an A/B test, and it didn’t reach significance. We were confounded.

Then, we realized where we had gone wrong. On any given day, about 95% of the company’s customers were existing customers. Sometimes a company’s existing customers are subscribers who are logging in (as with a SaaS company). Sometimes they are people who buy the product on an ongoing basis and who are returning to the site simply because they have run out and needed to reorder—and who will reorder regardless of how the product page looks. We realized that no matter how persuasive our new page was, the only thing that it could do was to increase the number of new customers—the green guys in this image:

If many of your customers are returning customers, then you need to filter them out when you’re optimizing a page that converts visitors into customers.

We ran the test again, but this time we measured the number of new customers. We discovered that our new page was generating 114% more new customers than the control. So we had more than doubled the conversion rate of new customers to the site. But our success had initially been hidden in the noise of repeat customers.

Whenever you try to increase the rate of new customer acquisition, make sure that you track the new customers.”

Iqbal Ali, Head of Optimization at Trainline

Iqbal Ali, Head of Optimization at Trainline
United Kingdom

“We had an experiment where the variation was winning for a number of weeks before over the space of a weekend, the results flipped and the variation ultimately became a loser. When we analysed why this change had occurred, we realised that the “flip” corresponded with some major train network issues.

It took a while to even recognise this external factor could have been to blame, instead we focused efforts on looking for defects in the codebase etc.”

James Flory, Senior Experimentation Strategist at WiderFunnel

James Flory, Senior Experimentation Strategist at WiderFunnel
Canada

“I ran an experiment on the homepage of an e-commerce company, inspired by an extremely successful experiment I had run on one of their key landing pages. For the homepage test, we designed a hero area that looked nearly identical to that of the performant landing page. The redesigned homepage hero had a different header and new imagery showing a user interacting with the product. There was one caveat: The homepage hero had an element that the performant landing page didn’t have: A video link. We accommodated the video link in the hero redesign and launched the test. It resulted in a nearly -11% decrease in transactions.

The surprising result was that a video link which had been re-styled and accommodated in the new hero area design received a 154% increase in engagement—an entirely unintentional result. Way more people were watching the video, but fewer people were converting.

This led to a new hypothesis: Video views may be negatively correlated to conversions in this funnel. We discovered there were prominent video links in the hero area across many of this organization’s key product pages, so we restyled the links to be far less prominent and replaced them with a simple, static value proposition of the product. The result was a 33% increase in conversions. The key takeaway? If your tests are structured properly, your “losing” tests can yield fantastic results, when you break them apart to figure out the why behind a result.”

Ayat Shukairy, co-founder and CCO at Invesp

Ayat Shukairy, co-founder and CCO at Invesp
United States

“With every experiment we run we estimate the uplift we expect to see. So it’s a very exciting experiment for us when we expect a 10% increase and end up seeing 60%. The experiment I’m highlighting is for a subscription based site. While we all know what a positive impact video has on conversion, the video they had featured in the header, the placement of it, and the contents, were not delivering. Through our review we noticed visitors were not scrolling past it and just bouncing off of the page at alarmingly high rates. We decided to keep access to the video, but through a link, change up the header to something statics with a more compelling headline. We estimated that the change would result in a 10% uplift. Boy were we off. The change resulted in a 60% increase in subscriptions.”

By Lina Hansson

Next steps

Do you also want to find gold in the form of great A/B test results? Here’s how you can start:

And of course, start testing your way to success in Optimize today.

Was this helpful?
How can we improve it?
Search
Clear search
Close search
Google apps
Main menu
Search Help Center
true
101337
false