How Do You Judge a CRO Agency Before You Hire One?

You judge a CRO partner by its full test log, losers included, and by how it plans each test. A good partner sets sample sizes from your traffic before launch, does not stop tests early, measures winners against revenue and builds tests that leave your store and search rankings intact. A conversion rate optimization agency that works this way will also tell you when your traffic is too low to test at all.

This guide covers only what is specific to testing partners. For shortlisting, references and contract terms, start with choosing an ecommerce marketing agency. Platform details are as of 27 October 2026.

Why should a CRO partner show you its losing tests?

A CRO partner should show you its losing tests because most experiments lose or change nothing, so a record of nothing but wins means results were selected, stretched or stopped early.

The largest testing programs report humbling numbers. Microsoft's experimentation team found in 2009 that only about one-third were successful among well-designed experiments built to improve a key metric.

Later estimates are lower. Ron Kohavi, Alex Deng and Lukas Vermeer, in a 2022 paper for the KDD conference, put success rates at about 15% for Bing and 10% for Booking.com, Google Ads and Netflix. They call these "ballpark estimates" built on different accounting schemes. Kohavi is also lead author of Trustworthy Online Controlled Experiments, written with Diane Tang and Ya Xu, the standard reference on the subject.

A fund manager who shows you only the stocks that went up has told you nothing about the fund, and a testing partner is no different. We would ask for the last ten tests, losers included, each with its hypothesis, sample, result and lesson. A partner with no losers on file is either not testing or not reporting them.

How much traffic does a store need before A/B testing pays?

A store needs enough visitors on the tested page to detect the smallest lift worth having, and for a modest lift that often means tens of thousands of sessions per variant, and far more for a small one.

Evan Miller's sample size calculator defines the inputs plainly. The minimum detectable effect is "the smallest effect that will be detected (1-β)% of the time." Statistical power is the "percent of the time the minimum effect size will be detected, assuming it exists." Optimizely's documentation treats the minimum detectable effect as a cost decision and tells teams to consider whether the traffic and time is worth it.

Working the numbers for one product page

These figures are illustrative, worked from the formula in the Kohavi, Deng and Vermeer paper at 80% power and 5% significance. A product page converting at 2.5% needs about 62,400 visitors in each variant to detect a 10% relative lift.

Halving the lift you want to detect roughly quadruples the sample. Detecting a 5% lift on that page takes close to 250,000 visitors per variant.

The paper's general guidance is that A/B tests work with "at least, thousands of active users, preferably tens of thousands." A partner proposing a fixed quota of tests every month before seeing your traffic is promising results the data may not support. A good one proposes fewer, bigger changes, or research and fixes that need no test.

What should a partner say about peeking and stopping tests early?

A good CRO partner fixes the sample size and duration before a test starts, and does not stop a fixed-horizon test the moment it looks significant.

Peeking is the most common way honest teams produce false winners. Evan Miller showed in How Not To Run an A/B Test that "if you peek at an ongoing experiment ten times, then what you think is 1% significance is actually just 5% significance." In his simulation of a test with no real effect, stopping at the first significant result produced false winners 26.1% of the time.

The rule depends on the statistics the tool uses. Optimizely's statistical methods overview says of fixed-horizon tests, "You cannot peek mid-test," while its sequential Stats Engine is built for continuous monitoring. Ask which method each test uses, since the KDD paper notes that stopping at significance was once "a procedure recommended by the company" behind Optimizely's early tool.

Low power makes every one of these errors worse. With a 10% win rate, the same paper estimates that about 22% of declared winners are false even at 80% power, and at 20% power "more than half of your statistically significant results will be false positives." Fix the sample size before the test starts, and write the stopping rule into the plan.

How should a partner measure a winning test against revenue?

A CRO partner should measure a winning test on revenue per visitor and gross margin, and treat conversion rate as one part of that result.

Conversion rate alone misleads in both directions. A free-shipping banner can lift conversion while cutting order value, and a bundle can lower conversion while raising revenue per visitor, which is conversion rate multiplied by average order value. Optimizely lists revenue per visitor among the uses of its total revenue metric.

Revenue is noisier than conversion, because most visitors spend nothing and a few spend a lot. Optimizely's outlier management page warns that "a few extreme values pull the average off-center, inflate variance, and delay a statistically significant result." A good partner sets an outlier rule, such as a cap on order value, before launch.

We would judge the winner on revenue per visitor first, then check discount depth, returns and the split between new and returning customers in Shopify. Shoppers who arrive from ChatGPT or Google AI Mode often land deep on a product page with their question already answered, so results split by traffic source can reveal a winner that works for one audience only. For where orders leak before any test is designed, see where DTC stores lose orders.

Treat large claimed lifts with suspicion. At Bing, the KDD paper notes, monthly revenue gains from many experiments combined were usually in the low single digits. Kohavi and Stefan Thomke did describe one Bing headline change in Harvard Business Review that raised revenue by 12%. Big wins do happen, but the KDD paper's rule for surprising numbers is Twyman's law, "any figure that looks interesting or different is usually wrong."

Who builds the tests, and where do the ideas come from?

Tests should be built by developers who know your Shopify theme and checked by someone other than the builder, with ideas drawn from your own data and customers.

Good partners form hypotheses from analytics funnels, session recordings, on-site surveys, reviews and support tickets, and write each test as a change, an expected effect and the metric that decides it. A competitor's page or a best-practice list is a weak source.

A variant that flickers or slows the page can lose for reasons unrelated to the idea. Google's guidance on A/B testing and Search asks testers to show Googlebot the same URLs as people, use rel="canonical" on variants, use 302 redirects and remove test code promptly. Redirect tests built with 301s put your organic traffic at risk.

Questions to ask a CRO partner, with good answers and red flags

Ask these of the person who will run your program, and ask for documents.

Frequently asked questions

What is a good win rate for a CRO partner?

There is no fixed benchmark. Large programs report roughly 8% to 33% of experiments succeeding, by their authors' ballpark estimates, so a partner claiming far more should explain how it counts a win.

Can a small Shopify store run A/B tests?

It can test large changes on its busiest pages, but small lifts need far more traffic, about 62,000 visitors per variant for a 10% lift on a page converting at 2.5%, and about 250,000 for a 5% lift. Below that, research and fixing known problems usually pay back faster than testing.

How long should an A/B test run?

A test should run until it reaches the sample size set before launch. Round that up to full weeks so weekday and weekend shoppers are both counted, unless the tool uses a sequential method built for continuous monitoring.