Minimum Detectable Effect (MDE)

Minimum detectable effect, usually abbreviated MDE, is the smallest change in a metric that an experiment is designed to reliably detect. It is set before a test begins and is a direct input into the sample size calculation. An experiment with an MDE of 10 percent is built to catch improvements of 10 percent or larger with the specified level of confidence and power; smaller genuine improvements will frequently go undetected by that test, appearing as inconclusive results rather than wins.

MDE is often misunderstood as a prediction of how much a variant will improve conversion. It is not a forecast. It is a sensitivity setting, in the same sense that a scale has a smallest measurable increment. Setting it does not make an effect more or less likely to occur; it determines whether the experiment has enough resolution to see the effect if it is there. Confusing the two leads to a familiar bad conversation after a test concludes, where a team argues that the variant "should have won" because the MDE was set at 15 percent, when in fact the MDE only ever described what the test could see.

The correct way to set MDE is commercially, not statistically. The question is what size of improvement would justify the cost of building, shipping, and maintaining the change, plus the opportunity cost of the traffic spent measuring it. On a high-volume checkout, a 2 percent relative lift may be worth a substantial annual figure and therefore worth a long test. On a low-traffic contact form, an improvement below 20 percent may not be worth detecting at all, because the absolute number of additional leads is negligible. Working backwards from the business value keeps the testing roadmap anchored to outcomes rather than to statistical convenience.

There is an unavoidable tension between an ambitious MDE and the traffic a site actually has. Teams under pressure to show results often resolve it in the wrong direction, quietly raising the MDE until the sample size calculation returns a comfortable number of days. This produces experiments that are technically valid but practically useless: they are only capable of detecting effects so large that they would have been obvious without testing. The honest alternative is to accept that some questions cannot be answered by experimentation at the current traffic level, and to route them to other methods, such as usability testing, analytics investigation, or expert review.

MDE also shapes the design of the variant itself. If a test needs to detect a 15 percent improvement to be worth running, a subtle change to button copy is unlikely to clear that bar, and the team is better served by testing a substantially different approach to the page: a restructured value proposition, a different form length, a changed pricing presentation. This is why programs with limited traffic tend to produce more interesting work than programs with unlimited traffic, which can afford to test trivia.

A recurring source of confusion is whether the effect is expressed in relative or absolute terms, and the difference is not trivial. On a baseline conversion rate of two percent, a ten percent relative improvement means moving to 2.2 percent, an absolute change of 0.2 percentage points. A ten percent absolute improvement would mean moving to twelve percent, which no realistic change produces. Sample size calculators, testing platforms, and stakeholders frequently switch between the two conventions without saying so, and a plan built on the wrong reading will either specify an experiment that cannot conclude or one that is wildly overpowered. Stating the convention explicitly in every test plan, along with the baseline and the resulting absolute figure, removes an ambiguity that otherwise resurfaces every time results are discussed with people outside the analytics function.

In practice, MDE decisions are made collectively during roadmap planning rather than by an analyst working alone. A CRO service engagement typically documents, for each planned experiment, the baseline rate, the MDE, the resulting sample size, and the estimated runtime, so that stakeholders can see the cost of each question before committing to it. For organizations running experiments across several properties or markets, this documentation also prevents the common error of applying a single MDE convention to sites with radically different traffic profiles, a pattern often seen in enterprise programs where a central standard is pushed down to teams whose traffic cannot support it.