Sequential A/B Test Monitor

Nobody runs an A/B test and waits politely for the last visitor. The dashboard is right there, the numbers move every hour, and somebody asks on Tuesday whether the new page is winning. So people look. The trouble is not the looking. The trouble is reading every look as if it were the only one.

A significance level is a promise about how often a test raises a false alarm. A level of 0.05 says that if the two versions are truly the same, only one test in twenty will claim a difference. That promise is made for one test, read once, at a sample size fixed in advance. Each extra reading is another chance for random noise to cross the line, and the chances add up, which is the multiple comparisons problem in its plainest form.

How fast do they add up? Take a test checked five times, at evenly spaced points, each time against the usual two-sided threshold of z = 1.96. If the versions are identical, the chance that at least one of those five checks crosses 1.96 is about 0.142, close to three times the rate the analyst thinks they are running. The figure comes from integrating the joint distribution of the five statistics, and a simulation of the same plan lands on it within its error.

The fix is not to stop looking. It is to decide, before the first visitor arrives, how many looks there will be and how much of the error budget each one may spend.

A budget for being wrong

Clinical trials met this problem decades ago, because a trial that keeps patients on an inferior treatment has a cost that grows with every week. Sequential analysis, first attributed to Abraham Wald, lets the sample size stay open and stops the study when a pre-defined rule is met. Group sequential designs, the form most trials use, plan a fixed number of interim analyses and set a stricter threshold at each one so that the overall error stays where it was promised.

The thresholds work because the looks are not independent. The statistic at the third look shares all the visitors of the second look plus some new ones. If the information reached at two looks is I1 and I2, the correlation between their z statistics is the square root of I1 over I2. Knowing that structure, one can compute the chance of crossing at any look, given that no earlier look crossed, and choose thresholds that keep the total at the planned level.

The oldest recipe, from Stuart Pocock in 1977, uses one threshold for every look. With five planned analyses and an overall two-sided level of 0.05, each look is tested at a nominal p of 0.0158, a z of about 2.41. That is easier to explain than it is to live with. A trial that would have passed at the end with z = 2.2 now fails, because the design paid for early chances it did not use.

O’Brien and Fleming chose the opposite shape in 1979: demand overwhelming evidence early and relax toward the end. For five looks the classic version asks for z of about 4.56 at the first look and about 2.04 at the last, so the final test sits close to the ordinary 1.96. Most of the error budget is saved for the end, which is what most teams want when the early data are thin and noisy.

The Haybittle-Peto rule is the simplest of all. Every interim look must reach a p of 0.001, and the final look is read at the usual level. It is easy to state and very hard to trigger early. It does not hold the error exactly at the planned level, and that small excess can be computed: for five looks at a final level of 0.05, the overall false alarm rate is about 0.0511.

Spending alpha when the calendar slips

The classic shapes assume the looks happen when planned. Real tests drift: traffic dips over a holiday, a report is due on Friday, the third look arrives at 55 percent of the visitors instead of 60. In 1983 Lan and DeMets replaced fixed thresholds with a spending function, a curve that says how much of the error budget may be used by the time a given fraction of the information has arrived. At each actual look the threshold is solved for whatever the curve allows at that fraction.

Their function that imitates O’Brien and Fleming spends almost nothing early. With a one-sided budget of 0.025 per side and five equal looks it gives thresholds of about 4.88, 3.36, 2.68, 2.29 and 2.03. Their Pocock-like function spends more evenly and gives about 2.44 at the first look, falling slowly to 2.39. The Hwang-Shih-DeCani family covers the range between with one parameter, gamma: a gamma of minus four behaves like O’Brien-Fleming, a gamma of one like Pocock.

The flexibility has one firm condition, and the FDA guidance on adaptive designs states it plainly. The timing of a look must not depend on the results. Moving the next analysis earlier because the current one came close to the boundary is exactly the kind of choice that inflates the error again. Looks may follow the calendar, the budget or the traffic; they may not follow the scoreboard.

The plan, worked through

Suppose control converts at 10 percent and the smallest lift worth detecting is 2 points, to 12 percent, at two-sided 0.05 with 80 percent power. A single fixed-horizon test needs 3,841 visitors per arm. Five looks with the Lan-DeMets O’Brien-Fleming function raise the maximum to 3,936 per arm, an inflation of about 2.5 percent. That is the price of permission to stop early, and with this shape the price is small.

Now the test runs. At the third look each arm has 2,362 visitors: 236 conversions in A and 290 in B, rates of 9.99 and 12.28 percent. The pooled z is 2.50 and the ordinary two-sided p-value is about 0.0125. Read once, that would be a win. Read as the third of five planned looks, at 60 percent of the planned information, the boundary is 2.68. The statistic has not crossed it, and the right decision is to keep going.

The repeated confidence interval says the same thing in units a product team can use. Instead of the difference plus or minus 1.96 standard errors, it uses the look’s boundary, 2.68, so that all the intervals of the plan together cover the true difference with the stated confidence. Here it runs from about minus 0.17 to plus 4.74 percentage points. A loss is still possible, and so is a gain twice the size planned for.

Futility rules work in the other direction. A non-binding futility boundary marks the point where the remaining chance of success is small enough that stopping is reasonable. Because it is non-binding, the efficacy boundaries are computed as if it did not exist, so ignoring it never raises the false alarm rate. The FDA guidance notes that adding such guidelines does not increase the type I error. Honouring every futility stop does cost some power, which a careful plan buys back with a slightly larger maximum.

Check the split before reading anything

There is a test that comes before all of this, and it is about plumbing, not statistics. If a test was meant to split visitors evenly and the counts arrive as 600 and 400, something in the assignment or the logging is broken. Pearson’s chi-squared test of the counts against the planned split gives a statistic of 40 and a p-value of about 2.54 times ten to the minus tenth. No conversion rate from such a test should be trusted until the cause is found.

This sample ratio mismatch check belongs at every look, and it goes first. A broken split does not just add noise. It can bias who ends up in which arm, and then no boundary, however carefully integrated, protects the reading. A strict threshold such as 0.001 keeps the check from crying wolf on ordinary randomness while still catching real breakage.

What an early stop does to the number you report

Stopping when the statistic is large selects the runs where chance happened to help. The estimate at that moment is therefore too big on average; it is a biased estimator. Both the FDA guidance and the sequential analysis literature warn that trials stopped early for benefit tend to overestimate the effect, and that ordinary confidence intervals do not keep their coverage.

The size of the overstatement can be seen directly. In 20,000 simulated runs of the worked plan with a true lift of 2 points, the tests that stopped at the second look showed an average lift of about 4.2 points, roughly twice the truth; those that stopped at the third look showed about 2.9. The simulation draws visitors from a seeded generator of the xorshift family, so anyone can reproduce the same runs. An early win means the effect is probably real. It does not mean the effect is as large as it looked.

None of this rescues a test whose plan was not followed, and it has limits of its own. The boundaries rest on the normal approximation to the two-proportion statistic, which is poor when conversions are very rare or samples tiny. They cover one primary metric between two arms; more metrics or more arms need their own correction. And the whole construction assumes the looks were planned before the data could argue for them. Within those limits, a sequential plan turns peeking from a source of false wins into what the dashboard always promised: an honest chance to stop early.