Bayesian Stopping Rules

Real-time monitoring with automated early-stop recommendations

Active Monitors

Ready to Stop

Graduated

86.6%

Avg Confidence

1 experiment ready to stop

Expected loss is below threshold — review and deploy winners

How Bayesian Stopping Works

Compute posterior distributions for each variant's conversion rate using Beta-Binomial model

Calculate expected loss — the cost of choosing the wrong variant if we stop now

Compare expected loss against your configured threshold (e.g., 0.05%)

When loss drops below threshold, recommend stopping and deploying the winner

Checkout Button Color

Recommend Stop

Day 18 · 49,000 samples

low risk

Control (Blue)

7.54%

1,847 / 24,500

Variant (Green)

8.80%

2,156 / 24,500

0.012%

Expected Loss

threshold: 0.05%

99.2%

P(B > A)

100%

Sample

95% Credible Interval (B − A)

0.68%

1.94%

Expected Loss Trend

Posterior Probability

Stop and deploy Variant (Green). Expected loss is well below threshold with 99.2% probability of being the winner.

Pricing Page Layout

Monitoring

Day 12 · 30,400 samples

medium risk

Control (Grid)

5.87%

892 / 15,200

Variant (Comparison)

6.14%

934 / 15,200

0.380%

Expected Loss

threshold: 0.1%

72.4%

P(B > A)

61%

Sample

95% Credible Interval (B − A)

-0.42%

0.96%

Expected Loss Trend

Posterior Probability

Continue monitoring. Expected loss (0.38%) still above threshold. Credible interval includes zero — insufficient evidence.

Onboarding Flow Length

Continue

Day 8 · 16,200 samples

high risk

Control (5 steps)

5.22%

423 / 8,100

Variant (3 steps)

5.49%

445 / 8,100

0.850%

Expected Loss

threshold: 0.2%

61.8%

P(B > A)

36%

Sample

95% Credible Interval (B − A)

-0.91%

1.45%

Expected Loss Trend

Posterior Probability

Continue — too early to call. Only 36% of required sample collected. Expected loss is 4x above threshold.

Hero Image A/B

Stopped

Day 24 · 76,000 samples

low risk

Control (Photo)

8.45%

3,210 / 38,000

Variant (Illustration)

9.30%

3,534 / 38,000

0.003%

Expected Loss

threshold: 0.05%

99.8%

P(B > A)

100%

Sample

95% Credible Interval (B − A)

0.42%

1.28%

Expected Loss Trend

Posterior Probability

Experiment stopped. Variant (Illustration) deployed as winner with 99.8% confidence.

Search Algorithm v2

Graduated

Day 30 · 104,000 samples

low risk

Control (v1)

9.85%

5,120 / 52,000

Variant (v2)

11.20%

5,824 / 52,000

0.001%

Expected Loss

threshold: 0.05%

100.0%

P(B > A)

100%

Sample

95% Credible Interval (B − A)

0.95%

1.75%

Expected Loss Trend

Posterior Probability

Graduated to production. Variant (v2) showed a consistent 1.35pp lift with near-certainty.

Live monitoring active · Last updated 1:54:37 AM