Welcome to Beta!AetherSoul is in closed beta. Explore our diverse cast of AI characters with voice, video calls, and more. Join the Waitlist

Bayesian Analysis

Probability-based experiment evaluation with credible intervals

Ship It

Weakly Informative Prior

Variant A is the clear winner with 97.7% probability to beat control and expected loss of 0.003%.

97.7%

Prob. to Beat Control

0.003%

Expected Loss

48,200

Total Samples

36d

Days Running

Posterior Distribution
PROBABILITY DENSITY OF CONVERSION RATE PER VARIANT
3.27%3.63%4.00%4.36%4.72%ControlVariant A
Variant Comparison
Bayesian metrics for each variant

Control

Control

Visitors

24,100

Conversions

904

Conv. Rate

3.75%

P(Beat Ctrl)

—

Expected Loss

0.420%

95% HDI: [3.51%, 4.00%]Mean: 3.75%

Variant A

Winner
+12.0%

Visitors

24,100

Conversions

1,012

Conv. Rate

4.20%

P(Beat Ctrl)

97.7%

Expected Loss

0.003%

95% HDI: [3.94%, 4.46%]Mean: 4.20%
Decision Threshold
Configure the maximum acceptable expected loss for shipping
Max Expected Loss0.10%

A variant must have P(Beat Control) ≥ 95% AND Expected Loss ≤ 0.10% to be recommended for shipping.

Bayesian vs Frequentist

Bayesian (This View)

  • Probability to beat control (intuitive)
  • Expected loss quantifies risk
  • Credible intervals = true probability range
  • Can peek at results without penalty

Frequentist (Traditional)

  • P-value (often misinterpreted)
  • No direct risk quantification
  • Confidence intervals ≠ probability
  • Peeking inflates false positive rate