The reactivation idea from the segment analysis needs a test before it ships. This calculator runs the same statistics as the platform's Python engine, so you can size the experiment and read a result the way the pipeline would.
Set the design parameters. The read updates live and is cross-checked against the grid.
Outcomes are the expected counts at these rates, so the reads are deterministic. The synthetic-draw path in the Python engine is not reproduced here.
12.00%
13.80%
15.0%
0.1415
Conversion rate for each arm, normal approximation
Separation between the curves is what a significant result looks like. Drop the sample size and watch them merge.
Nearest precomputed cell: At Risk, lift 15%, 95%
| Statistic | Stored in grid | This calculator |
|---|---|---|
| Relative lift | 26.77% | 26.77% |
| p-value | 0.0193 | 0.0193 |
| CI lower | 0.00466 | 0.00466 |
| CI upper | 0.05256 | 0.05256 |
| Power | 0.64501 | 0.64501 |
The calculator is fed this cell's exact visitor and conversion counts, then its Welch t-test, delta-method interval, and power are compared to what the Python engine wrote. Agreement here is the parity check.
The A/B engine run ahead of time across design lift and confidence, per segment.
| 95% CI (abs) | ||||||
|---|---|---|---|---|---|---|
| 5% | 80% | 17.0% | 0.1309 | 0.591 | [0.0028, 0.0335] | sig |
| 5% | 90% | 17.0% | 0.1309 | 0.446 | [-0.0016, 0.0379] | n.s. |
| 5% | 95% | 17.0% | 0.1309 | 0.325 | [-0.0054, 0.0416] | n.s. |
| 5% | 99% | 17.0% | 0.1309 | 0.142 | [-0.0128, 0.0490] | n.s. |
| 10% | 80% | 20.2% | 0.0734 | 0.694 | [0.0062, 0.0371] | sig |
| 10% | 90% | 20.2% | 0.0734 | 0.556 | [0.0018, 0.0415] | sig |
| 10% | 95% | 20.2% | 0.0734 | 0.430 | [-0.0020, 0.0453] | n.s. |
| 10% | 99% | 20.2% | 0.0734 | 0.214 | [-0.0095, 0.0527] | n.s. |
| 15% | 80% | 26.8% | 0.0193 | 0.853 | [0.0129, 0.0443] | sig |
| 15% | 90% | 26.8% | 0.0193 | 0.754 | [0.0085, 0.0487] | sig |
| 15% | 95% | 26.8% | 0.0193 | 0.645 | [0.0047, 0.0526] | sig |
| 15% | 99% | 26.8% | 0.0193 | 0.403 | [-0.0029, 0.0601] | n.s. |
| 20% | 80% | 35.3% | 0.0024 | 0.959 | [0.0218, 0.0536] | sig |
| 20% | 90% | 35.3% | 0.0024 | 0.916 | [0.0173, 0.0581] | sig |
| 20% | 95% | 35.3% | 0.0024 | 0.857 | [0.0134, 0.0620] | sig |
| 20% | 99% | 35.3% | 0.0024 | 0.673 | [0.0057, 0.0696] | sig |
| 30% | 80% | 42.5% | 0.0003 | 0.990 | [0.0293, 0.0615] | sig |
| 30% | 90% | 42.5% | 0.0003 | 0.975 | [0.0247, 0.0660] | sig |
| 30% | 95% | 42.5% | 0.0003 | 0.949 | [0.0208, 0.0700] | sig |
| 30% | 99% | 42.5% | 0.0003 | 0.846 | [0.0130, 0.0777] | sig |
| 50% | 80% | 67.3% | < 0.0001 | 1.000 | [0.0552, 0.0886] | sig |
| 50% | 90% | 67.3% | < 0.0001 | 1.000 | [0.0505, 0.0934] | sig |
| 50% | 95% | 67.3% | < 0.0001 | 1.000 | [0.0463, 0.0975] | sig |
| 50% | 99% | 67.3% | < 0.0001 | 0.998 | [0.0383, 0.1055] | sig |
| 100% | 80% | 118.9% | < 0.0001 | 1.000 | [0.1093, 0.1449] | sig |
| 100% | 90% | 118.9% | < 0.0001 | 1.000 | [0.1042, 0.1499] | sig |
| 100% | 95% | 118.9% | < 0.0001 | 1.000 | [0.0999, 0.1543] | sig |
| 100% | 99% | 118.9% | < 0.0001 | 1.000 | [0.0913, 0.1628] | sig |
| 200% | 80% | 226.1% | < 0.0001 | 1.000 | [0.2223, 0.2609] | sig |
| 200% | 90% | 226.1% | < 0.0001 | undefined | [0.2169, 0.2663] | sig |
| 200% | 95% | 226.1% | < 0.0001 | 1.000 | [0.2121, 0.2711] | sig |
| 200% | 99% | 226.1% | < 0.0001 | 1.000 | [0.2029, 0.2803] | sig |
| 500% | 80% | 580.3% | < 0.0001 | 1.000 | [0.6017, 0.6385] | sig |
| 500% | 90% | 580.3% | < 0.0001 | 1.000 | [0.5965, 0.6437] | sig |
| 500% | 95% | 580.3% | < 0.0001 | 1.000 | [0.5919, 0.6482] | sig |
| 500% | 99% | 580.3% | < 0.0001 | 1.000 | [0.5831, 0.6571] | sig |
Each row is one run of the Python A/B engine at a fixed design lift and confidence level. Power is undefined where the solver saturates at the top of the lift range. The calculator above reproduces any of these rows from its stored counts.
Significance. A two-sided Welch t-test on the two arms' per-user conversion outcomes. Welch does not assume equal variance, which matters when the arms convert at different rates.
Interval. A delta-method 95 percent interval on the absolute difference in conversion rate, using the normal quantile.
Power. Post-hoc power for the observed effect size, computed the way statsmodels' two-sample t-power does, with a noncentral t distribution.
Action bar. The verdict treats a significant result under a 10 percent relative lift as real but not worth a rollout, which mirrors the bar-raiser rule the platform documents.