A/B Test Analysis Guide
Provides rigorous A/B test analysis with statistical significance, practical impact assessment, and clear ship/no-ship recommendations.
Data & Analytics · advanced · ChatGPT
Prompt template
Analyze the results of the following A/B test.
Test name: {{test_name}}
Hypothesis: {{hypothesis}}
Primary metric: {{primary_metric}}
Secondary metrics: {{secondary_metrics}}
Test duration: {{duration}}
Sample size: Control = {{control_n}}, Variant = {{variant_n}}
Results:
Control: {{control_metrics}}
Variant: {{variant_metrics}}
Please provide:
1. Statistical significance calculation (p-value, confidence level)
2. Effect size and practical significance
3. Power analysis - was the sample size sufficient?
4. Segment breakdown if applicable
5. Novelty effect considerations
6. Revenue impact projection
7. Clear recommendation: ship, iterate, or abandon
8. Next experiment suggestions based on learningsExample output
Results Summary: - Control conversion rate: 3.2% (n=15,000) - Variant conversion rate: 3.8% (n=15,200) - Relative lift: +18.75% - P-value: 0.003 (statistically significant at 95% confidence) - Power: 92% (sufficient sample size) Recommendation: SHIP the variant. The 0.6pp increase in conversion rate projects to approximately $180K additional annual revenue. No negative impact on secondary metrics (bounce rate, time on site). Caution: Monitor for novelty effect over the next 2 weeks post-rollout.
Use cases
- Growth teams evaluating experiment results
- Product managers making data-driven decisions
- Optimization specialists running conversion tests
Tags
AB-testing, experimentation, statistics, optimization