A/B Test Analysis Guide

Provides rigorous A/B test analysis with statistical significance, practical impact assessment, and clear ship/no-ship recommendations.

Data & Analytics · advanced · ChatGPT

Prompt template

Analyze the results of the following A/B test.

Test name: {{test_name}}
Hypothesis: {{hypothesis}}
Primary metric: {{primary_metric}}
Secondary metrics: {{secondary_metrics}}
Test duration: {{duration}}
Sample size: Control = {{control_n}}, Variant = {{variant_n}}

Results:
Control: {{control_metrics}}
Variant: {{variant_metrics}}

Please provide:
1. Statistical significance calculation (p-value, confidence level)
2. Effect size and practical significance
3. Power analysis - was the sample size sufficient?
4. Segment breakdown if applicable
5. Novelty effect considerations
6. Revenue impact projection
7. Clear recommendation: ship, iterate, or abandon
8. Next experiment suggestions based on learnings

Example output

Results Summary:
- Control conversion rate: 3.2% (n=15,000)
- Variant conversion rate: 3.8% (n=15,200)
- Relative lift: +18.75%
- P-value: 0.003 (statistically significant at 95% confidence)
- Power: 92% (sufficient sample size)

Recommendation: SHIP the variant. The 0.6pp increase in conversion rate projects to approximately $180K additional annual revenue. No negative impact on secondary metrics (bounce rate, time on site).

Caution: Monitor for novelty effect over the next 2 weeks post-rollout.

Use cases

  • Growth teams evaluating experiment results
  • Product managers making data-driven decisions
  • Optimization specialists running conversion tests

Tags

AB-testing, experimentation, statistics, optimization