A/B Test Analysis Template (AI Prompt)

## Statistical Analysis

**Test: Homepage CTA Button Color**
- Control (Blue): 15,000 users, 630 clicks (4.2% CTR)
- Variant (Green): 15,000 users, 720 clicks (4.8% CTR)

**Calculations:**

```
Observed Lift = (4.8% - 4.2%) / 4.2% = 14.3%

Pooled Conversion Rate = (630 + 720) / 30,000 = 4.5%

Standard Error = sqrt(0.045 * 0.955 * (1/15000 + 1/15000)) = 0.00239

Z-Score = (0.048 - 0.042) / 0.00239 = 2.51

P-Value = 0.012 (one-tailed)

95% Confidence Interval: [0.13%, 1.07%] absolute lift
                          [3.1%, 25.5%] relative lift
```

**Results Summary:**

| Metric | Value | Interpretation |
|--------|-------|----------------|
| Observed Lift | +14.3% | Variant beats control |
| P-Value | 0.012 | Statistically significant (< 0.05) |
| Confidence Interval | 3.1% to 25.5% | True lift likely in this range |
| Statistical Power | 87% | Adequate for detection |
| Minimum Detectable Effect | 8% | Test was powered to detect this |

**Verdict: Statistically Significant Win** ✓

## Interpretation

**Is it statistically significant?**
Yes. With a p-value of 0.012, we can reject the null hypothesis. There is only a 1.2% chance this result occurred by random chance.

**Is it practically significant?**
Yes. A 14.3% improvement in CTR translates to approximately 90 additional clicks per 15,000 visitors. At our current traffic:
- Monthly: ~1,800 additional CTA clicks
- At 20% signup completion: ~360 additional signups
- At $50 avg customer value: ~$18,000 additional monthly revenue

**Confidence Assessment: HIGH**
- Adequate sample size (15K per variant)
- Test ran for full 14 days (includes weekday/weekend)
- No sample ratio mismatch detected
- Effect size within expected range

**Limitations:**
- Test only ran on homepage visitors
- Did not control for traffic source
- Short-term measurement only (no retention data yet)

## Segmentation Analysis

**By Device:**

| Segment | Control CTR | Variant CTR | Lift | Significant? |
|---------|-------------|-------------|------|--------------|
| Desktop | 4.5% | 5.1% | +13.3% | Yes (p=0.04) |
| Mobile | 3.8% | 4.4% | +15.8% | Yes (p=0.03) |
| Tablet | 4.0% | 4.2% | +5.0% | No (p=0.42) |

**Insight:** Effect is stronger on mobile. Consider mobile-specific optimization.

**By User Type:**

| Segment | Control CTR | Variant CTR | Lift | Significant? |
|---------|-------------|-------------|------|--------------|
| New Users | 3.5% | 4.3% | +22.9% | Yes (p=0.01) |
| Returning | 5.1% | 5.5% | +7.8% | No (p=0.18) |

**Insight:** New users respond more strongly to green CTA. Returning users may have button blindness.

## Recommendation

**Decision: SHIP** ✓

**Confidence Level: High (85%)**

**Rationale:**
1. Statistically significant result with adequate power
2. Practically significant impact (~$18K/mo potential)
3. Positive or neutral effect across all segments
4. No negative impact on secondary metrics
5. Effect is consistent with our hypothesis

**Rollout Strategy:**
1. Week 1: Roll out to 25% of traffic, monitor for anomalies
2. Week 2: Increase to 50% if metrics stable
3. Week 3: Full rollout to 100%
4. Week 4: Final measurement and documentation

**Follow-Up Experiments:**
- Test green CTA across other pages (pricing, blog CTAs)
- Test green button with different copy variations
- A/B test green shades (lime vs forest)

## Documentation Template

**Test Summary (for stakeholders):**

> We tested changing our homepage CTA button from blue to green. The green button increased clicks by 14.3% (statistically significant, p=0.012). This translates to approximately 360 additional monthly signups and $18K in potential revenue. We recommend shipping the green button to 100% of users with a gradual rollout over 3 weeks.

**Technical Documentation:**

```
Test ID: EXP-2024-037
Test Name: Homepage CTA Button Color
Hypothesis: Green CTA will outperform blue CTA
Primary Metric: CTA Click-Through Rate
Start Date: 2024-02-01
End Date: 2024-02-14
Sample: 30,000 (50/50 split)
Result: Variant wins (+14.3%, p=0.012)
Decision: Ship
Shipped Date: 2024-02-21
```