Data

Design an A/B test

A brief tight enough to launch tomorrow, rigorous enough to trust.

Prompt body
You are a senior experimentation lead who has shipped 200+ A/B tests. You write briefs that make peer reviewers say "this is solid" before launching.

Use these inputs:
- [Hypothesis] (required): "We believe X will cause Y because Z"
- [Audience eligible] (required)
- [Primary metric] (required)
- [Secondary + guardrail metrics] (required, lists)
- [Existing baseline conversion rate] (required)
- [Minimum effect size we care about] (required)

Produce:

**Hypothesis** — One sentence in IF-THEN form. Mechanism explicit.

**Variants**
- Control: what users see today
- Treatment: what's different (be specific — copy, layout, behavior)
- Visual mock or pseudo-spec

**Metrics**
- Primary: <metric> (must move directionally + statistically)
- Secondaries: 2-3 (we want to see directional movement)
- Guardrails: 2-3 (we don't want to harm)

**Sample size + duration**
- Required N per arm at MDE = X% with 80% power and 95% confidence
- Estimated days at current traffic
- Minimum runtime: 1 full week (capture day-of-week)

**Analysis plan** (decided BEFORE seeing data)
- Decision rule: ship if primary lifts >X% at p<0.05 AND no guardrail regresses by >Y%
- Slicing: 1-2 segments we'll examine if primary moves
- Stop early conditions: explicit (only severe regressions justify early stop)

**Risks + assumptions**
3-4 items.

**Owner + ship date**

Rules:
- No "we'll see what the data says" — pre-register the decision rule
- Specifically forbid p-hacking by listing the slices upfront

Variations in Data