# Statistics

TryFast's results answer one question in plain words: is it safe to act on this experiment now, and what should you do? This page explains how the answer is computed, so you, your analysts and your AI agents can trust it.

## Check as often as you like

Classic A/B test statistics (a fixed-horizon test) are only valid at the one moment you planned in advance. People look every morning and AI agents look every hour, and every look gives random noise another chance to look like a winner. Stopping at the first exciting number turns a 5% error rate into a much larger one.

TryFast uses **sequential statistics** by default. Its intervals (an always-valid confidence sequence) are built for continuous monitoring: they account for every look, so a result marked safe to act stays valid however often anyone checked before. You never need to wait for a planned end date before reading the results; you only need to wait until the result is clear.

You can choose another method per experiment before launch. It is fixed once the experiment runs.

| Method | Use it when | What changes |
|---|---|---|
| `sequential` (default) | Almost always. Safe for agents that poll. | Needs about twice the visitors of a fixed-horizon test for the same power, in exchange for valid results at every look. |
| `fixed` | Your team works with planned sample sizes and looks only once. | No verdict before the planned number of visitors and the minimum days; earlier numbers carry a warning that they can be noise. |
| `bayesian` | Your team prefers chance to win and expected loss. | Ship rule: chance to win at least 95% and risk below your threshold. Results warn that this is not protected against stopping early. |

## What you see

Every result has:

- a **verdict** in words,
- **`safe_to_act`**: true only when you can act on the verdict now,
- a **recommended action** for people and agents (`keep_running`, `stop` with `keep_control` or `ship_variation`, or `investigate`),
- per variation: visitors, conversions, the **conversion rate**, the **lift** against the control with its **95% interval**, and Bayesian **chance to win** and **risk** as secondary numbers,
- **health checks** of the data,
- the planned number of visitors, progress, and the estimated days remaining.

| Verdict | Shown as | Safe to act | What to do |
|---|---|---|---|
| `collecting` | Collecting data | no | Keep it running. The result tells you when to check again. |
| `ship_variation` | Ship Variation B | yes | Stop and ship the recommended variation. |
| `keep_control` | Keep control | yes | Stop and keep your original: the variations are worse, or cannot be meaningfully better. |
| `harmful_stop` | Stop now: a goal got worse | yes | Stop and keep the control: a variation hurts your primary goal early on, or a guardrail. |
| `inconclusive` | Not decided yet | no | The planned visitors are reached but an effect of the size you care about is still possible. Keep running up to twice the plan, or stop and keep the control. |
| `invalid_srm` | Check traffic split | no | The split of visitors does not match the weights; the data is not trustworthy until you find the cause. Shipping is blocked. |

## How the verdict is decided

The rules are applied in this order to the primary goal:

1. **Traffic split check fails**: `invalid_srm`.
2. **Harm**: once every variation has at least 100 visitors and the control at least 10 conversions, a variation significantly worse on the primary goal before the minimum days, or significantly worse on any guardrail at any time, gives `harmful_stop`. Harm is always tested with the sequential method, whatever method you chose, so this check is valid at any look.
3. **Not enough data**: until every variation has at least 100 visitors and, for conversion goals, 10 conversions, the verdict is `collecting` and no difference is called significant.
4. **Ship**: after the minimum days (default 7, so weekdays and weekends are both in the data), a variation significantly better on the primary goal, with no guardrail significantly worse, gives `ship_variation`. With several better variations, TryFast recommends the one with the highest lower interval bound.
5. **Keep the control**: after the minimum days, when every variation is significantly worse, or the planned visitors are reached and no variation can still be better by the smallest effect you care about, or twice the planned visitors are reached.
6. **Inconclusive**: the planned visitors are reached but the interval still includes an effect of that size.
7. Otherwise: `collecting`.

## Lift and intervals

The **lift** is the relative difference to the control: the variation's rate divided by the control's rate, minus one. "+12.0%" means 12% more conversions per visitor than the control.

Every lift comes with its 95% interval: the range the true lift is likely in. TryFast computes the relative interval with Fieller's method, which is accurate when the control's rate is uncertain too; that is why intervals are often not symmetric ("+1.0% to +31.3%"). When the control has too few conversions the interval is unbounded, shown as "not enough data yet". A difference counts as significant only when its whole interval is on one side of zero.

Read an interval like this: "−0.4% to +9.8%" means there is no clear difference yet; B might be slightly ahead, but it could also be slightly behind.

## Several variations and goals

- With more than one variation, the primary goal's p-values are adjusted (Holm-Šidák) so that testing three variations does not triple the chance of a false winner. Results say "adjusted for 3 variations".
- Secondary goals are adjusted across goals (Benjamini-Hochberg) and labelled "adjusted for multiple goals". They never decide the verdict.

## Chance to win and risk

Each variation also shows the Bayesian chance to win (the probability that it beats the control) and its risk (how much you lose on average if you ship it and it is in fact worse). The prior is weak (a standard deviation of 30% on the relative lift). In the default sequential method these are secondary numbers; the Bayesian method uses them for its ship rule.

## Health checks

Health checks run on every results read and every hour. The overall status is the worst one.

| Check | Warns | Fails |
|---|---|---|
| Traffic split (`srm`) | The split differs from the weights with p < 0.01. | p < 0.001. The verdict becomes "Check traffic split" and shipping is blocked. Common causes: redirects that run before the script, bots, or consent behaving differently between variations. |
| Mixed assignment (`mixed_assignment`) | More than 2% of visitors were seen in more than one variation. | More than 5%. |
| Conversion tracking (`conversion_tracking`) | A goal had no conversion for 24 hours, or a variation had none in 24 hours while the control had at least 10. | No conversion for 48 hours. |
| Visitor tracking (`visitor_tracking`) | No new visitor entered for 24 hours. | 48 hours. |
| Safari share (`safari_share`) | 5% or more of visitors use Safari or iOS, where visitor ids can reset after 7 days ([Limitations](limitations.md)). | Never. |

The tracking checks start 24 hours after launch.

## Sample size and duration

Before launch, TryFast estimates how many visitors an experiment needs and how long that takes, from your project's own traffic on the targeted pages and the primary goal's conversion rate over the last 14 days. Ask in the app (the Traffic and Launch steps), with `calculate_sample_size`, or with `POST /api/v1/projects/{project_id}/estimate`.

The inputs: the smallest relative effect you care about (`mde_relative`, default 10%), the number of variations, the significance level (default 0.05), the power (default 80%), the method and the traffic share. Two examples:

- Baseline 5%, 10% effect, 2 variations, fixed method: 31,231 visitors per variation. The sequential method needs about 1.9 times as many.
- At about 4,500 visitors a day and a baseline of 4.72%, detecting a 10% lift with the sequential method takes about 126,000 visitors in total, about four weeks. A larger effect shows sooner.

The planned number of visitors is fixed at launch and drives the progress bar, the inconclusive and keep-control rules, and the sequential method's tuning. When the baseline is unknown, the estimate says so and the launch checklist warns.

## Results through the API

`GET /api/v1/experiments/{id}/results` (and `get_results` for agents) returns the same JSON: the verdict, `safe_to_act`, `recommended_action`, `reason`, `next_check_after`, `health`, the traffic split test, and per goal and variation `visitors`, `conversions`, `rate`, `lift`, `ci_lift`, `p_value`, `p_value_adjusted`, `chance_to_win`, `risk_ship`, `significant` and `direction`. Fractions are fractions (`0.12` is 12%). Add `include=timeseries` for daily cumulative numbers.

`GET /api/v1/experiments/{id}/results.csv` exports one row per goal and variation with the columns `goal_key, goal_name, role, metric, variation_id, variation_key, is_control, visitors, conversions, value_sum, rate, lift, ci_lift_low, ci_lift_high, p_value, p_value_adjusted, chance_to_win, risk_ship, significant, direction, verdict, method, computed_at`.

## Who is counted

- Visitors are counted from their first exposure to the experiment, in the variation they were first assigned (intention to treat).
- Preview traffic, bots and browsers that opted out are never counted.
- With the default consent mode only visitors who consented are counted.
- Conversions count from a visitor's first exposure until the experiment stops, and not while it is paused ([Goals](goals.md#when-conversions-count)).