All docs pages

Statistics#

TryFast's results answer one question in plain words: is it safe to act on this experiment now, and what should you do? This page explains how the answer is computed, so you, your analysts and your AI agents can trust it.

Check as often as you like#

Classic A/B test statistics (a fixed-horizon test) are only valid at the one moment you planned in advance. People look every morning and AI agents look every hour, and every look gives random noise another chance to look like a winner. Stopping at the first exciting number turns a 5% error rate into a much larger one.

TryFast uses sequential statistics by default. Its intervals (an always-valid confidence sequence) are built for continuous monitoring: they account for every look, so a result marked safe to act stays valid however often anyone checked before. You never need to wait for a planned end date before reading the results; you only need to wait until the result is clear.

You can choose another method per experiment before launch. It is fixed once the experiment runs.

Method Use it when What changes
sequential (default) Almost always. Safe for agents that poll. Needs about twice the visitors of a fixed-horizon test for the same power, in exchange for valid results at every look.
fixed Your team works with planned sample sizes and looks only once. No verdict before the planned number of visitors and the minimum days; earlier numbers carry a warning that they can be noise.
bayesian Your team prefers chance to win and expected loss. Ship rule: chance to win at least 95% and risk below your threshold. Results warn that this is not protected against stopping early.

What you see#

Every result has:

Verdict Shown as Safe to act What to do
collecting Collecting data no Keep it running. The result tells you when to check again.
ship_variation Ship Variation B yes Stop and ship the recommended variation.
keep_control Keep control yes Stop and keep your original: the variations are worse, or cannot be meaningfully better.
harmful_stop Stop now: a goal got worse yes Stop and keep the control: a variation hurts your primary goal early on, or a guardrail.
inconclusive Not decided yet no The planned visitors are reached but an effect of the size you care about is still possible. Keep running up to twice the plan, or stop and keep the control.
invalid_srm Check traffic split no The split of visitors does not match the weights; the data is not trustworthy until you find the cause. Shipping is blocked.

How the verdict is decided#

The rules are applied in this order to the primary goal:

  1. Traffic split check fails: invalid_srm.
  2. Harm: once every variation has at least 100 visitors and the control at least 10 conversions, a variation significantly worse on the primary goal before the minimum days, or significantly worse on any guardrail at any time, gives harmful_stop. Harm is always tested with the sequential method, whatever method you chose, so this check is valid at any look.
  3. Not enough data: until every variation has at least 100 visitors and, for conversion goals, 10 conversions, the verdict is collecting and no difference is called significant.
  4. Ship: after the minimum days (default 7, so weekdays and weekends are both in the data), a variation significantly better on the primary goal, with no guardrail significantly worse, gives ship_variation. With several better variations, TryFast recommends the one with the highest lower interval bound.
  5. Keep the control: after the minimum days, when every variation is significantly worse, or the planned visitors are reached and no variation can still be better by the smallest effect you care about, or twice the planned visitors are reached.
  6. Inconclusive: the planned visitors are reached but the interval still includes an effect of that size.
  7. Otherwise: collecting.

Lift and intervals#

The lift is the relative difference to the control: the variation's rate divided by the control's rate, minus one. "+12.0%" means 12% more conversions per visitor than the control.

Every lift comes with its 95% interval: the range the true lift is likely in. TryFast computes the relative interval with Fieller's method, which is accurate when the control's rate is uncertain too; that is why intervals are often not symmetric ("+1.0% to +31.3%"). When the control has too few conversions the interval is unbounded, shown as "not enough data yet". A difference counts as significant only when its whole interval is on one side of zero.

Read an interval like this: "−0.4% to +9.8%" means there is no clear difference yet; B might be slightly ahead, but it could also be slightly behind.

Several variations and goals#

Chance to win and risk#

Each variation also shows the Bayesian chance to win (the probability that it beats the control) and its risk (how much you lose on average if you ship it and it is in fact worse). The prior is weak (a standard deviation of 30% on the relative lift). In the default sequential method these are secondary numbers; the Bayesian method uses them for its ship rule.

Health checks#

Health checks run on every results read and every hour. The overall status is the worst one.

Check Warns Fails
Traffic split (srm) The split differs from the weights with p < 0.01. p < 0.001. The verdict becomes "Check traffic split" and shipping is blocked. Common causes: redirects that run before the script, bots, or consent behaving differently between variations.
Mixed assignment (mixed_assignment) More than 2% of visitors were seen in more than one variation. More than 5%.
Conversion tracking (conversion_tracking) A goal had no conversion for 24 hours, or a variation had none in 24 hours while the control had at least 10. No conversion for 48 hours.
Visitor tracking (visitor_tracking) No new visitor entered for 24 hours. 48 hours.
Safari share (safari_share) 5% or more of visitors use Safari or iOS, where visitor ids can reset after 7 days (Limitations). Never.

The tracking checks start 24 hours after launch.

Sample size and duration#

Before launch, TryFast estimates how many visitors an experiment needs and how long that takes, from your project's own traffic on the targeted pages and the primary goal's conversion rate over the last 14 days. Ask in the app (the Traffic and Launch steps), with calculate_sample_size, or with POST /api/v1/projects/{project_id}/estimate.

The inputs: the smallest relative effect you care about (mde_relative, default 10%), the number of variations, the significance level (default 0.05), the power (default 80%), the method and the traffic share. Two examples:

The planned number of visitors is fixed at launch and drives the progress bar, the inconclusive and keep-control rules, and the sequential method's tuning. When the baseline is unknown, the estimate says so and the launch checklist warns.

Results through the API#

GET /api/v1/experiments/{id}/results (and get_results for agents) returns the same JSON: the verdict, safe_to_act, recommended_action, reason, next_check_after, health, the traffic split test, and per goal and variation visitors, conversions, rate, lift, ci_lift, p_value, p_value_adjusted, chance_to_win, risk_ship, significant and direction. Fractions are fractions (0.12 is 12%). Add include=timeseries for daily cumulative numbers.

GET /api/v1/experiments/{id}/results.csv exports one row per goal and variation with the columns goal_key, goal_name, role, metric, variation_id, variation_key, is_control, visitors, conversions, value_sum, rate, lift, ci_lift_low, ci_lift_high, p_value, p_value_adjusted, chance_to_win, risk_ship, significant, direction, verdict, method, computed_at.

Who is counted#

View as markdown