All docs pages

Experiments#

An experiment tests one idea on one project: your page as it is (the control, letter A) against one or more variations (B, C and so on). TryFast splits the visitors who match the experiment's targeting, shows each one the same variation on every visit, counts conversions on your goals and tells you when the result is safe to act on.

Types#

A/B test (ab): 2 to 8 variations including the control. Variations change the page with declarative changes, CSS and JavaScript. A test whose variations change nothing is an A/A test: allowed, and the launch checklist warns you.
Split URL test (split_url): every variation except the control sends its visitors to another URL on the same site. The control stays on the original page. See Redirects.

What an experiment is made of#

Part	What it does	More
Name and key	The key (`pricing-cta`) is unique in the project and used by `tryfast.activate()` and `tryfast.variation()`.
Hypothesis	What you expect and why. Optional for a draft, required to launch.
Variations	The control and the changed versions, each with a weight.	Variations, Visual editor
Targeting	Which pages, when on the page, and which visitors.	Targeting
Traffic	The share of targeted visitors who enter, and the split between variations.	Traffic
Goals	One primary goal decides the verdict; secondary goals and guardrails are reported too.	Goals
Statistics settings	Method (sequential by default), minimum days, the smallest effect you care about.	Statistics
Schedule	Optional start and end times.	Lifecycle

Create an experiment#

In the app: Experiments, New experiment. Choose the type and a name, the page, and how to build the change (visual editor, code, or ask your agent), then refine the variations, targeting, traffic and goals. The last step is a launch checklist.
With your AI agent: ask in plain words. The agent calls create_experiment, which builds a complete draft in one call and returns preview links (MCP reference).
With the API: POST /api/v1/projects/{project_id}/experiments with the whole definition:

{
  "name": "Free shipping headline",
  "hypothesis": "Naming free shipping in the headline makes more visitors start checkout.",
  "type": "ab",
  "targeting": {
    "pages": {"include": [{"match": "exact", "value": "/pricing"}], "exclude": []},
    "trigger": {"type": "immediate"},
    "audience": null
  },
  "coverage_bp": 10000,
  "variations": [
    {"key": "control", "name": "Control", "is_control": true, "weight_bp": 5000},
    {"key": "b", "name": "Free shipping", "weight_bp": 5000, "changes": [
      {"op": "set_text", "selector": "#hero h1", "value": "Free shipping on every order"}
    ]}
  ],
  "goals": [{"goal_id": "goal_Pu9rC4s3Ez", "role": "primary"}],
  "stats": {"method": "sequential", "mde_relative": 0.1, "min_days": 7}
}

Everything except name and variations has a default: the whole site, when the page loads, every visitor, 100% traffic split equally, sequential statistics. Weights default to an equal split; if you set them they must add up to 10,000.

What you can change, and when#

Experiments are drafts until you launch them. Once an experiment runs, changes that would mix two different tests in one result are refused:

Field	Draft, scheduled	Running, paused	Stopped, shipped, archived
Name, hypothesis, description, tags	yes	yes	yes
Key	yes	no	no
Targeting (pages, trigger, audience)	yes	no	no
Variations (changes, CSS, JavaScript, redirects, weights, adding or removing)	yes	no	no
Variation names	yes	yes	yes
Experiment-wide CSS and JavaScript	yes	no	no
Primary goal	yes	no	no
Secondary goals and guardrails	yes	yes	no
Statistics settings	yes	no (fixed at launch)	no
Traffic share (`coverage_bp`)	yes	only with the traffic action	no
End date	yes	yes	no
Start date	yes	no	no

A refused change answers 409 experiment_running_immutable. To change a running test, stop it and duplicate it: the copy is a draft with the same definition and a fresh split. Adding a secondary goal to a running experiment is fine and even retroactive, because conversions are recorded by event key (Goals).

When someone without launch permission (an agent with the default permission, or the visual editor) changes a scheduled experiment, it goes back to draft, so nothing goes live that a person with launch permission did not see.

In this section#

Variations: changes, CSS, JavaScript and redirects.
Visual editor: change your live site by clicking.
Targeting: pages, triggers and audiences.
Traffic: how visitors are split and how to ramp up.
Lifecycle and approvals: launch, pause, stop, ship, schedule, and the launch checklist.
QA and preview: preview links, debugging, excluding your own browser.

Limits#

Up to 8 variations per experiment, 500 experiments per project (archived ones included) and 20 goals per experiment. Running experiments per workspace depend on the plan: unlimited on every account plan, 3 in a workspace without an account, where shipped experiments count as running (Pricing).

View as markdown