All docs pages
Limitations#
What TryFast does not do yet, and the limits every A/B testing tool that runs in the browser shares, with what you can do about each.
Not available yet#
- Personalization, feature flags and multivariate tests. TryFast runs A/B and split URL tests. If you need these today, Kameleoon, VWO and Optimizely offer them.
- Country targeting and a country breakdown of results. Also planned: time windows, weekdays, "saw experiment X" and "converted on goal Y" as audience conditions.
- Server-side conversions, webhooks and native connections to analytics tools. Use the data layer and window events and
tryfast.track()in the browser. - Changing weights of a running experiment. Only the traffic share can change; to rebalance, stop and duplicate (Traffic).
- A one-file export of a workspace. Results export as CSV and JSON, and every definition is available through the REST API.
- Automatic import from other tools. The concepts carry over one to one; rebuild running tests by hand or ask your AI agent to rebuild them from the old code.
- Single sign-on. On the roadmap for Business and Enterprise.
- The visual editor cannot select elements inside iframes or shadow DOM.
- Split URL tests redirect only within the same site (registrable domain), so a visitor keeps their id and variation.
Limits of testing in the browser#
Ad and content blockers#
Some blockers stop scripts from third-party hosts, which can include cdn.tryfast.eu. Those visitors see your original page and are not counted, in every variation alike, so results stay unbiased but smaller. With the recommended snippet the page waits at most 800 ms before showing the original.
Safari and visitor ids#
Safari limits cookies set by JavaScript to 7 days. A Safari or iOS visitor who returns after more than 7 days gets a new visitor id and may be assigned a different variation. Results stay unbiased (each id is counted once, in its first variation), but the variation that visitor sees can change, which blurs the difference between variations a little for tests that run for weeks and depend on returning visitors. The safari_share health check warns when 5% or more of an experiment's visitors use Safari.
Flicker#
The original page can show for a moment before the variation when the snippet is not at the top of <head>, when it is loaded through a tag manager, when a script optimizer delays it, or with triggers other than "when the page loads". Install the recommended snippet as described in Install the snippet. When the script does not arrive within the timeout (800 ms by default), the visitor sees the original and is not entered on that page view.
Consent#
With the default consent mode, visitors who refuse or ignore your cookie banner see your original pages and are not counted, so experiments need more time on sites where few visitors accept. In the "before consent: in memory" mode a visitor who never consents can see different variations on different pages (Privacy and consent).
Traffic and sample size#
Reliable answers need visitors and conversions. A page with a few hundred visitors a week can only detect large effects; small improvements need tens of thousands of visitors per variation. Check the estimate before you launch (Statistics), test the pages with the most traffic, and pick a goal that converts often.
Pages that change on their own#
Changes apply to elements by CSS selector. When your site's markup changes (a redesign, a renamed class), a selector can stop matching and the variation silently looks like the control. Prefer stable selectors (id, data-testid), and check variations again after site releases; with ?tf_debug=1 selectors that match nothing are listed.
Events#
Events are accepted from the project's domains only, and are dropped when they arrive more than 7 days late (a browser that was offline for a week). Bots that announce themselves and automated browsers are not counted.
Page checks#
The TryFast inspector reads your page's HTML from our server and does not run JavaScript, so pages that render everything in the browser, or that block automated requests, show little or nothing. Your agent can use its own browser instead, or you can open the visual editor and pick elements on the real page.
Something missing from this list? Tell us at support@tryfast.eu.