Experiments

Every test you run, in one place: site tests on your website, and the splits and holdouts inside your journeys. Open Experiments in the sidebar. Site tests come first; journey tests follow.

Site tests

A site test shows your visitors different versions of a page and measures which one earns more. You build the versions on your own site, in the Chrome extension's site editor – no code, no change to your HTML – and the Website SDK shows them to visitors. Nothing reaches a visitor until you press Publish.

You need three things:

the Website SDK on the site,
the Chrome extension in Chrome, in its current version,
the browser connected: open Connect this browser from the extension's sidebar (or go to /extension/connect in the app) and press the button once.

Three types

A/B – your page as it is (the Original) against up to 7 versions. A version can change any number of things on the page at once: the headline, the button, an image, a section hidden.
Multivariate – up to 3 parts of a page tested together, each with the Original and up to 3 alternatives. Every combination is shown (at most 24), and the results say which part moved the number.
Site change – change the site for everyone, or for one audience, without a test. Nothing is measured. Use it for a seasonal banner or a fix you want live today.

Starting one

On Experiments, press New site test. Pick the site, the type and a name; add a hypothesis if you have one ("a shorter headline gets more trials, because the offer is clearer") and the page to start on. Create draft makes the test and opens your site in the editor.

You can also start from your site: open the extension's sidebar, find Site editor, and press New test.

The editor

The editor docks on the right of your page. At the top: the test's name, type and status, who you are signed in as, and whether the Website SDK was found on the page.

Versions. Tabs across the top – Original, B, C, + – hold the versions. The Original is your page as it is and is never changed. Pick a tab to see that version on the page and its list of changes; each change says whether its element was found on this page ("1 match", or "not found on this page").

Making a change. In Edit mode, move the pointer over the page: the element under it is outlined and named. Click it (or press Enter) for the menu:

Edit text – type over the words. Links, icons and anything else inside the element stay exactly where they were.
Replace image – paste an address or upload a picture (PNG, JPEG, WebP, GIF or AVIF, up to 5 MB).
Hide – the element is not shown in this version.
Style – colour, size, weight, spacing, borders and similar.
Link – where a link goes, and its title.
Use as click goal – count clicks on this element as a goal.
Select parent – the element around it, when you picked something too small.

The arrow keys move to the element around or inside the one outlined; Esc closes the menu. Switch to Browse mode to use your site normally – open a menu, a tab or a pop-up, or sign in – then back to Edit to change what you opened.

When an element appears more than once on the page, the editor says how many it matched and asks whether the change applies to all of them.

Saving. Changes save to the draft as you work, and Undo and Redo step through them. If someone changes the same test in the app while you edit, the editor says so and loads their copy.

Where and to whom. In the editor you also set the pages the test runs on (an exact path, a path that starts with something, an address that contains or matches something, or a query parameter – and pages to leave out), the devices, an audience if you want one (audiences reach signed-in or identified visitors only), the share of visitors in the test, the split between versions, and an optional start and end date.

The plan. Say how small a change you want to be able to detect (10 % by default) and the fewest days to run (7 by default). Once the Original has 100 visitors, the test works out how many visitors each version needs.

Goals

A goal is what "better" means for this test:

Click – a click on an element you pick on the page.
Reaches a page – the visitor gets to a page, such as your thank-you page.
Custom event – an event your site sends with AICMO.track.
Revenue – conversions your site sends with AICMO.trackConversion, summed by their value (optionally only one conversion type).

Every A/B and multivariate test needs one primary goal before it can be published; the others are secondary and are watched for harm. A goal counts only for visitors who were shown the test, and only from that moment.

Preview links

Copy preview link – in the app or in the editor – gives a link that opens your site with the test applied, for any version you pick. A small badge in the corner names the test, switches between versions and has Exit. Preview links work on drafts too, nobody is counted while previewing, and a link works for 24 hours.

Publishing

Publish sends the test live: visitors who match it start seeing a version from their next page view. Before publishing, the editor checks that every version has a change, that there is a primary goal, that the SDK is on the page, and whether another live test on the site changes the same element on the same pages – showing both to the same visitors would muddy both results, so you are asked first.

Edits to a running test collect in the draft and go live only with Publish changes, as a new version. The test's timeline records every version and who published it. While a test runs, its versions, their split, its sections and its goals stay as published, and the share of visitors in it can only go up – a visitor is never moved to another version mid-test.

Pause stops showing the test (everyone gets your page as it is) without ending it; Resume carries on. Stop ends it: choose Keep the Original or No decision, add a note, and the results of that moment are kept with the decision. For a site change, Turn off does the same.

Reading the results

Open a test on Experiments. For each goal, the table shows every version with:

Visitors and Converted – how many were shown it, and how many of them reached the goal.
Rate, with its 95 % interval underneath.
Value / visitor for revenue goals, in your brand's currency, with its interval.
Lift over the Original, with its interval.
P(beats Original) – the chance this version really does better than the Original.
p (Holm) – the p-value, corrected for comparing several versions at once.

Above the table, three lines say whether the numbers can be trusted yet:

Split check – whether visitors are splitting between the versions the way the test is set up. A mismatch means something is losing one version's visitors (a redirect, a page that breaks), and every number below it is suspect until it is fixed.
Sample – the smallest version's visitors against the number planned.
Days – how long the test has run against its minimum.

A multivariate test adds a table by section: each alternative against its own section's Original. Under the results, a health line says how often the changes were applied, how often an element was not found on the page, and how often a page redrew an element and the change was put back.

Results are recalculated every few minutes; Refresh asks for fresh ones.

What is counted, honestly:

Every visitor who was shown the test counts – signed in or not – once per test, from the moment it applied on their page. A visitor keeps their version on every visit, also after signing in or out.
A visitor is a browser. The same person on their phone and their laptop is two visitors, and clearing the site's data in a browser starts over.
Visitors outside the test's share, on pages or devices it does not target, or outside its audience, see your page as it is and are not counted.
When your site asks for consent in categories, tests run for visitors who allowed marketing. Withdrawing it puts the page back as it is.
Identified visitors who were in a journey with a holdout in the last 90 days see your page as it is and are not counted, so the two measurements never share a person.
Revenue is what your site reports through the Website SDK. Orders that reach us only from a connected store or other integration are not part of site test results yet.

Shipping a winner

Ship winner serves one version to everyone the test targets, from their next page view, and stops measuring. The test is ready to ship when:

every version has reached its planned number of visitors (at least 100),
it has run its minimum days,
the split check is clean,
the leader on the primary goal has at least a 95 % chance of beating the Original, or a corrected p-value under 0.05,
and no secondary goal got significantly worse.

The dialog says which of these are not met yet. You can still ship – Ship anyway – with a reason of at least 10 characters; the reason, the results of that moment and who decided are kept with the test.

A shipped version keeps being served by the Website SDK. Once the change is in your site's own code, Retire the test so the SDK stops applying it.

Archive, restore, duplicate

Archive moves a draft or a stopped test out of the list (Show archived brings the list back) and Restore returns it. Duplicate copies a test's draft into a new draft, with a fresh split, so the copy's visitors are assigned anew.

Who can do what

A brand's owners, admins, managers and editors make, edit and publish its site tests; a view-only role reads them. A site shared into a brand from another brand is tested on the brand it belongs to.

For your developers

The Website SDK exposes the test a visitor is in:

AICMO.getVariant('<test id>') returns the version the visitor sees, or null.
The page receives an aicmo:experiment event (detail: { id, arm }) when a test applies.
When Google Tag Manager's dataLayer is on the page, an aicmo_experiment event is pushed with the test and the version.
AICMO.debug(true) logs every decision in the browser console.

While the site editor is open on a page, the SDK applies no tests there, so what you edit is your page as it is.

Journey tests

Journeys can split people between two messages (an A/B split step) or hold a share of people out to measure what the journey earns (a holdout), on every channel a journey sends: email, SMS, WhatsApp, push, on-site messages and promos. Each one shows here on its own, with the people reached, what was sent, opened and clicked, revenue per person and the probability the difference is real.

Journey tests are measured on identified people, over an equal window for everyone. They are made and changed on the journey canvas – see Journeys – so on Experiments they are read-only.