A/B testing

The A/B testing plugin runs experiments on content. It assigns visitors to variants deterministically, records impressions and conversions from block events, and forwards results to analytics. It has a server half and a client half.

Installation

Add the server plugin

import { createCMS } from '@createcms/core';
import { abTest } from '@createcms/core/plugins/ab-test';

export const cms = createCMS({
  db,
  collections,
  media,
  authMiddleware,
  plugins: [abTest()],
});

Update the database

The plugin adds tables for tests, variants, and events. Regenerate the schema file, then apply it to your database with your Drizzle migration workflow:

npx createcms generate
npx drizzle-kit generate && npx drizzle-kit migrate

The drizzle-kit commands need a drizzle.config.ts at your project root — see the Quickstart for the one to create.

Add the client plugin

import { createCMSClient } from '@createcms/core/react';
import { abTestClient } from '@createcms/core/plugins/ab-test/client';
import type { cms } from './cms';

export const cmsClient = createCMSClient<typeof cms>()({
  baseURL: '/api/cms',
  plugins: [abTestClient()],
});

Usage

The client exposes an abTest namespace. Record an impression for the variant a visitor sees, read their assignment, and manage consent:

cmsClient.abTest.recordImpression(testId, branchId);

const { variantId, branchId, assignedAt } = await cmsClient.abTest.getVariant(testId);

cmsClient.abTest.setConsent({ analytics_storage: 'granted' });

Live results

useLiveResults streams result deltas to the dashboard over the public ab:live:<testId> channel. It's imported from its own subpath (which pulls in the optional @upstash/realtime peer) and rides the shared RealtimeProvider connection — the same one useNotifications uses:

import { useLiveResults } from '@createcms/core/plugins/ab-test/live';

const { results, isLive } = useLiveResults({
  testId,
  initial, // SSR snapshot from getResults
  getResults: () => client.abTest.getResults({ query: { testId } }), // reconcile on (re)connect
});

Server endpoints

The plugin registers ten endpoints under the abTest namespace (cms.api.abTest.<name> on the server, client.abTest.<name> on the client), plus the per-collection resolveAbVariant read (cms.api.<collection>.resolveAbVariant). The client convenience actions build on these, and every endpoint also surfaces as a fully typed fetcher through typeof cms, so the input and return shapes below are inferred end to end.

Resolve a Variant for a Path

Look up the single running test that varies a request path's render — the page root or a transitively-embedded block — and get back the variants to bucket across, or { test: null } when nothing applies. It carries no visitor input, so it's safe to cache from the edge. It lives as a collection endpoint (cms.api.<collection>.resolveAbVariant).

const data = await cms.api.pages.resolveAbVariant({
  query: { path: '/pricing' }, // required
});

const { data, error } = await client.pages.resolveAbVariant({
  query: { path: '/pricing' }, // required
});

Create a Test

Create an A/B test on a root. It opens in draft, every variant's branch must already be published, and you get back the new test's id.

const data = await cms.api.abTest.createTest({
  body: {
    rootId: 'root_home', // required
    collection: 'pages', // required
    name: 'Hero copy test', // required
    variants: [
      { branchId: 'br_control', name: 'Control', weight: 50, isControl: true },
      { branchId: 'br_bold', name: 'Bold headline', weight: 50 },
    ], // required
    trafficPercentage: 50, // optional, defaults to 100
  },
});

const { data, error } = await client.abTest.createTest({
  body: {
    rootId: 'root_home', // required
    collection: 'pages', // required
    name: 'Hero copy test', // required
    variants: [
      { branchId: 'br_control', name: 'Control', weight: 50, isControl: true },
      { branchId: 'br_bold', name: 'Bold headline', weight: 50 },
    ], // required
    trafficPercentage: 50, // optional, defaults to 100
  },
});

Update a Test

Rename a test, retarget its traffic or goal, replace its variants, or advance its status through the state machine. Pass null for goalHandle/goalEvent to clear the goal; you get back the test's id.

const data = await cms.api.abTest.updateTest({
  body: {
    testId: 'test_hero', // required
    status: 'running', // optional
  },
});

const { data, error } = await client.abTest.updateTest({
  body: {
    testId: 'test_hero', // required
    status: 'running', // optional
  },
});

Get a Test

Fetch a single test together with its variants.

const data = await cms.api.abTest.getTest({
  query: { testId: 'test_hero' }, // required
});

const { data, error } = await client.abTest.getTest({
  query: { testId: 'test_hero' }, // required
});

List Tests

Page through your tests, optionally narrowed to one collection or status.

const data = await cms.api.abTest.listTests({
  query: { status: 'running', limit: 50 }, // all optional
});

const { data, error } = await client.abTest.listTests({
  query: { status: 'running', limit: 50 }, // all optional
});

List Goal Events

Enumerate the conversion goals you can pick for a root: one candidate per functional block instance and declared event, across the root's own tree and its embedded reusable blocks.

const data = await cms.api.abTest.listGoalEvents({
  query: { rootId: 'root_home' }, // required
});

const { data, error } = await client.abTest.listGoalEvents({
  query: { rootId: 'root_home' }, // required
});

Assign a Variant

Deterministically bucket a visitor into a running test server-side and get back their variant, its branch, and whether they fell inside the test. It reads (assigns without mutating test config), so its operation is read despite the POST.

const data = await cms.api.abTest.assignVariant({
  body: {
    testId: 'test_hero', // required
    context: { key: 'visitor_abc' }, // required
  },
});

const { data, error } = await client.abTest.assignVariant({
  body: {
    testId: 'test_hero', // required
    context: { key: 'visitor_abc' }, // required
  },
});

Track an Event

Record an impression, conversion, or any analytics event from a visitor. It uses a distinct abTestEvent permission resource, so an app can open it to anonymous visitors without exposing test management. A/B-attributed events must resolve to a variant of testId via variantId or branchId; it also accepts the optional analytics-stitching fields source, interactionId, and transport, and returns an empty ack.

const data = await cms.api.abTest.trackEvent({
  body: {
    eventType: 'conversion', // required
    testId: 'test_hero',
    branchId: 'br_bold',
  },
});

const { data, error } = await client.abTest.trackEvent({
  body: {
    eventType: 'conversion', // required
    testId: 'test_hero',
    branchId: 'br_bold',
  },
});

Get Results

Pull the aggregated results for a test — per-variant impressions, conversions, and rates, plus the totals. Bound the reporting window with from/to.

const data = await cms.api.abTest.getResults({
  query: { testId: 'test_hero' }, // required
});

const { data, error } = await client.abTest.getResults({
  query: { testId: 'test_hero' }, // required
});

Flush Events

Force buffered events to persist on adapters that buffer (Upstash, say); adapters that don't throw AB_TEST_FLUSH_NOT_SUPPORTED. Pass testId to flush one test, or omit it to flush all.

const data = await cms.api.abTest.flushEvents({
  body: { testId: 'test_hero' }, // optional
});

const { data, error } = await client.abTest.flushEvents({
  body: { testId: 'test_hero' }, // optional
});

Delete a Test

Delete a test. Only tests in draft or completed status can be deleted.

const data = await cms.api.abTest.deleteTest({
  body: { testId: 'test_hero' }, // required
});

const { data, error } = await client.abTest.deleteTest({
  body: { testId: 'test_hero' }, // required
});

Variant selection

Several functions and endpoints all "pick a variant", at different layers. Reading the ladder from the lowest-level primitive up to the entry points you actually call:

The three bold verbs are the intended entry points: decideEdgeVariant at the edge, assignVariant in a client/server pipeline, and pickVariant when rendering. See Edge A/B for the full edge primitive list and React for the renderer.

getPublishedContent returns one entry per published branch in variants (see Publishing). With no A/B test there is exactly one, so variants[0] is the page. With this plugin, resolve the visitor's branch and select it with pickVariant:

import { pickVariant } from '@createcms/core/react/variant';

const tree = pickVariant(variants, branchId) ?? variants[0].tree;

See the pickVariant reference for the full signature.

Schema

Options

Server, abTest(options)

Client, abTestClient(options)

Client-side vs server-side measurement

By default the client forwards events to the browser dataLayer (for GTM/GA4). If you instead configure server-side forwarding with the ga4 option, set disableDataLayerSink: true on the client so the same event is not counted twice.

Edge resolution cost

By default the edge middleware resolves the active tests with one fetch per request, and a middleware fetch bypasses the Next.js Data Cache, so it is not cached. This is intentional: edge runs as many short-lived isolates with no shared memory, and an uncached read keeps assignment correct the moment a test starts or stops. For high-traffic apps, pass the resolve option to swap in your own source (back it with Edge Config or KV) so the per-request read is served close to the edge instead of hitting your origin.

Error codes