> For the complete documentation index, see [llms.txt](https://alhena.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://alhena.gitbook.io/docs/developer-reference/website-sdk/ab-testing.md).

# A/B Testing

Measure the impact of Alhena AI by comparing key metrics against alternative support solutions like Zendesk or Intercom.

### Overview

> **Note:** A/B testing is available by request. [Contact support](/docs/troubleshooting/troubleshooting.md#contact-support) to enable this feature for your account.

The A/B testing experiments feature splits your website visitors into two groups:

* **Test Group**: Users who get the change you're testing
* **Control Group**: Users who don't

What the change is depends on the experiment type. In a **Widget visibility** experiment (the original kind), the test group sees Alhena AI and the control group doesn't. You can also test **Nudge visibility**, **Embeddables visibility**, or a new **Launcher icon** (see [Creating an Experiment](#creating-an-experiment)).

### How It Works

1. **Create an Experiment**: Define your audience split (e.g., 50% test, 50% control) in the Analytics dashboard
2. **User Assignment**: When a visitor loads your website, they are automatically assigned to a group based on their unique fingerprint
3. **Consistent Experience**: The same user will always be assigned to the same group throughout the experiment
4. **Track Metrics**: Monitor key metrics like add-to-cart value and checkout revenue for each group

### The `experiment:loaded` Event

When an active experiment is running, the Alhena widget fires the `experiment:loaded` event after determining which group the user belongs to. This event allows you to:

* Conditionally load other SDKs based on the experiment group
* Customize your page behavior for control group users
* Implement your own tracking logic

#### Event Signature

```javascript
window.gleenWidget.on('experiment:loaded', (experimentGroup) => {
    // experimentGroup is either 'test' or 'control'
});
```

**Parameters:**

* `experimentGroup` (string): Either `'test'` (user sees Alhena AI) or `'control'` (user doesn't see Alhena AI)

### Example: Loading Zendesk for Control Group

A common use case is to show an alternative support solution (like Zendesk) to users in the control group who don't see Alhena AI:

```html
<script src="https://app.alhena.ai/sdk/gleenWidget.js"></script>

<script>
    window.gleenWidget.on('experiment:loaded', (experimentGroup) => {
        if (experimentGroup === 'control') {
            // Dynamically load Zendesk for control group users
            const script = document.createElement('script');
            script.id = 'ze-snippet';
            script.src = 'https://static.zdassets.com/ekr/snippet.js?key=7460dcd8-6a51-442e-9675-9781f5954552';
            document.body.appendChild(script);
        }
    });
</script>
```

This ensures:

* **Test group** users see Alhena AI
* **Control group** users see Zendesk instead
* Fair comparison between the two solutions

### Important Notes

* The `experiment:loaded` event only fires when an **active** experiment is running
* User assignment is deterministic and based on their fingerprint UUID, ensuring consistency across sessions
* In a **Widget visibility** experiment, users in the **control group** won't see the Alhena AI widget or any [embeddables](/docs/features/embeddable-agents.md) on your pages (both are automatically disabled)
* In an **Embeddables visibility** experiment, only embeddables are hidden from the control group; the chat widget shows as usual
* Make sure to set up your **cart and checkout events** before starting an experiment to track revenue impact

### Setting Up Experiments

#### Prerequisites

Before creating an experiment, ensure you have:

1. **Cart Events**: Implemented using `gleenWidget.sendCartEvent()`
2. **Checkout Events**: Implemented using `gleenWidget.sendCheckoutEvent()`

See the [Cart and Checkout Event API documentation](/docs/developer-reference/website-sdk/cart-checkout-events.md) for setup instructions.

#### Creating an Experiment

1. Navigate to **Analytics > Experiments** in your dashboard
2. Click **Create experiment** (or **Create first experiment** if you haven't run one yet). You can only run one experiment at a time, so the button is disabled while another is running.
3. Enter an **Experiment Name**
4. Under **Schedule**, choose an **End date**:
   * **No end date** — the experiment runs until you end it yourself. This is the default.
   * **Choose an end date** — pick a **Scheduled end date**. The experiment runs through the end of that day in your timezone and then completes automatically (the end date is checked hourly).
5. Under **Experiment Type**, choose what to test:
   * **Widget visibility** — show Alhena AI to the test group and hide it from the control group. This hides both the chat widget and any embeddables.
   * **Embeddables visibility** — show [embeddables](/docs/features/embeddable-agents.md) to the test group and hide them from the control group, while the chat widget stays the same for everyone. Make sure your embeddables are configured and turned on before you start.
   * **Nudge visibility** — show nudges to the test group and hide them from the control group. Make sure your [nudges](/docs/features/chat-widget/nudges.md) are set up before you start.
   * **Launcher icon** — show a new launcher icon to the test group, while the control group keeps the icon from your website settings. Upload the **Test group launcher icon** in the form.
6. Under **Audience Split**, set the percentage of website traffic for the **Test Group** and **Control Group** (50/50 by default)
7. Check that add to cart and checkout events are being received. **Start Experiment** stays disabled until both have arrived; use the refresh button to check again.
8. Click **Start Experiment**

#### Monitoring Results

The experiment dashboard shows real-time metrics for both groups:

* **Exposure**: Number of unique users in each group
* **Add to Cart Value**: Total cart value added
* **Checkout Revenue**: Total revenue from completed checkouts
* **Delta**: Percentage difference between test and control groups

#### Ending an Experiment

When you're ready to conclude:

1. Navigate to your experiment in the dashboard
2. Click **Finish experiment** (shown when the test group is winning) or **Abandon experiment**
3. Review the final results and statistical significance
4. The system automatically determines if the experiment was Won, Lost, or Inconclusive

An experiment with a scheduled end date completes on its own at the end of that day. An experiment with no end date keeps running until you end it.

#### Changing the End Date

While an experiment is running, click **Edit end date** to add, move, or remove its end date. Choose **No end date** to let it run until you end it manually, or **Choose an end date** to set a new date in the future, then click **Save**.

### How It Works Under the Hood

When a user visits your website:

1. The Alhena widget loads and checks if an experiment is active
2. The user is assigned to a group based on their unique fingerprint using a deterministic algorithm
3. The `experiment:loaded` event fires with either `'test'` or `'control'`
4. For control group users, whatever the experiment tests (the widget, nudges, or embeddables) is automatically disabled

This ensures:

* **Consistency**: The same user always sees the same experience
* **Even distribution**: Users are evenly split between groups
* **No bias**: Assignment is random and not based on user behavior

### Best Practices

1. **Run for adequate duration**: Allow at least 1-2 weeks to gather meaningful data
2. **Don't modify mid-experiment**: Changing audience split invalidates results
3. **Track other metrics**: Use the event to send experiment group to your analytics platform
4. **Test your integration**: Verify the control group experience works as expected
5. **Statistical significance**: Wait for enough data before drawing conclusions

### FAQ

**Q: Can I run multiple experiments simultaneously?**

A: No, only one experiment can be RUNNING at a time per bot profile.

**Q: What happens to users after an experiment ends?**

A: All users will see the normal Alhena AI widget (no experiment assignment).

**Q: Can I change the audience split after starting?**

A: No, this would invalidate the experiment. You'll need to end and create a new one.

**Q: Is the event fired on every page load?**

A: Yes, the event fires each time the widget loads and detects an active experiment.