A/B Testing

Measure the impact of Alhena AI by comparing key metrics against alternative support solutions like Zendesk or Intercom.

Overview

Note: A/B testing is available by request. Contact support to enable this feature for your account.

The A/B testing experiments feature splits your website visitors into two groups:

• Test Group: Users who get the change you're testing

• Control Group: Users who don't

What the change is depends on the experiment type. In a Widget visibility experiment (the original kind), the test group sees Alhena AI and the control group doesn't. You can also test Nudge visibility, Embeddables visibility, or a new Launcher icon (see Creating an Experiment).

How It Works

1. Create an Experiment: Define your audience split (e.g., 50% test, 50% control) in the Analytics dashboard

2. User Assignment: When a visitor loads your website, they are automatically assigned to a group based on their unique fingerprint

3. Consistent Experience: The same user will always be assigned to the same group throughout the experiment

4. Track Metrics: Monitor key metrics like add-to-cart value and checkout revenue for each group

The experiment:loaded Event

When an active experiment is running, the Alhena widget fires the experiment:loaded event after determining which group the user belongs to. This event allows you to:

• Conditionally load other SDKs based on the experiment group

• Customize your page behavior for control group users

• Implement your own tracking logic

Event Signature

window.gleenWidget.on('experiment:loaded', (experimentGroup) => {
    // experimentGroup is either 'test' or 'control'
});

Parameters:

• experimentGroup (string): Either 'test' (user sees Alhena AI) or 'control' (user doesn't see Alhena AI)

Example: Loading Zendesk for Control Group

A common use case is to show an alternative support solution (like Zendesk) to users in the control group who don't see Alhena AI:

<script src="https://app.alhena.ai/sdk/gleenWidget.js"></script>

<script>
    window.gleenWidget.on('experiment:loaded', (experimentGroup) => {
        if (experimentGroup === 'control') {
            // Dynamically load Zendesk for control group users
            const script = document.createElement('script');
            script.id = 'ze-snippet';
            script.src = 'https://static.zdassets.com/ekr/snippet.js?key=7460dcd8-6a51-442e-9675-9781f5954552';
            document.body.appendChild(script);
        }
    });
</script>

This ensures:

• Test group users see Alhena AI

• Control group users see Zendesk instead

• Fair comparison between the two solutions

Important Notes

• The experiment:loaded event only fires when an active experiment is running

• User assignment is deterministic and based on their fingerprint UUID, ensuring consistency across sessions

• In a Widget visibility experiment, users in the control group won't see the Alhena AI widget or any embeddables on your pages (both are automatically disabled)

• In an Embeddables visibility experiment, only embeddables are hidden from the control group; the chat widget shows as usual

• Make sure to set up your cart and checkout events before starting an experiment to track revenue impact

Setting Up Experiments

Prerequisites

Before creating an experiment, ensure you have:

1. Cart Events: Implemented using gleenWidget.sendCartEvent()

2. Checkout Events: Implemented using gleenWidget.sendCheckoutEvent()

See the Cart and Checkout Event API documentation for setup instructions.

Creating an Experiment

1. Navigate to Analytics > Experiments in your dashboard

2. Click Create experiment (or Create first experiment if you haven't run one yet). You can only run one experiment at a time, so the button is disabled while another is running.

3. Enter an Experiment Name

4. Under Schedule, choose an End date:

   • No end date — the experiment runs until you end it yourself. This is the default.

   • Choose an end date — pick a Scheduled end date. The experiment runs through the end of that day in your timezone and then completes automatically (the end date is checked hourly).

5. Under Experiment Type, choose what to test:

   • Widget visibility — show Alhena AI to the test group and hide it from the control group. This hides both the chat widget and any embeddables.

   • Embeddables visibility — show embeddables to the test group and hide them from the control group, while the chat widget stays the same for everyone. Make sure your embeddables are configured and turned on before you start.

   • Nudge visibility — show nudges to the test group and hide them from the control group. Make sure your nudges are set up before you start.

   • Launcher icon — show a new launcher icon to the test group, while the control group keeps the icon from your website settings. Upload the Test group launcher icon in the form.

6. Under Audience Split, set the percentage of website traffic for the Test Group and Control Group (50/50 by default)

7. Check that add to cart and checkout events are being received. Start Experiment stays disabled until both have arrived; use the refresh button to check again.

8. Click Start Experiment

Monitoring Results

The experiment dashboard shows real-time metrics for both groups:

• Exposure: Number of unique users in each group

• Add to Cart Value: Total cart value added

• Checkout Revenue: Total revenue from completed checkouts

• Delta: Percentage difference between test and control groups

Ending an Experiment

When you're ready to conclude:

1. Navigate to your experiment in the dashboard

2. Click Finish experiment (shown when the test group is winning) or Abandon experiment

3. Review the final results and statistical significance

4. The system automatically determines if the experiment was Won, Lost, or Inconclusive

An experiment with a scheduled end date completes on its own at the end of that day. An experiment with no end date keeps running until you end it.

Changing the End Date

While an experiment is running, click Edit end date to add, move, or remove its end date. Choose No end date to let it run until you end it manually, or Choose an end date to set a new date in the future, then click Save.

How It Works Under the Hood

When a user visits your website:

1. The Alhena widget loads and checks if an experiment is active

2. The user is assigned to a group based on their unique fingerprint using a deterministic algorithm

3. The experiment:loaded event fires with either 'test' or 'control'

4. For control group users, whatever the experiment tests (the widget, nudges, or embeddables) is automatically disabled

This ensures:

• Consistency: The same user always sees the same experience

• Even distribution: Users are evenly split between groups

• No bias: Assignment is random and not based on user behavior

Best Practices

1. Run for adequate duration: Allow at least 1-2 weeks to gather meaningful data

2. Don't modify mid-experiment: Changing audience split invalidates results

3. Track other metrics: Use the event to send experiment group to your analytics platform

4. Test your integration: Verify the control group experience works as expected

5. Statistical significance: Wait for enough data before drawing conclusions

FAQ

Q: Can I run multiple experiments simultaneously?

A: No, only one experiment can be RUNNING at a time per bot profile.

Q: What happens to users after an experiment ends?

A: All users will see the normal Alhena AI widget (no experiment assignment).

Q: Can I change the audience split after starting?

A: No, this would invalidate the experiment. You'll need to end and create a new one.

Q: Is the event fired on every page load?

A: Yes, the event fires each time the widget loads and detects an active experiment.