A/B Test Guidelines

Configure different AI responses for A/B testing based on user group assignment.

This example shows how to configure Alhena AI to respond differently based on whether a user is part of a Variant or Control group, enabling you to run A/B tests on AI behavior.

What You'll Learn

Pass A/B test group assignments via user metadata
Configure guidelines with Trigger/Action format
Differentiate AI responses based on group membership

Use Case

A/B testing your AI agent allows you to:

Test different greeting styles or conversation openers
Compare conversion rates between response strategies
Measure the impact of different AI behaviors on user engagement
Validate changes before rolling out to all users

For example:

Variant Group: Display a greeting message to the user
Control Group: Ask the user for their email address

Prerequisites

Alhena SDK installed on your website
An A/B testing system that assigns users to groups

Supported Variables

Alhena supports two metadata variables for A/B testing:

Variable

Purpose

ab_control_group

Identifies Control Group users

ab_variant_group

Identifies Variant/Test Group users

Code Example

Adding user to Control Group

If the user is in your control group, pass ab_control_group in user metadata:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>A/B Test - Control Group</title>
</head>
<body>

<h1>Welcome</h1>

<!-- Alhena SDK -->
<script>
document.gleenConfig = {
    company: 'your-company-key',
    apiBaseUrl: 'https://app.alhena.ai',
    userMetadata: {
        "ab_control_group": "1"
    }
};
</script>
<script src="https://app.alhena.ai/sdk/gleenWidget.js"></script>

</body>
</html>

Adding user to Variant Group

If the user is in your variant group, pass ab_variant_group:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>A/B Test - Variant Group</title>
</head>
<body>

<h1>Welcome</h1>

<!-- Alhena SDK -->
<script>
document.gleenConfig = {
    company: 'your-company-key',
    apiBaseUrl: 'https://app.alhena.ai',
    userMetadata: {
        "ab_variant_group": "1"
    }
};
</script>
<script src="https://app.alhena.ai/sdk/gleenWidget.js"></script>

</body>
</html>

Note: Don't pass both control and variant group for the same user.

Configuring Guidelines

Navigate to AI Settings > Guidelines and create guidelines using the Trigger/Action format to respond differently based on user group.

Variant Group Guideline

Trigger: Metadata in User Information contains "ab_variant_group"

Action:

Reply with a friendly greeting at the beginning of the conversation. Welcome the user warmly before addressing their question.

Control Group Guideline

Trigger: Metadata in User Information contains "ab_control_group"

Action:

Ask for the user's email address at the beginning of the conversation before providing assistance.

How It Works

Group assignment: Your A/B testing system assigns users to either control or variant group
Metadata passed: The SDK sends the group assignment via userMetadata to the AI.
Guideline triggers: Alhena detects the metadata and activates the matching guideline
Differentiated response: The AI follows the action defined for that group

Customization

Dynamic group assignment

Integrate with your A/B testing platform:

// Example with a hypothetical A/B testing library
var userGroup = ABTestingLib.getGroup('ai-greeting-test');

document.gleenConfig = {
    company: 'your-company-key',
    apiBaseUrl: 'https://app.alhena.ai',
    userMetadata: userGroup === 'variant'
        ? { "ab_variant_group": "1" }
        : { "ab_control_group": "1" }
};

Multiple A/B tests

You can run multiple tests by using descriptive metadata values:

document.gleenConfig = {
    company: 'your-company-key',
    apiBaseUrl: 'https://app.alhena.ai',
    userMetadata: {
        "ab_variant_group": "greeting_test_v2"
    }
};

Then create guidelines with more specific triggers:

Trigger: Metadata in User Information "ab_variant_group" equals "greeting_test_v2"

Guidelines - Full guideline configuration documentation
User Metadata - All metadata options
JavaScript API - Widget control methods

PreviousCode Examples NextInternationalization

Last updated 5 months ago