Alhena Experiments: How to A/B Test Your AI Chatbot Revenue Without External Tools

What Is Alhena Experiments?

Alhena Experiments is a native A/B testing feature built into the Alhena AI platform. It splits your website visitors into test and control groups, shows one group the AI shopping assistant and hides it from the other, then measures which group generates more revenue. No external tools required.

The core question it answers: Does your AI chatbot actually increase conversions and revenue, or are you just assuming it does?

According to DRIP's 2026 A/B testing benchmarks, only 20 to 30 percent of e-commerce A/B tests produce a statistically significant winner. That means most brands running chatbot experiments with generic tools are burning time on inconclusive results. Alhena Experiments keeps audience assignment, revenue tracking, and statistical analysis in one place.

Why Do You Need Built-In A/B Testing for Your AI Chatbot?

Most e-commerce teams measure chatbot performance with vanity metrics. They look at chat volume, maybe CSAT scores, and call it a day. None of those prove your AI drives revenue.

A 2026 study by Master of Code found that 80 percent of e-commerce businesses now use chatbots, but fewer than 15 percent can tie chatbot interactions to completed orders. That gap exists because most chatbot platforms don't track revenue natively, and plugging in an external A/B testing tool like Optimizely or VWO adds complexity most CX teams can't handle alone.

Alhena Experiments closes that gap. Here's what it replaces:

• No separate A/B testing platform needed. Audience splitting, tracking, and reporting all live inside Alhena's analytics dashboard.
• No developer setup for each experiment. You configure the test in the dashboard, choose your split, and start it. You'll see results in the same reports you already check daily.
• No manual revenue stitching. Alhena's cart and checkout tracking connects each purchase to the correct experiment group automatically.

For a deeper look at measuring AI-driven revenue, see our guide to holdout tests and incremental revenue measurement.

How Does Alhena Experiments Work?

Four layers work together: the analytics dashboard, the website SDK, experiment events, and revenue tracking. Here's the step-by-step flow.

How Do You Create an Experiment?

Go to Analytics > Experiments inside your Alhena dashboard. Give your experiment a name and choose the audience split. Most brands start with a 50/50 split between test and control.

Before the experiment can start, Alhena checks that your cart and checkout event tracking is already installed and working. If it's not, the dashboard tells you what's missing. No tracking, no experiment. That guard rail is there for a reason. You'll discover any tracking gaps before they waste your testing window.

How Does Visitor Assignment Work?

When a shopper lands on your site, the Alhena website SDK checks whether an active experiment exists. If one does, the SDK assigns the visitor to test or control before anything renders on screen. There's no visible flash of the wrong experience.

Assignment is deterministic, not random. Alhena takes the shopper's persistent fingerprint and converts it into a numeric bucket that maps to your configured split. For a 50/50 experiment:

• Half the buckets = test group (sees Alhena AI)
• The other half = control group (does not see Alhena AI)

The same shopper always lands in the same group, even across multiple sessions. No more crossover contamination from session-based randomization. For e-commerce testing, clean assignment is everything.

What Is the experiment:loaded Event?

Once assignment happens, the SDK fires an experiment:loaded event with either test or control as the value. Your team can use this event to:

• Load an alternate support tool (like a legacy chatbot) for the control group
• Send the experiment group to Google Analytics 4 or Segment for cross-platform analysis
• Trigger custom page behavior like product recommendations based on group assignment

How Does Revenue Tracking Connect to Experiments?

Every cart addition and checkout event gets linked back to the same user fingerprint and experiment group. Alhena's revenue attribution engine handles this automatically, so you don't need to build custom data pipelines or configure third-party analytics.

What Can You Test with Alhena Experiments?

Three experiment types are available right now:

1. Widget visibility. The test group sees the Alhena AI chat widget. The control group doesn't. Most brands run this first because it answers the fundamental question: does showing AI to shoppers increase conversions and revenue?
2. Nudge visibility. The test group sees proactive AI nudges (product recommendations, exit-intent messages, free shipping reminders). The control group gets the standard chat widget without nudges. That isolates whether proactive messages actually move the needle on revenue.
3. FAQ answering mode. Compares different product FAQ answer experiences, like inline answers on the product page versus answers delivered through the chat widget. You'll see which format drives more add-to-cart actions.

The main dashboard flow focuses on "Alhena AI versus no Alhena AI" revenue experiments. This is the test most e-commerce brands run first because it directly proves ROI to leadership.

If you want to go beyond these three test types, our ecommerce AI A/B testing guide covers seven high-impact chatbot tests for conversion rate optimization.

What Metrics Does Alhena Experiments Track?

Both groups get compared across 11 revenue and engagement metrics:

• Total exposed users per group
• Conversion rate (purchases divided by exposed users)
• Total cart value across all add-to-cart events
• Users who added to cart
• Average cart value
• Total revenue from completed purchases
• Total purchases
• Average checkout value
• Average items per purchase
• Revenue per user (RPU), the primary metric for statistical testing
• Delta between test and control, the difference that tells you if AI is winning

Revenue per user is the metric Alhena uses for significance calculations. It's a better signal than conversion rate alone because it accounts for order value differences between groups. A test group might convert at the same rate as control but generate 20 percent higher order values. RPU catches that.

You can monitor all of these in real time from the Alhena analytics dashboard.

How Does Alhena Determine a Winner?

You can't declare a winner based on gut feeling. The system enforces two hard requirements before marking a result:

1. Minimum checkout volume across both groups combined. Alhena won't declare a result until enough shoppers in both groups have completed purchases.
2. Statistical significance. Alhena runs a statistical test on revenue per user and only flags a result when the confidence level is high enough to rule out random chance.

Once those thresholds are met, the experiment gets one of three outcomes:

• WON: The test group (with Alhena AI) outperformed the control group with statistical significance. Your AI is driving real revenue.
• LOST: The control group outperformed the test group. Rare, but useful. It points to what needs fixing before you go wider.
• INCONCLUSIVE: Not enough data or no statistically reliable difference between groups. You either need more traffic or a longer test window.

Experiments default to a 30-day run. They can also end automatically when the configured end date passes. Only one experiment can run at a time per bot profile, which prevents overlapping tests from contaminating each other's data.

Brands like Tatcha have seen 3x conversion rates and 38 percent AOV uplift with Alhena AI. Puffy hit 63 percent automated inquiry resolution with 90 percent CSAT. These numbers come from real experiment data, not guesswork.

How Is This Different from Using Optimizely or Google Optimize?

You could technically A/B test your AI chatbot with a standalone experimentation platform. But here's why built-in testing inside Alhena is better for e-commerce teams:

• Single-system attribution. External tools need you to stitch experiment data to revenue data across platforms. Alhena does this natively because it already tracks carts and checkouts.
• Fingerprint-based persistence. Most external tools rely on cookies or session IDs, which break when shoppers switch devices or clear cookies. Alhena's fingerprint UUID persists across sessions, giving you cleaner group assignment.
• No engineering overhead. Setting up Optimizely or LaunchDarkly for chatbot experiments requires developer time for SDK integration, event forwarding, and data pipeline configuration. Alhena Experiments needs zero additional code if you already have the widget installed.
• Revenue-first metrics. External A/B tools track clicks, pageviews, and conversions generically. Alhena tracks e-commerce-specific metrics like average cart value, checkout value, and revenue per user out of the box.

That said, if you're already running a mature experimentation program with LaunchDarkly or VWO for product features beyond AI, you can still use the experiment:loaded event to feed Alhena experiment data into those platforms. The Alhena SDK reference documents exactly how.

What Do You Need Before Running Your First Experiment?

Before you click "Start Experiment," make sure these prerequisites are in place.

Pre-Experiment Checklist

1. Cart tracking installed. Your store needs to fire add-to-cart events with product IDs and values. The dashboard will block experiment creation if this isn't working.
2. Checkout tracking installed. Same requirement for completed purchases. Both cart and checkout events must fire reliably before you start.
3. Enough traffic. You need enough checkout volume across both groups to reach significance. The exact threshold depends on your traffic and order patterns. Low-volume stores may need to run experiments longer than 30 days. Don’t stop early just because the numbers look good.
4. A plan for the control group. Control users won't see the Alhena AI widget. If your brand still wants support coverage for those visitors, set up an alternate support tool using the experiment:loaded event callback.
5. One experiment at a time. Only one experiment can run per bot profile. Finish or stop the current experiment before starting a new one.

Need help with the initial setup? Alhena deploys in under 48 hours, and the team can walk you through experiment configuration during onboarding. Book a demo to get started.

When Should You Run an Alhena Experiment?

Not every moment is the right time to test. Here are the scenarios where experiments deliver the most value:

• Right after deploying Alhena AI. Run a 30-day "AI on vs. AI off" experiment during your first month to establish a revenue baseline. This gives you hard data for your next budget review.
• Before rolling out a new AI behavior. Testing nudge visibility or a new FAQ format on 50 percent of traffic reduces rollout risk. If the new behavior hurts conversions, you’ll catch it before it affects everyone.
• During seasonal traffic spikes. Higher traffic means faster statistical significance. Black Friday or back-to-school periods can cut your experiment duration in half. Our Shopify AI chatbot ROI guide explains how to time experiments for maximum impact.
• When leadership asks "what's the ROI?" An experiment with WON status and a clear revenue-per-user delta is the strongest evidence you can present. It's causal proof, not a correlation chart. Don't wait for the next planning cycle to start.

What Happens After the Experiment Ends?

Once an experiment hits its end date or you manually stop it, Alhena locks the results. You'll see a clear verdict: WON, LOST, or INCONCLUSIVE.

If the result is WON, you have the data to justify keeping Alhena AI on for 100 percent of your traffic. Share the revenue-per-user delta with your CFO. That single number tells the story better than any slide deck.

If the result is LOST or INCONCLUSIVE, don't panic. This usually points to a configuration issue, not a platform problem. Common causes include:

• Greeting messages that don't match shopper intent
• Product catalog gaps in the AI's knowledge base
• Nudge timing that interrupts rather than assists, or offers and discount messaging that feels pushy

Review your CRO chat platform checklist to identify what to tune, then run a follow-up experiment after making changes.

Key Takeaways

• Alhena Experiments is a built-in e-commerce A/B testing feature that splits visitors into test and control groups and measures revenue impact directly.
• No external tools required. Audience assignment, event tracking, revenue attribution, and statistical significance all happen inside Alhena.
• Deterministic fingerprint-based assignment keeps shoppers in the same group across sessions, eliminating cookie-based contamination.
• Three e-commerce experiment types: widget visibility, nudge visibility, and FAQ answering mode.
• Results require real evidence: sufficient checkout volume and statistical significance before Alhena declares a winner.
• Revenue per user is the primary metric, not just conversion rate, because it captures order value differences between groups.
• One experiment per bot profile at a time to prevent data contamination.
• Cart and checkout tracking must be installed first. The dashboard blocks experiment creation until tracking is verified.

Ready to prove your AI chatbot drives revenue, not just conversations? Book a demo with Alhena AI or start for free with 25 conversations.

Alhena AI

Schedule a Demo

Frequently Asked Questions