Every merchant tries the same thing first. Paste the FAQ into a chatbot builder, drop it on the site, hope for the best. Six weeks later the tickets are still there.
The bot handles policy questions fine. Then someone asks about a specific order, checks stock on a variant, or tries to start a return, and it falls apart. A recent Shopify community thread captured the frustration exactly: merchants feel stuck between expensive human agents and AI that hallucinates or answers in generalities.
The gap is not the model. Every major vendor is running a capable frontier model in 2026. The gap is what the model is allowed to reach.
- FAQ-only chatbots typically cap at 20 to 30 percent auto-resolution, regardless of which model powers them.
- A cleaned, structured knowledge base gets an ecommerce AI agent to roughly 40 percent. Connected action endpoints take it to 70 percent.
- Deflection is not resolution. An agent that says "check our FAQ page" deflected the ticket. It resolved nothing.
- A realistic ramp is 40 percent at day 30, 55 percent at day 60, 70 percent at day 90.
- When resolution plateaus, the fix is almost always rewriting knowledge articles, not retraining the model.
- Alhena customers have reached 63 percent automated resolution (Puffy), 86 percent deflection (Crocus), and 80 percent inquiry automation (Manawa).
This is the resolution half of AI agent training: knowledge pipelines, action endpoints, guardrails, and measurement, scored on auto-resolution rate and CSAT. If your priority is conversion rather than ticket volume, start with the companion guide on training ecommerce AI agents to sell, which covers discovery logic, catalog reasoning, and brand voice.
What is the difference between deflection and resolution?
Deflection means the AI handled a conversation without routing it to a human. Resolution means the customer's problem was actually solved. A bot that quotes your return policy is doing retrieval. A bot that looks up order #48291, confirms it falls inside the return window, and issues a prepaid label is doing resolution.
This distinction decides everything downstream, because it decides what you build. Optimise for deflection and you will build a very good deflector: an agent that ends conversations without solving anything, and customers who email you twice.
| Retrieval / deflection | Resolution | |
|---|---|---|
| What it does | Returns information the customer could have found | Changes the state of an order, account, or subscription |
| Needs | A knowledge base | A knowledge base plus connected action endpoints |
| Ceiling | 20–30% of ecommerce tickets | 65–70% within 90 days |
| Customer outcome | "Check your email for tracking" | "Your order ships tomorrow, here's the tracking number" |
| Follow-up rate | High — the issue is still open | Low — the issue is closed |
Why does FAQ-only training cap at 30 percent?
Because the remaining 40-plus percentage points are not language problems. They are data and permission problems. No amount of prompt engineering lets an agent read an order it cannot access or cancel a subscription it has no endpoint for.
Most DTC brands load their help center into a chatbot and stop. The agent gets good at the questions that were already answered on a public page, which is exactly the segment of tickets that was cheapest to handle in the first place. Everything expensive stays in the queue.
Five questions your FAQ bot will always get wrong
Run this diagnostic before you rebuild anything. Ask your current bot these five and watch.
"Where's my order #48291?"
An FAQ bot has no order data. It quotes the shipping policy or tells the customer to check their email. A data-connected agent pulls live status from Shopify or WooCommerce and answers directly.
"Do you have the navy hoodie in medium?"
FAQ bots do not know your inventory. They link to the product page and hope. A connected agent checks real-time stock and says yes or no in the chat.
"I bought this last week and want to return it."
The bot may recite the policy, but it cannot verify the purchase date, confirm eligibility, or generate a label. That needs order history and policy logic working together.
"Can I change my shipping address?"
This requires an action, not information. An FAQ bot cannot modify an order. A connected agent checks whether it has shipped and updates the address if it has not.
"I got the wrong item."
The hardest of the five. It needs the order contents, the fulfilment record, your replacement policy, and the authority to act on all three. Most bots apologise and escalate.
Fail three or more and you are sitting at the FAQ ceiling. The fix is not better prompts. It is connecting the agent to live store data. For the technical picture, see how ecommerce AI gets fresh data without a warehouse and why grounded agents beat waiting for unified commerce.
What data should you feed an ecommerce AI agent?
Four tiers, each on a different refresh cadence. Training an ecommerce agent is less machine learning than pipeline construction: the right sources, kept current, structured so the agent can reason over them.
| Tier | What goes in | Sync cadence | What it unlocks |
|---|---|---|---|
| 1. Static knowledge | Help center articles, return and warranty policies, shipping timelines, sizing guides | Daily | The foundation. Policy answers. |
| 2. Product catalog | Titles, descriptions, attributes, pricing, inventory levels, stock status, reviews | Hourly | Availability answers and recommendations. Without live inventory, the agent hallucinates stock. |
| 3. Order & subscription data | Order status, tracking numbers, subscription schedules, payment history | Real time | Actual resolution. This is the tier most vendors skip. |
| 4. Ticket transcripts | Resolved support conversations from your helpdesk | Batch ingest | How real customers phrase things, and the resolution workflows no FAQ page documents. |
One detail most vendors skip: versioning. You need a snapshot of what the agent knew during any given conversation, or you cannot audit a bad answer after the fact. Keeping four tiers current is its own discipline — our knowledge freshness guide for operators covers sync schedules and staleness detection, and Training Monitor covers auditing individual training runs.
How do you clean your docs and connect actions?
Break every article into atomic question-and-answer pairs, delete contradictions, tag everything with metadata, then map your top ten ticket reasons to real endpoints. The cleaning gets you to 40 percent. The endpoints get you to 70.
Raw help center articles are written for humans scanning a page, not for retrieval. A 2,000-word returns guide should become eight to twelve discrete chunks. Three articles giving different answers about your return window will damage accuracy more than a missing article would — contradiction is worse than absence. Add structured metadata for audience, region, and SKU scope so the agent can disambiguate rather than guess. If your content lives in a helpdesk already, you can train directly from your Zendesk or Freshdesk knowledge base.
Then map ticket reasons to endpoints. The first six on this list require the agent to do something:
| Ticket reason | Requires |
|---|---|
| Order tracking lookup | Action — live order read |
| Return initiation | Action — eligibility check plus label generation |
| Cancellation, refund, or exchange | Action — write access to the order |
| Shipping address change | Action — fulfilment status check plus edit |
| Promo code validation | Action — live discount rules |
| Subscription pause or cancel | Action — subscription platform write |
| Product recommendation | Knowledge — catalog reasoning |
| Cart and inventory questions | Knowledge — live stock read |
If you need different specialists owning different workflows — one agent for subscriptions, another for returns — custom agents let you scope each one to a single job with its own tools and boundaries.
How do you stop an ecommerce AI agent from hallucinating?
Four guardrails, all set before launch: topic allow and deny lists, confidence thresholds that trigger an honest "I don't know," human approval queues for knowledge updates, and red-team prompt testing after every major change.
In ecommerce a hallucinated answer costs money twice — once on the wrong outcome, once on the trust that does not come back. Accuracy is not a nice-to-have here, it is the precondition for letting the agent act at all.
Topic allow and deny lists
Never answer medical or legal questions. Never quote competitor pricing. Explicit denial prevents the most damaging failure modes before they can happen.
Confidence thresholds
Every response carries a confidence score. Below your threshold, the agent says it does not know and routes to a human. An agent that admits uncertainty earns more trust than one that always has an answer.
Approval queues
When the system suggests an automatic knowledge base update, a human reviews it before it goes live. No autonomous edits to the source of truth.
Red-team prompt testing
Before launch, try to break it. Get it to invent products, fabricate policies, or leak internal data. Re-run these tests after every significant knowledge update, not just once.
PII handling belongs in the same conversation. Redact personally identifiable information at ingestion, mask it at inference, and define retention periods for any customer data used in training under GDPR and CCPA. Data-connected must not mean data-leaky: for order lookups, Alhena runs identity verification gates before sharing anything sensitive. See how Alhena verifies shoppers before sharing order data.
How do you measure your way to 70 percent?
Four metrics predict resolution rate, and none of them is deflection. Track coverage, accuracy, action success rate, and escalation precision weekly.
| Metric | What it asks | Threshold | If it's low |
|---|---|---|---|
| Coverage | What share of questions can your knowledge base answer at all? | 80%+ | You have knowledge gaps. Write, don't tune. |
| Accuracy | When it answers, is it right? Sample 50 conversations weekly. | 90%+ | You have contradictory sources. Deduplicate. |
| Action success rate | Do attempted actions complete without error or timeout? | 95%+ | Integration problem, not a model problem. |
| Escalation precision | When it escalates, did it need to? | Context-dependent | Confidence thresholds are set too conservatively. |
The weekly review is the part teams skip, and it is the part that compounds. Our guide to coaching your AI agent every week covers the tuning cadence in detail; brands that follow a structured playbook in the first 30 days reach 80 percent automation by month three.
How Alhena handles this end to end
Most tools hand you a framework and leave the pipeline, the integrations, and the guardrails as an exercise for your team. That is why so many implementations stall at 30 percent with a capable model underneath.
The Support Concierge ingests from help desks, knowledge bases, prior tickets, product feeds, and CRM platforms, then connects natively to Shopify, WooCommerce, and Salesforce Commerce Cloud, alongside helpdesks including Zendesk, Freshdesk, Gorgias, Intercom, and Zoho Desk. When a conversation does need a person, Agent Assist passes the full history across so the customer never repeats themselves.
Read the full stories from Puffy, Crocus, and Manawa. Setup runs in days, not the six-week implementations enterprise vendors quote: create an account, connect knowledge sources, install the widget, set brand voice, go live. No developer resources required.
Resolution is only half of what a commerce agent should do. The same platform runs an AI Shopping Assistant trained on discovery and conversion rather than tickets — that curriculum is covered in the companion guide on training ecommerce AI agents to sell. Alhena also runs social commerce across Instagram DMs and WhatsApp and Voice AI for phone support, all sharing one context.
Before you start
Two things worth reading first. If you are still comparing vendors, the AI agent evaluation checklist gives you ten questions that separate a real platform from an expensive chatbot, and our comparison of 16 ecommerce AI agents scores the field. If you already own a tool that is underperforming, the nine mistakes brands make implementing AI agents will probably diagnose it faster than a rebuild would.
Skip the six-week implementation
Connect your store and helpdesk, and see resolution on your own ticket history. Free plan includes 25 conversations.
Frequently asked questions
Build a tiered knowledge pipeline: help center content and policies synced daily, product catalog with inventory synced hourly, order and subscription data synced in real time, and past ticket transcripts batch-ingested. Clean everything into atomic question-and-answer pairs, remove contradictions, and tag with metadata like region and audience. Then connect action endpoints for order tracking, returns, and cancellations. The knowledge work gets you to roughly 40 percent resolution; the endpoints take you to 70.
Different curriculum, different scoreboard, same agent. Resolution training centres on knowledge pipelines, action endpoints, and confidence thresholds, measured in auto-resolution rate and CSAT. Selling training centres on discovery logic, catalog reasoning, and brand voice, measured in conversion rate and average order value. For the selling side, see our guide on training ecommerce AI agents to sell.
A well-configured agent reaches 65 to 70 percent autonomous resolution within 90 days. FAQ-only chatbots cap at 20 to 30 percent regardless of the underlying model. The jump comes from connecting action endpoints and removing contradictions from the knowledge base. Alhena customers have reached 63 percent automated resolution (Puffy) and 86 percent deflection (Crocus).
Deflection means the AI handled the conversation without routing it to a human. Resolution means the customer's issue was actually solved. An agent that replies "check our FAQ page" counts as a deflection but resolved nothing, and that customer usually comes back. True resolution requires the agent to take an action: look up order status, process a return, pause a subscription.
Four guardrails. Topic allow and deny lists that block medical, legal, and competitor pricing questions. Confidence thresholds that trigger an "I don't know" below a set score. Approval queues so no knowledge base update goes live without human review. Red-team prompt testing before launch and after every major knowledge change. Alhena additionally restricts responses to verified company source material, with full auditability on every answer.
Four tiers on four cadences: static knowledge such as help center articles, policies, and sizing guides (daily); product catalog including pricing, inventory, and reviews (hourly); transactional data covering order status, tracking, and subscription schedules (real time); and historical ticket transcripts from your helpdesk (batch). Tier three is the one most implementations skip, and it is the one that converts deflection into resolution.
Track four metrics weekly: coverage (what share of questions the knowledge base can answer), accuracy (correctness on a sample of 50 conversations), action success rate (whether attempted actions complete without error), and escalation precision (whether escalated tickets genuinely needed a human). If accuracy falls below 90 percent, the fix is almost always rewriting knowledge articles rather than reconfiguring the agent.
