10 Questions That Reveal Your AI Agent's Maturity Gap

Score each question yes or no based on behaviour you have personally observed in production. Not on a demo. Not on a capability page. Observed.

What are the ten diagnostic questions?

QUESTION 1 · 4 OF 15Can it complete a real action end to end?

A return, a cancellation, an address change: performed, not described. Task completion is the whole test here, and every other question is downstream of it. The hardest bar in the study, cleared by 4 of 15, and Agentic Capabilities scored 1.6 of 3.0.

QUESTION 2 · 6 OF 15Can it guide a shopper to a specific product page?

Not a category. Not a search result. The specific product page, opened from the conversation. Only 6 of 15 could; dead-end recommendation appeared in 6 of 15. Evaluate this one end to end, from the recommendation to the page actually opening.

QUESTION 3 · 6 OF 15Does it preserve context as the shopper moves across pages?

If the conversation resets on navigation, chat and storefront are two separate surfaces and no amount of tool use will bridge them. UI disconnect: 6 of 15.

QUESTION 4Can it recommend a bundle under multiple constraints?

Not one product: a coordinated set, which is a multi-step reasoning problem rather than a lookup. A full fragrance-free AM/PM routine under budget. An outfit. A necklace and matching earrings, both nickel-free.

QUESTION 5 · THE FASTESTCan it explain why a product fits?

Ask “why this one?” Evaluate the answer, not the confidence: a reasoning system explains the causal link. A ranking system restates the description or cites a star rating.

This is the fastest single diagnostic on the list, and it detects catalog dumping (5 of 15).

QUESTION 6Does it use reviews at the decision stage?

Not a wall of stars up front, but the one review that addresses the specific doubt the shopper just voiced. Most agents surface neither.

QUESTION 7Can it handle an emotional moment with context intact?

Frustration, urgency, a gift with a deadline. Watch for coherence: does the tone shift without the constraints falling out? Every conversation in the study included an emotional moment for exactly this reason.

QUESTION 8 · 1 OF 15Does it remember responsibly across sessions?

Close the tab. Return tomorrow. This is the one question where a multi-day gap is the whole point. Only 1 of 15 deployments cleared this: the single most discriminating question on the list.

QUESTION 9 · 3 OF 15 FAILEDDoes it avoid overclaiming in safety-sensitive categories?

Ask something it should refuse. The correct answer is a graceful decline with a named referral, while keeping the rest of the conversation useful. Unsafe confidence: 3 of 15.

QUESTION 10Can it show measurable impact?

Conversion, AOV, deflection and ticket quality, measured in production rather than modelled in a deck. An AI agent that explains a return without completing it looks excellent on containment and resolves nothing.

How should you score the result?

For calibration against the wider benchmark, field-wide dimension averages ran from 3.0 on Context down to 1.6 on Agentic Capabilities. Understanding is solved. Doing is not.

What does a “no” answer actually tell you?

A score is a diagnosis, not a verdict. Each no on this list points at a specific missing layer, and knowing which one turns an evaluation into a conversation you can actually have with a vendor.

Read the pattern rather than the individual rows. Failures clustered in questions 1 to 3 are an execution problem: the AI agent understands and cannot act. Failures in 4 to 6 are a reasoning problem, usually a ranking model wearing a conversational interface. Failures in 7 and 8 are a state problem. Failures in 9 and 10 are governance, and they tend to be the cheapest to fix.

This is also where a trace earns its place. When an AI agent fails task completion on question 1, the output the shopper sees looks identical whether an API call returned an error, the tool was never called at all, or the write was refused by your own permissions. Only a trace of the execution separates those three, and only your vendor can produce that trace. Ask for it before you conclude anything about correctness.

What does a completed scorecard look like?

An illustrative walkthrough. The scores below are a composite of patterns Alhena observed, not a single tested deployment.

A mid-market apparel brand decides to evaluate its incumbent AI agent ahead of renewal. One conversation, eleven turns, spanning discovery, a delivery complaint and a return request. Then a second visit the next morning.

Three yes answers. Question 5: asked “why this one?”, the agent explained the fit and fabric reasoning rather than restating the description. Question 6: it surfaced the review that addressed the shopper's stated doubt about sizing. Question 7: the tone held when the conversation turned to the late delivery, and the size constraint survived the shift.

Seven no answers, and they cluster. Questions 1, 2 and 3 all failed: the AI agent explained the returns process rather than starting the return, named a jacket without opening its page, and lost the thread when the shopper navigated away. That is an execution cluster, and it is the same missing layer three times. Question 8 failed the next morning, as it does almost everywhere. Questions 4, 9 and 10 failed too.

Total: 3 of 10. A capable assistant with a structural ceiling, which is where most of the market sits. The whole evaluation took under two hours including the return visit.

What the team did next matters more than the number. They did not replace the agent on the strength of one evaluation. They wrote success criteria for the three questions that mattered most to their own queue, asked the vendor for a trace of the failed return attempt, and re-ran the same test six weeks later. The trace showed the write was never attempted, which turned a vague quality complaint into a specific contractual question about scope. That is the point of scoring at all.

How do you turn this into a repeatable test?

A score you cannot reproduce is an opinion with a number attached, and an evaluation nobody can repeat is not an evaluation. Ten yes-or-no answers become useful the moment you can run them again in three months and compare like with like.

• Write each question as a test case. Not “can it complete an action” but “cancel order 48211 on this account, then confirm the cancellation appears in the order record.” Define the pass condition and its parameters before you start. Written success criteria are what stop a second evaluation from drifting into a different test.
• Capture what you saw, not what you concluded. One line per question: the input you gave, the output you got, and where execution stopped. That record is what lets you debug the difference when a question that passed in March fails in June.
• Score with two people the first time. A second evaluator will disagree on at least one question, and the disagreement shows you which pass conditions are still vague.
• Re-run the whole set as a regression check. Quarterly, and after any release. Agent behaviour is non-deterministic, so run the borderline questions twice before recording a yes.
• Track the score over time, not the absolute number. Capture each quarter's total so the trend is visible: moving from 3 to 5 tells you more about a vendor than either number alone, and it is the measure a renewal conversation actually turns on.

Ten questions, scored the same way each quarter, is a small enough loop that a support lead can own it and iterate on the pass conditions without a project plan. That is the whole validation exercise: a repeatable framework and a closed loop, not a one-off audit.

What makes an AI agent evaluation repeatable and trustworthy?

A useful AI agent eval is more than a scorecard. It is a repeatable evaluation framework with clear success criteria, representative behaviour and enough evidence to explain why an agent passed or failed.

For each test case, define the customer input, the expected outcome and what counts as proof. That proof may be a correct answer, but for an agentic workflow it may also be task completion: a return processed, an address changed or a specific product page opened. The key distinction is between a plausible output and a completed action.

THE EVALUATION LOOPTest the workflow, not just the prompt

A production evaluation should follow the full path from input to outcome. Start with the prompt and context, observe the reasoning and workflow, check any tool use or API execution, then validate the final result against the expected outcome. This is how you evaluate an AI agent end to end rather than judging a single response in isolation.

• Use representative test cases. Include ordinary requests, multi-step conversations and the edge cases your customers actually create.
• Define deterministic pass conditions where possible. AI output can be non-deterministic, but a completed cancellation or updated order record can still have a clear pass or fail.
• Capture the evidence. Record the input, output and execution result so another evaluator can reproduce the score.
• Use observability to investigate failures. A trace can show whether the agent selected the wrong workflow, never attempted a tool call, passed the wrong parameters or hit an API failure.
• Compare against ground truth. For real actions, the source of truth is what happened in the underlying system, not what the agent claimed happened.

This creates a practical evaluation pipeline: define → test → observe → measure → debug → iterate → regression test. The loop matters because AI agents change with prompts, models, integrations and production data.

OBSERVABILITY AND REGRESSIONWhy one passing demo is not enough

A single successful run does not prove reliable behaviour. Agent behaviour can vary across inputs, context and execution paths. That is why important workflows need regression testing after meaningful changes.

Use the same dataset of core test cases to evaluate critical behaviours over time. When something changes, compare the new result with the previous benchmark. If a workflow that once passed now fails, the score tells you that a regression happened. Observability and traces help explain where it happened.

For automated evals, an evaluator or model-based judge can help score relevance, coherence and correctness at scale. But automated evaluation should not replace human validation when the question involves judgement, emotional handling or safety. The strongest framework combines automated checks with observable production behaviour and ground truth.

That is the standard Alhena recommends for evaluating AI agents: measure what the system says, what it does and whether the intended outcome actually happened.

How do you run this properly?

1. One continuous conversation, at least ten turns, covering discovery, then post-purchase, then an action, then an emotional moment.
2. State a hard constraint early and test it at the end, so the workflow is evaluated under load rather than in isolation.
3. Use your vertical's signature task, not a generic question. Evaluate the AI agent on the hardest thing your best salesperson does, because generic prompts produce generic passes.
4. Come back the next day. Question 8 is where the field separates, and where most evaluations quietly stop.
5. Observe, don't infer. If you did not watch it happen, mark it no. Judge the outcome, not the explanation of the outcome.

Frequently asked questions