How to Measure Product Recommendation Accuracy in Conversational Commerce

A shopper asks for waterproof trail shoes under $120, available in US size 9 with a wide fit. The shopping assistant returns three options, each with a clear description and a purchase link.

One meets the brief. One costs $135. The third comes only in a regular width.

The assistant understood the category. It may even have described every product fact correctly. But two of its three recommendations gave the shopper something they had already ruled out.

That is the gap a recommendation accuracy audit needs to catch.

To measure product recommendation accuracy, check each suggested product and variant against the shopper’s active requirements and verified product information. Then evaluate the complete recommendation list, whether requirements survive follow-up questions, and whether the assistant handles missing information or unavailable matches appropriately. Track shopping outcomes alongside these quality measures.

For the broader role of conversational discovery, Alhena’s guide to AI shopping assistants covers how these experiences work. This article focuses on a narrower operational question: how to tell whether the products being recommended actually belong in the conversation.

What counts as an accurate product recommendation?

An accurate product recommendation identifies a suitable product and purchasable variant, meets the shopper’s stated must-have requirements, and supports the reasons for recommending it with reliable product evidence.

The variant matters. A shoe available in size 9 does not satisfy a request for a size 9 wide if that particular combination is unavailable. A product family containing a cotton version does not establish that the recommended colour and finish use the same material.

Before scoring anything, separate the shopper’s requirements into three groups.

Interpret the wording carefully. Colour can be a preference in one conversation and a hard requirement in another. The evaluator should follow what the shopper said, rather than impose a fixed rule for every attribute.

Use three labels for product fit:

• Pass: The product and relevant variant meet every hard requirement, suit the stated use, and have supporting evidence.
• Fail: At least one hard requirement or essential use condition is contradicted by the evidence.
• Unverified: There is not enough evidence to establish suitability, with no confirmed contradiction already making the item a fail.

An unverified recommendation does not earn a pass. However, missing width information is a different problem from a confirmed regular-width recommendation. Preserve that distinction so the audit leads to the right correction.

Keep preference matching separate from mandatory eligibility. A preferred color should not compensate for the wrong size in an averaged score.

Build an evaluation set that reflects real shopping conversations

Start with a manageable diagnostic review, such as 50–100 conversations. Treat that as an initial workload suggestion, not a statistically sufficient sample for every store.

Use a random sample of recent shopping conversations to understand ordinary performance. Keep complaints, failed conversations, and deliberately difficult test cases in a separate diagnostic set. Those cases are valuable, but their error rate cannot stand in for the error rate across all shoppers.

Include different kinds of requests:

• A straightforward product question with one requirement.
• A request combining budget, size, material, and intended use.
• A follow-up that adds a preference while leaving earlier requirements active.
• A follow-up that explicitly changes a requirement.
• A request for an unavailable variant or a combination the catalogue cannot satisfy.
• A request that needs clarification before a useful recommendation is possible.

For each case, preserve the conversation, product and variant IDs, recommendation order, storefront, and timestamp. Save the relevant product evidence or a reference to a version that can be retrieved later.

Evaluate price and availability at the time of the recommendation. Today’s stock status cannot establish whether yesterday’s recommendation was wrong. If the historical evidence is unavailable, record that limitation as unverified.

Have merchandising or a product specialist identify acceptable options independently of the assistant’s answer. There may be several correct products. A reference set containing only the item eventually purchased can unfairly mark other suitable recommendations as wrong.

Ask a second reviewer to score a subset. If reviewers disagree about what “suitable for everyday use” means, clarify the rubric before expanding the audit. Otherwise, changes in reviewer judgment may look like changes in AI performance.

Which product recommendation metrics should you track

Use measures that distinguish product selection, evidence quality, and conversational behaviour. The following scorecard is a recommended merchant evaluation method; these names do not imply standardized industry benchmarks or automatically available dashboard fields.

First, establish the units:

• A product recommendation is a distinct product and relevant variant offered as a match to the active shopping brief. Count it once per assistant turn.
• A recommendation turn is an assistant response containing one or more such recommendations.
• An eligible recommendation opportunity is a turn where a product recommendation is appropriate to the shopper’s current request, enough shopper information is available, and at least one suitable, available variant exists in the catalogue, established independently of the assistant’s response.

Clearly labelled alternatives that require changing the brief should have separate worksheet rows. They are not verified matches to the original request. If the assistant presents them as satisfying the unchanged brief, include and score them as recommendations.

Calculate each rate below as the numerator divided by the denominator, multiplied by 100.

Publish the numerator, denominator, review period, and sample type beside each percentage. If the denominator is zero, report “N/A.” Also show the share of recommendations labelled 'unverified'; it should not disappear inside the fit score.

For claim support, agree in advance which factual claims to audit, including decision-critical claims about size, price, material, functionality, and availability. Several correct minor details should not hide one unsupported claim that determines whether a shopper buys.

For valid-option coverage, include eligible responses that contain no recommendation at all. Otherwise, an assistant could appear accurate simply by avoiding difficult requests.

For retention, assess every proposed match against the carried-over hard requirements. Score new requirements through product fit, and remove requirements the shopper explicitly changes.

Read the item and turn the measures together. Long recommendation lists contribute more product rows, while the turn-level violation rate shows how often shoppers encounter a problematic list.

For broader commercial and operational reporting, use these measures alongside your AI shopping assistant KPIs.

Where precision and recall fit

Established ranking metrics can supplement this scorecard when your team has reliable relevance labels.

Precision@k measures relevant items within the first k positions, divided by k. It differs from product-fit rate, which uses the number of items actually recommended. For example, one suitable result has 100% product fit, but Precision@3 is 33.3% under a fixed-k definition. Apache Spark’s ranking evaluation documentation explicitly retains k as the denominator when fewer results are returned.

Recall@k measures how many relevant items were retrieved within those positions compared with the full reference set of relevant items. It requires knowing which products qualify. Valid-option coverage asks a different question: did an eligible shopping opportunity receive at least one suitable option?

Use these metrics only where their reference data and assumptions fit the task. An assistant should not add unsuitable products merely to fill three positions.

A worked example of recommendation accuracy

Return to the shopper asking for waterproof trail shoes under $120, in US size 9 with a wide fit.

The following products and results are fictional. Assume the product facts and availability were verified for the time of the conversation, and all three options were offered as matches to the unchanged request.

The scores for this single fictional turn are:

• Verified product-fit rate: 1 suitable recommendation ÷ 3 recommendations = 33.3%.
• Hard-requirement violation rate: 1 affected turn ÷ 1 reviewed recommendation turn = 100%.
• Valid-option coverage: 1 eligible opportunity receiving a suitable option ÷ 1 eligible opportunity = 100%.

Those results are consistent. The assistant found a valid option, but most of its recommendations did not qualify. Coverage alone would miss that problem.

The individual product specifications could still be factually correct. That would not make the selection appropriate. Any additional statement that “all three meet your requirements” would itself be an unsupported suitability claim and should fail the relevant check.

These percentages explain the scoring method. They are not evidence of a real system’s performance.

Test what happens after the first answer

Conversational commerce gives shoppers room to refine a request. Your evaluation needs to follow those changes.

Suppose the next message is: “Do you have anything lighter?”

The budget, waterproof requirement, size, and width are still active. The assistant now needs evidence for the weight comparison as well. If weight is missing from the product information, it should say that it cannot verify which option is lighter.

Now suppose the shopper adds: “I can go up to $140, but a wide fit is essential.” The budget has changed. Recommending a $135 option no longer violates that requirement, provided the other conditions still hold.

A helpful no-match answer identifies the blocker and gives the shopper a choice: change the budget, consider another specification, or wait for availability where that option exists.

Distinguish a verified no-match result from an evidence gap. “We do not have a qualifying item” and “I cannot verify that these items qualify” describe different situations. The evaluator should know which one the catalog supports.

Research on evaluating conversational recommender systems supports evaluating the wider interaction and user experience alongside recommendation algorithms. A single successful first answer cannot show how well the conversation handles changing requirements.

Use a worksheet that makes each score explainable

Copy the following record for each distinct product and variant recommended in a turn. The entries illustrate Product A from the fictional example.

Keep a separate turn log that includes clarification, no-match, and empty-result responses. A worksheet containing only recommended products cannot measure missed opportunities to recommend.

For objective fields, automate direct comparisons where practical: whether a SKU exists, whether its price meets the stated limit, and whether the required variant was available. Use product specialists for intended-use judgments and ambiguous requests.

An AI evaluator can help extract requirements or propose scores, but give it the actual evidence and check its judgments against human-reviewed examples. Research on LLMs as judges has documented position, verbosity, and self-enhancement biases. A fluent explanation of why a recommendation was “good” should not become the evidence that it was correct.

How to put the review into an Alhena workflow

Teams using Alhena’s AI Shopping Assistant can connect the scorecard to documented conversation review, testing, and analytics workflows.

1. Inspect the conversation and its sources. Open the relevant exchange in Conversations and check Answer Sources to see which documents informed the answer. The Adding Human Feedback guide explains how to review sources and provide corrective feedback.
2. Investigate flagged conversations. Alhena’s Quality Assurance tools flag conversations that may need review. Use these to investigate likely problems, alongside a separate representative sample for reporting overall quality.
3. Replay the shopping scenarios. Use Playground to test responses and changes to guidelines, sources, or product recommendations before they affect live shoppers. Repeat the follow-up messages as well as the opening question.
4. Review shopping outcomes. Use Revenue Impact alongside the audit. Checkout revenue reporting depends on connected checkout data; cart GMV is not the same as completed sales.

Maintain the proposed product-fit, retention, and coverage calculations in your evaluation worksheet or analytics environment. These are measures for your team to implement using the review evidence, rather than assumed native dashboard fields.

Turn recommendation errors into specific improvements

A useful review ends with an owner and a retest.

For catalog issues, Alhena’s guide to product data quality for AI shopping assistants covers the underlying remediation work.

Start with a regular review cadence your team can sustain, such as weekly, and add targeted checks after catalog, promotion, model, or guideline changes. Keep a fixed regression set for comparing versions and a fresh production sample for detecting new problems.

Set targets from your baseline and the consequences of each failure. A minor preference mismatch and a confirmed violation of a mandatory requirement should remain visible as different outcomes.

This recommendation audit can sit within your broader AI chatbot quality assurance program, with clear responsibilities for reviewing, correcting, and retesting issues.

Connect accuracy to commercial outcomes

Clicks, add-to-cart activity, purchases, and returns help explain what happened after a recommendation. They do not independently prove that the product met the shopper’s requirements.

A suitable recommendation may not convert because the shopper is still comparing options. An unsuitable product may sell because the customer overlooks the mismatch, then generate a return later.

Where the data allows, connect reviewed recommendations to the specific variants purchased and relevant return reasons. Compare similar categories and allow time for returns to appear before drawing conclusions. Alhena’s revenue attribution guide explains how conversations can be connected to commerce events.

Keep attribution separate from causal impact. Shoppers who choose to engage with an assistant may already differ from those who do not. Use a properly designed randomized experiment when claiming a change caused higher revenue; the guide to measuring incremental revenue with holdout tests covers that measurement approach.

The practical goal is to find and fix the reasons recommendations fail, then check whether shoppers have a better experience and the business benefits. Revenue adds context to the quality audit; it does not replace the product evidence.

Frequently asked questions

How do you calculate product recommendation accuracy?

For a practical product-fit measure, divide the number of verified suitable product and variant recommendations by all recommendations audited, then multiply by 100. Keep unverified items in the denominator and report them separately. Pair the result with turn-level violations and valid-option coverage so it reflects both suitability and usefulness.

Does a higher conversion rate mean recommendations are more accurate?

Not necessarily. Conversion records a purchase, while recommendation accuracy checks whether the proposed product satisfies the shopper’s requirements. Measure both, and examine the purchased variant and relevant return reasons where possible. A randomized experiment is needed to support a causal claim about revenue improvement.

What is a good accuracy rate for an AI shopping assistant?

A useful target depends on the scoring definition, catalog, request complexity, and consequences of errors. Establish your own baseline before comparing results. Ask what a reported percentage counts, how unknowns are handled, and whether the sample represents normal traffic. A single headline percentage is insufficient for a purchasing decision.

How should you score a response when no product matches?

Score whether the assistant correctly identifies that no qualifying option exists, explains the relevant limitation, and offers an appropriate next step without silently changing the requirements. Exclude verified no-match cases from valid-option coverage. If the evidence is incomplete, record uncertainty rather than treating it as proof that no match exists.

How do you test whether an assistant remembers shopper requirements?

Use conversations where follow-up messages add preferences or explicitly change one condition. At each recommendation turn, compare the suggested variants with all requirements still in force. “Anything lighter?” should preserve the budget and size; “I can spend up to $140” should update the budget while preserving the remaining conditions.

Can AI automatically evaluate its own recommendations?

AI can assist with requirement extraction, claim identification, and initial scoring. Reliable evaluation still needs approved product evidence, direct checks for objective fields, and calibrated human review for ambiguous cases. Asking the same assistant whether it gave a good recommendation is not independent verification.

Make every recommendation earn its place

Begin with the shopper’s brief and the exact variant being offered. Check the evidence, score the complete list, follow the conversation, and assign each failure a correction.

That process makes an accuracy score useful: your team can explain which recommendations passed, which failed, what remains unverified, and what needs to change.

Bring a few challenging product questions to an Alhena demo and use this scorecard to assess how the assistant interprets requirements, selects options, and supports its answers.