Short answer: to check what AI shoppers get wrong, start with one question that matters to your customers. Compare the full answer with the exact product and a current source. Establish how the assistant retrieved the information before choosing a fix. If the evidence cannot settle the disagreement, keep it open for review.

Start with the answer you saved

Use the product-question playbook if you need to choose a question. For this diagnosis, keep the complete answer, exact product or variant and dated source together in the product-answer log.

Preserve uncertainty and substitutions. An answer that explicitly compares a near-match is different from one claiming it found the requested model. A tool name without its returned content does not show what the assistant received.

Did it answer the same product question you checked?

Resolve identity and context before calling the answer wrong.

Compatibility: compare the exact part and target model. “Listed as compatible” reports a catalog relationship; it does not certify physical fit beyond the supplied data.
Specifications: check units, model year and the distinction the customer asked about. Two similar names can identify different products.
Variants: match the chosen options, market, currency and time. A captured price or stock value may describe a different context from the current page.
Sizing: check the measurement, chart version and applicable product. Preserve instructions for measurements between sizes.

If the reference omits availability but the assistant reports stock, that is a question to investigate. It is not proof that the assistant invented inventory. Likewise, a missing product in a partial catalog capture does not prove the assistant failed to find it.

Separate a wrong fact from an unknown source

If the answer contradicts a reliable source for the exact item, record the disputed claim. Then ask whether you can inspect the content the assistant received. Without it, the claim can be wrong while its cause remains unknown.

Shopify distinguishes Catalog distribution and open-web discovery from tools in a storefront browser session. A consumer-app answer and a developer-API simulation are separate observations. Use the source-to-answer trace when you need to establish which data reached a particular route.

Choose the smallest supported correction

Scroll sideways for columns out of view. Keyboard: focus the table and use the Left and Right arrow keys.

Finding	Next decision
Maintained product fact is wrong	Correct it with the source owner; keep units, exceptions and identity intact.
Page or channel uses the wrong source	Trace that source in the product-data guide before changing a mapping or adding data.
Store-policy claim is wrong	Inspect the policy and Knowledge Base route.
Retrieved fact is correct but the answer misreads it	Keep both records; duplicating the same fact is not yet a justified fix.
Native browser response misses a useful maintained fact	Evaluate a supported additional tool against that particular gap.
Source, context or reference cannot settle the claim	Mark unresolved and recover the missing record; do not invent a cause.

Choose the specific branch: compatibility, size charts, variant price or out-of-stock answers.

Compare after the correction

Use the before-and-after method to keep the same question, product and relevant context comparable. Preserve no-change and worse results. A browser-simulation improvement does not establish a change in a consumer assistant’s Catalog or search answer.

Where StoreKnows fits

StoreKnows focuses on checking product answers and evaluating proposed tools for compatible agents visiting a storefront. Its checks are simulated; publication uses a saved copy of selected product information. An agent must support and choose the tools. Tool availability does not establish answer correctness, current inventory or use by real shoppers.

The useful decision is whether a proposed intervention helps with your important question. Review the reference, try a supported proposal where one is available, and judge the observed answer before deciding what to publish. That decision can reasonably be “the native tools are enough” or “we need better source information first.”

Next step: choose the correction in the table above for one disputed claim. If the evidence is incomplete, keep the diagnosis unresolved.

Correction, 13 September 2026: the original version led with automated grades from a September research batch. Review identified uncertain references, incomplete catalog coverage and a mixed sample; those grades do not establish public-store accuracy or the cause of a disagreement. We have replaced the aggregate claims with this question-by-question diagnostic. This update reports no new shopper tests or merchant outcomes.

StoreKnows is independently developed and not affiliated with, endorsed by or sponsored by Shopify or any AI model provider.

What AI shoppers get wrong: how to check one product answer