How often do AI assistants answer product questions correctly on specialist stores?

In our September 2026 batch of 340 questions across 34 option-heavy storefronts, 45% of answers were fully right, 42% partly right and 12% wrong, using one model (OpenAI's GPT-5.4 mini via the API) and only Shopify's native storefront tools. Fit questions were right 81% of the time; comparisons 24%.

What kind of question do AI shoppers get wrong most?

Comparisons between two products and filtered searches (in-stock, under a price, with a property). Both need a fact per product that the storefront tools do not return, such as a spec, a variant price or a stock state.

Does this mean Shopify's AI tools are broken?

No. On a small text-rich store the same tools answered 10 of 10 questions. The misses concentrate where product facts live in metafields, options with hundreds of values, or spec tables the tools cannot read.

What AI shoppers get wrong about specialist products: 34 stores, 340 questions

We ran simulated AI shoppers against 34 public Shopify storefronts selling parts, tools, cameras and outdoor gear, using only the storefront tools every store already has. 45% of answers were fully right. Four of five classifiable misses were a single missing fact about a single product, not a failure to find it.

Short answer: across 340 shopper questions on 34 specialist Shopify storefronts, a simulated AI shopper using only the store’s native tools got 45% fully right, 42% partly right and 12% wrong. The assistant almost always found the product. What it missed was one fact about it: the variant’s price, whether it was in stock, a spec, a compatibility note. Of the 143 misses our judge could classify, 119 were that shape.

Why we ran this

Shoppers now ask assistants the questions specialist merchants field all day: will this fit, which size, what is actually different between these two. Shopify has done the plumbing: its catalog is syndicated to ChatGPT, Google, Copilot and Meta by default, and since 21 August 2026 every Liquid storefront exposes storefront tools that browser-based agents can call. What nobody had published is what an agent actually answers when it uses those tools on a store where the deciding facts are specs, options and charts.

So we measured it, on stores chosen because they are hard: bike parts, tool makers, camera and film shops, outdoor gear, 3D-printer parts, espresso equipment, phone accessories. This is not a random sample of Shopify stores and it supports no claim about stores in general. It is a picture of where the details decide the sale.

Method, in full

Stores: 40 public storefronts listed; 34 scanned. Three could not be scanned because their pages exposed no storefront tools (headless or custom builds); three failed at question generation. Six control stores selling food, drink and basics apparel ran alongside.
Questions: 10 per store, generated from the store’s own public catalog with a recorded reference answer for each, in six types: fit, compatibility, spec filter, comparison, policy, and open-ended “help me choose”.
Shopper: OpenAI’s GPT-5.4 mini via the API, at most six tool calls per question, using only Shopify’s native storefront tools and public pages. No StoreKnows tools were present. One model; not ChatGPT, not Gemini, not Claude.
Judge: a separate model (OpenAI’s GPT-5.6 Sol) graded each answer against its reference as fully right, partly right or wrong, and where it could, classified the gap as attribute-shaped (a fact about one product) or relationship-shaped (a link between two).
Identity and cost: every request identified as StoreKnowsBot, one store at a time, public pages only. Total model spend: $9.63. Run on 4 September 2026.

Results

Question type	Questions	Fully right	Partly	Wrong
Fit: does this option exist, is it available	68	55 (81%)	9	4
Compatibility: does A work with B	66	42 (64%)	21	3
Policy: shipping and returns	39	14 (36%)	15	10
Spec filter: in stock, under a price, with a property	60	17 (28%)	30	13
Open-ended: help me choose	39	10 (26%)	26	3
Comparison: what is different between A and B	68	16 (24%)	43	9
All	340	154 (45%)	144 (42%)	42 (12%)

Three things stand out.

Finding the product is solved. Fit questions, “which colour is actually available”, “does the 5-roll bundle exist”, were right four times in five. The search and browse tools work.

Comparing and filtering is not. “What is actually different between the 4-person tent and its vestibule?” was fully right one time in four; 43 of 68 comparisons came back partly right, typically listing the two products without the fact that separates them. Filtered searches (“in-stock black-and-white 35 mm film under $10”) were fully right 28% of the time and wrong 22%.

The misses are single facts. Of the 143 gaps the judge could classify, 119 (83%) were attribute-shaped: a spec, a variant price, a stock state that the tools never returned. Only 24 were relationship-shaped, the “does A fit B” kind that fitment apps exist to solve. The comparison misses alone account for 48 attribute gaps: the assistant had both products and lacked the one number that told them apart.

The six control stores, small text-rich catalogs, scored 25 of 60 fully right (42%), with 22 attribute gaps and one relationship gap. Even where the catalog is simple, the missing piece is a fact about one product.

Why a fact goes missing

Three mechanisms, each seen directly in the tool responses:

One variant stands in for the product. The get_product storefront tool returns a price range and a single variant unless the agent asks for a specific option. On a 1,055-product accessories store we checked separately on 2 September, six of nine partial answers were this: the agent quoted the product’s $69.99 minimum when the chosen variant costs $74.99.
Specs live where no tool looks. Metafields were invisible from outside on 41 of 41 storefronts we crawled: not in /products.json, not in the WebMCP tools, not in the UCP catalog endpoint. If the weight, the thread pitch or the compatible model is a metafield rendered by the theme, an agent reading through the tools cannot see it.
Filters live in tags and options the search tool does not return. Free-text search returns ten summary cards; “MagSafe, kickstand, navy, under $30” is a set intersection the agent has to reconstruct by opening products one at a time, and it runs out of tool calls.

What this is not

It is not evidence that Shopify’s tools are weak. On a 150-product coffee store, run two days earlier with the same setup, Anthropic’s Claude Opus 5 answered 10 of 10 and OpenAI’s GPT-5.6 9 of 10 using native tools alone. The tools read what the catalog holds; specialist catalogs hold their deciding facts somewhere the tools do not reach.

It is not a prediction of what ChatGPT or Gemini will say to a real shopper. Simulated checks call developer APIs and are graded against a reference we built from the store’s own public data; consumer assistants add their own retrieval and caching. One model, one run per store, ten questions each: treat the percentages as a shape, not a benchmark.

What to do with it

If your catalog looks like these (options with dozens of values, specs in metafields, size charts, compatibility tables), the fix is not rewriting descriptions. It is making the one fact per product readable where agents look: a correct first variant, a complete schema block, and, where the facts sit in metafields, a read-only tool that answers “which variant, what price, in stock or not, does it fit”. StoreKnows runs this check on your own store for free, shows you every answer with the product data behind it, and lets you try a fix with your own question before publishing anything.

Results come from a simulated check run by StoreKnows on 4 September 2026 against 34 public storefronts (and on 2 September against two further stores). Questions were answered by OpenAI’s GPT-5.4 mini, GPT-5.5 and GPT-5.6 and Anthropic’s Claude Opus 5, called through their developer APIs, not the consumer apps, and graded by a separate judge model. StoreKnows is independently developed and not affiliated with, endorsed by or sponsored by OpenAI, Google, Anthropic or Shopify.