Same store, same questions, better answers: what changed when product data became readable

Short answer: on a 144-product development store where the deciding specs live in metafields, making those specs readable through five read-only tools took Google’s Gemini 3.8 Flash from 7 of 14 fully right answers to 13 of 14, and OpenAI’s GPT-5.4 mini from 4 of 14 to 8 of 14, on the same 14 questions. One store, one run each, both models called through their APIs. Here is the setup, the numbers, and what they do and do not show.

Why run the same questions twice?

Our batch of 340 questions across 34 specialist storefronts showed where simulated AI shoppers miss: almost never in finding the product, almost always in reading one fact about it. A spec, a variant price, a stock state, a compatibility note. On Shopify stores that fact usually sits in a metafield, and on 5 September 2026 metafields were invisible from outside on 41 of 41 storefronts we crawled.

A diagnosis is only useful if the fix can be measured. So the question for this article is narrow: if the same store serves the same metafield facts through tools an agent can call, and the same questions are asked again, what changes? Not “does it help in general”. Just: same store, same questions, what moved.

The setup

• Store: our own development store, seeded with 144 bicycle products whose deciding specs live only in metafields. This is a synthetic catalog built so that the facts are where specialist stores keep them, which is a strength for isolating the mechanism and a limit for generalising, and we come back to it below.
• Questions: 14 shopper questions generated from the store’s own catalog, each with a recorded reference answer. The same 14, word for word, in both runs.
• Shoppers: Google’s Gemini 3.8 Flash via the API and OpenAI’s GPT-5.4 mini via the API, each answering through the storefront’s tools in a browser session against the live store. No consumer apps.
• Judge: OpenAI’s GPT-5.6 Sol, a separate model, grading each answer against its reference as fully right, partly right or wrong. No model is judged by itself.
• Before: only the ten native storefront tools every Liquid storefront has carried since 21 August 2026 (search, browse, product details, cart and checkout, orders, policies).
• After: the same ten tools plus five read-only StoreKnows tools served by the theme app embed on the live storefront: answer “does this fit?”, recommend a size, filter by spec, compare products, and check a captured variant’s price and stock. The run loaded nothing itself. The storefront served the tools through the theme, the way a real agent would find them; the store’s verification listed fifteen tools, ten Shopify’s and five ours.
• Runs: one before, one after, on 5 September 2026.

The results

Assistant (via the API)	Before: fully right	After: fully right	Wrong before → after
Google’s Gemini 3.8 Flash	7 / 14	13 / 14	2 → 0
OpenAI’s GPT-5.4 mini	4 / 14	8 / 14	2 → 2

The remainder in each cell is partly right: Gemini went from five partial answers to one; the mini model from eight to four.

Two things are worth saying plainly.

The gain is large and uneven. Gemini picked up six questions and lost its two wrong answers. The mini model picked up four and kept two wrong. Same store, same tools, same questions; the difference is how each model uses a tool it is offered. We saw the same spread in a separate check on a 1,055-product accessories store on 2 September, where one model adopted added tools immediately and another kept searching and reading one product at a time (that check is written up here). Tool adoption is model-dependent, which is why the app reports per assistant and not as one score.

Wrong went to zero for one model, and stayed at two for the other. The run log records the counts: two wrong answers from GPT-5.4 mini before, two after. It does not record which questions those were, or whether they were the same two, and we are not going to guess. What we can say is that a readable spec removes the reason for the most common miss we see, an answer that says the information is not available when the store holds it. It does not stop a model from misreading a fact it has been given, or from not asking for it.

What changed, and what did not

The run log records totals per model, not a per-question breakdown, so we will not claim which question types moved. What the setup does tell you is what each tool could and could not do.

The five tools each answer one shape of question from the metafield data: “which pads fit these calipers” from a compatibility field, “which of these weighs 24 g or less” from a weight field, “my head is 56 cm, which size” from a size chart, “what is different between these two” from the fields both products share, “is this variant in stock and what does it cost” from a captured copy of the variant. If the fact is in the field and the model calls the tool, the answer has the fact in it. If the model does not call the tool, nothing has changed for that question: it is still reading the same product card it read before.

Nothing else moved. The catalog was not rewritten. No description was edited, no tag added, no option renamed. The tools are read-only and serve a saved copy of the approved fields; they cannot change a price, a stock level or a product. The difference between the two runs is entirely that the facts in the metafields became callable.

One question, before and after

The clearest way to see the mechanism is a single answer pair. This one comes from the demo catalog in our listing video, not from the 144-product run above: a fictional store of eight published bicycle products on a separate development store, recorded on 9 September 2026 in a simulated check, answered by Anthropic’s Claude Sonnet 5 via OpenRouter. The pads’ product record carries a compatibility metafield, “Compatible with: Cedar C2 calipers”, and the catalog has no separate calipers product.

The question: Is Cedar C2 calipers listed as compatible with Cedar C2 Brake Pads? The recorded reference: “Yes. Cedar C2 calipers is listed as compatible with Cedar C2 Brake Pads.”

Before, with native tools only, the assistant found the product and reported, in part: “So I could not determine whether Cedar C2 Calipers are listed as compatible with Cedar C2 Brake Pads — because there is no ‘Cedar C2 Calipers’ product in the catalog, and the brake pad listing contains no compatibility/fitment information to check against.” Graded not answered correctly, with the judge’s note: “The captured product record declares Cedar C2 Brake Pads compatible with Cedar C2 calipers.”

After, with the “does this fit?” tool published: “Yes — the Cedar C2 Brake Pads (handle: cedar-c2-brake-pads) are listed in the catalog with ‘Compatible with: Cedar C2 calipers.’ So the merchant’s own data confirms compatibility between the Cedar C2 Brake Pads and Cedar C2 calipers.” Graded answered correctly. The assistant also noted that it “could not find a separate product listing for ‘Cedar C2 Calipers’” and that the compatibility “is only documented from the brake pads’ side”, which is exactly right: the link is one-directional in the data, and the tool said so rather than inventing the other direction.

The first answer is not a hallucination. It is an accurate description of what the native tools returned: a product card with no compatibility field on it. The store knew; the tools the assistant had did not carry it. That is the whole before-and-after in one exchange. Two caveats travel with it: it is listed compatibility, not a physical-fit certification, and it is one graded answer on a demo catalog, not a result we count.

The limits, stated once

• One prepared store. The 144-product catalog was seeded so that its deciding facts live in metafields. That is how many specialist stores are built, and it is also the case the tools are designed for. A store whose specs already sit in titles and descriptions would show a smaller gap and a smaller gain; on a text-rich 150-product coffee store on 2 September, native tools alone answered 10 of 10 for one model.
• One run each. Runs vary. A second before-run would not score exactly 7 and 4, and a second after-run would not score exactly 13 and 8. The direction is what the numbers support; the exact figures are one draw.
• Fourteen questions. Enough to see a mechanism, not enough to quote a percentage with confidence. We give the counts, not the percentages, for that reason.
• APIs, not consumer apps. Both models were called through developer APIs in a browser session against the storefront’s tools. What ChatGPT, Gemini or any consumer assistant says to a real shopper adds retrieval, caching and product knowledge of its own on top, and we make no claim about that.
• No Claude before-and-after on this store yet. Anthropic’s Claude Sonnet 5 via OpenRouter has been part of every check since 8 September 2026, so new checks report three assistants. The 5 September run predates it, and we have not yet published a Claude before-and-after on the 144-product store. The single Cedar pair above is one question on a demo catalog, not a scored run.
• Saved data, not live inventory. The tools answer from a copy of the approved fields, frozen at publication. Price and stock, where a tool returns them, are captured values; the storefront and checkout remain the source of truth, and the tools say so.

How the same loop runs on a real store

The measurement above is the loop StoreKnows runs on any Liquid storefront, with you deciding at each step.

1. Check. StoreKnows reads your catalog (products, variants, metafields, metaobjects, size charts) and builds up to 14 shopper questions with a recorded reference each. Simulated shoppers on Gemini, GPT and Claude models answer them through your storefront’s native tools. Free, one per store, in a simulated check.
2. Read the answers. Each answer is shown with the assistant that gave it, its grade, the reference, and the product data it should have found. Where a miss is a fact the store holds, the app says which field.
3. Try the fix with your own question. Where your data supports it, StoreKnows proposes a read-only tool and lets you ask it anything in a preview that uses saved product data. Ask a question you already know the answer to. See the answer and the rows it came from. Nothing is published by trying.
4. Publish what you approve. Enabling is a one-time charge per store, with no subscription. You see exactly which products, fields and charts become public, inspect the values, and approve. Your previous published version stays in place until you approve a replacement, and a catalog refresh or a change of selection asks for approval again.
5. Switch on the theme app embed. One click in the theme editor adds the tools beside Shopify’s own. Switching it off removes them.
6. Storefront verification. StoreKnows visits your storefront, confirms the published tools are served and match the approved version, and keeps checking daily. This proves the tools are available; it does not test answers.
7. The same questions again. A successful verification starts the comparison: the same 14 questions, the same assistants, side by side with the baseline, so you see per question and per assistant what changed, what stayed the same, and anything that got worse. Enable includes that first after-check and two more you can start yourself, for instance after refreshing your catalog.

That is the whole product: a check you can read, a fix you can try with a question you know, a publication you approve field by field, and the same questions asked again so the claim is yours to verify rather than ours to make.

If your catalog keeps its deciding facts in metafields, run the free check and read what the assistants say today. Then try one fix with a question you already know the answer to, and decide for yourself whether the after is worth publishing.

Results in this article come from a simulated check run by StoreKnows on 5 September 2026 against our own 144-product development store, and one answer pair from a fictional eight-product demo catalog on a separate development store on 9 September 2026. Questions were answered by Google’s Gemini 3.8 Flash and OpenAI’s GPT-5.4 mini, called through their developer APIs, and by Anthropic’s Claude Sonnet 5 via OpenRouter, not the consumer apps, and graded by a separate judge model. StoreKnows is independently developed and not affiliated with, endorsed by or sponsored by OpenAI, Google, Anthropic or Shopify.