Which product questions should you test with an AI shopper?

A useful AI-shopping test starts with a decision the customer needs to make, not a generic prompt such as “Tell me about this store.” For a specialist catalog, one exact fit, specification or variant answer can be more informative than dozens of vague recommendations. This is a method for choosing questions, not a claim that any category converts better.

If you already have a disputed answer, use the answer investigation guide. Use this page before testing, to decide which questions have a reliable answer key.

Choose four questions with different jobs

Start with a product your team knows well. The following prompts are illustrative, not observed merchant incidents.

Replace these with questions from presales support, a product-choice discussion, or an active AI-shopping project. Record why each matters to your operator. If nobody can validate the answer, keep it as a research question, not a scored failure.

Make the answer key judgeable first

For each chosen question, establish the source owner, exact product or variant, units and conditions. Write only what the source supports: yes, no, or not established. A list of compatible models is not automatically an exhaustive list of incompatible models; today’s price cannot settle last month’s quote.

An answer key should say what a useful response must include and when it must ask for clarification. For the exact-fit example above, that means naming Model X revision 2 and keeping any installation condition with the answer. If nobody can validate the key, keep the question as research rather than a scored failure.

Use the product-answer log to record the answers and session conditions once your questions are ready.

Keep the question set separate from the app baseline

Choose the source and route that can answer your question: Catalog, store FAQs or visiting-agent tools. A question about discovery and a question about product fit need different evidence.

Your merchant-chosen question set is a test plan, not a StoreKnows entitlement. StoreKnows’s free baseline builds its own questions from fixed catalog templates. Where a supported fix is proposed, Try a shopper question lets you evaluate your own question in its preview. How the check works explains that scope.

Finish with a small set you can judge

Four questions are an illustrative starting set, not a required sample size or an accuracy score. Add a question when it represents another meaningful decision or source uncertainty. Keep the expected clarification and unknown cases alongside the easy answers.

Next step: put your chosen questions and answer keys in the product-answer log. Use the diagnosis guide after a disputed result and the comparison method after a change.

StoreKnows is independently developed and not affiliated with, endorsed by or sponsored by Shopify or the AI platforms discussed.