Most AI visibility reports are built on one run per prompt. That is a coin flip reported as a measurement. When Thinking Machines Lab sent 1,000 identical requests to Qwen3-235B at temperature 0, it got back 80 distinct completions, the first divergence appearing at token 103 (Thinking Machines Lab, 10 September 2025). Consumer assistants add personalization, locale and silent model updates on top. This guide is for growth and SEO leads who have been asked whether the brand shows up in ChatGPT, Gemini or Google's AI Mode and need an answer that survives being challenged. By the end you will have a prompt panel, a run count justified by the precision you need, a scoring rubric, and a rule for deciding when a change between waves is real.
Measure two binary events, not a rank
There is no rank in an AI answer. There is a paragraph that either names you or does not, and a source list that either contains your domain or does not. The unit of measurement is one observation: one prompt, run once, on one assistant, in one language and locale, on one date. Each yields two binary outcomes.
- Mention: the brand name appears in the answer text.
- Citation: a URL on your domain appears in the sources or inline links.
Everything else derives from these. Share of voice is your mentions over all brand mentions in the same answer set. Citation efficiency is citations over mentions: it tells you whether you are being described from model memory or actually read.
Binary outcomes turn this into proportion estimation, which has well documented interval methods (Brown, Cai and DasGupta, Statistical Science, 2001). A composite "AI visibility score" is comparable to nothing, including its own value last month: we could find no definition shared across vendors, so treat cross-tool comparisons as unusable.
Build the prompt panel, then lock it
A panel is a fixed list of prompts you re-run every wave. If the list changes between waves, nothing you measure is a change in visibility.
- Source prompts from real demand. Pull question-shaped queries from Search Console, on-site search logs and sales transcripts. Prompts invented in a workshop measure your team's imagination.
- Stratify by intent. Four strata: category-level ("best X for Y"), comparison ("A vs B"), problem-first ("how do I fix Z") and brand-specific ("is Brand any good"). Fix the count per stratum so a shift in mix cannot masquerade as a shift in visibility.
- Size it at 40 to 80 prompts. Below 40 the panel interval is too wide to act on. Above 80 the run cost grows faster than the precision.
- Treat each language as its own panel. A Turkish prompt and its English translation return different brands, different citations, often a different category frame. Google added Turkish when AI Mode expanded to 53 new languages on 19 February 2026 (language list). Report only English numbers for a Turkish market and you are describing a market you do not serve.
- Freeze the panel for a quarter and version it. Add prompts only at a quarter boundary, and re-baseline when you do.
Decide how many runs per prompt
Each run is a draw, not a lookup, so one run per prompt gives a value with no precision. The table gives the 95% Wilson interval half-width for a per-prompt mention rate observed at 50%, the worst case and so the safe planning assumption.
| Runs per prompt | Margin of error | What you can honestly say |
|---|---|---|
| 1 | Undefined | Nothing. This is an anecdote. |
| 5 | ±33 pts | Triage: often, sometimes, never |
| 10 | ±26 pts | Strong prompt vs dead prompt |
| 20 | ±20 pts | Rank prompts into three tiers |
| 30 | ±17 pts | Detect a large per-prompt change |
| 100 | ±10 pts | Report a per-prompt rate as a number |
Per-prompt precision is expensive and rarely needed. The number the business asks about is the panel-level rate: total mentions over total observations. A 60-prompt panel at 5 runs gives 300 observations per assistant per wave, a naive half-width near ±5.7 points at 50%.
That is optimistic, because runs of the same prompt are correlated. Apply a cluster design effect: multiply the variance by 1 + (runs per prompt − 1) × ICC, the intra-prompt correlation, estimated from your own first wave rather than assumed. If it lands near 0.5 at 5 runs, the effective sample is closer to 100 observations and the honest half-width nearer ±10 points. Plan for that before promising a dashboard.
Our default: 60 prompts, 5 runs each, per assistant, per language, in four-week waves. Report panel and stratum level with intervals, per-prompt results as tiers only.
Control the run conditions
Nondeterminism is the floor. Everything below is variance you add yourself.
- Turn off personalization. In ChatGPT use Temporary Chat, which does not use or create memories. Memory otherwise draws on custom instructions, past chats and files (Memory FAQ), so a logged-in analyst measures their own history.
- Fix the locale and interface language and record both.
- Record model version and timestamp on every observation, or a model update between waves will look like your campaign working.
- Use one collection method per assistant all quarter. API and consumer app have different retrieval; mixing them inside one series makes it meaningless.
- Randomize prompt order and spread runs across the day, so server load, which drives the batch-size variation behind nondeterminism, is not confounded with the prompt list.
Score every observation with a fixed rubric
Two analysts must produce the same score from the same answer. Use a closed scale and write the tie-breaks down.
| Score | Condition | Also record |
|---|---|---|
| 0 | Brand absent from answer and sources | Which brands did appear |
| 1 | Brand named, no link to your domain | Sentiment: positive, neutral, wrong |
| 2 | Brand named and your domain cited | Which URL was cited |
| 3 | Brand named first or recommended, and cited | Cited URL and competitor set |
Score a factually wrong mention as 1 and flag it. A confident, incorrect description of your pricing or coverage is a visibility win and a commercial problem at once, and it is the finding that most often pays for the exercise. Calibrate by having two people score the same 30 answers in wave one; more than three disagreements means the rubric is too loose.
Decide whether a change is real
Run this in order and stop at the first failure.
- Do the wave-over-wave 95% intervals overlap? If yes, report no detected change, not the point difference.
- Did the model version change? If yes, the comparison is confounded: re-run the earlier wave against the new version first.
- Did the change hold for two consecutive waves? One wave is a signal, two is a finding.
- Is it concentrated in one stratum? A move in brand-specific prompts with nothing at category level is usually brand demand, not AI visibility work.
- Did citation efficiency move with mention rate? Mentions rising while citations stay flat means the model is describing you from memory: worth less, and it decays.
Connect the panel to something commercial
Visibility is a leading indicator, not a result. Segment assistant referrals as their own analytics channel and watch sessions and their conversion rate against organic search. Expect low volume, a higher rate per session, and absolute numbers that disappoint anyone shown a rising visibility chart first. Say so in advance.
Platform weighting matters. Similarweb's panel data for June 2025 to May 2026 puts ChatGPT near 53% of generative AI platform share, down from roughly 76% a year earlier, with Gemini at 27 to 28% and Claude near 9% (Similarweb). Panel estimates, not a census: use them to choose which assistants to include, not to weight a headline number.
[INTERNAL DATA NEEDED: median mention rate and citation efficiency from Switas panels across e-commerce, travel and health clients, plus the intra-prompt correlation seen in our own waves.]
Where this method breaks down
- Answers you cannot collect at scale. Google's AI Mode has no public measurement API and automated collection runs into terms of service. Treat it as a smaller manual sub-panel, labelled as such.
- It does not measure classic SERP loss. In Türkiye, where AI Overviews rolled out from 18 February 2026, one study of 1,035 live queries across 17 sectors found AI Overviews on 59% of them and clicks on informational queries down about a third by June 2026 while average position barely moved (Stradiji). A panel would not catch that. Search Console would.
- Short baselines in late-rollout markets. Report levels and intervals, not trend lines, until you have four waves.
- Nondeterminism is not fixable from outside. Batch-invariant kernels remove it only for whoever runs the inference. As a consumer of hosted assistants you can only average over it.
- Panel drift. A frozen panel is right for comparability and wrong about the market after two or three quarters.
Frequently asked questions
How many prompts do I need for a first panel?
Forty is the practical floor and 60 is our default. Below 40 the panel interval is wide enough that most quarter-over-quarter changes are indistinguishable from noise. Stratify by intent and keep the count per stratum fixed.
Does setting temperature to zero make results reproducible?
No. Temperature zero removes sampling randomness but not the variation from batch size changing with server load, which is why 1,000 identical temperature-zero requests produced 80 different completions in the Thinking Machines experiment. Average over runs instead of chasing determinism.
Can I just use a vendor AI visibility score?
Use vendor tools for collection if they save time, but rebuild the metric from raw observations. Composite scores are not defined consistently across tools, and several report a single run per prompt. Ask any vendor how many runs sit behind one data point.
How often should I run a wave?
Every four weeks for most brands. Weekly waves mostly re-measure noise and model updates. After a large content or structured-data change, add a wave four weeks later, not the week after.
Should Turkish and English prompts sit in one panel?
No. Run separate panels with separate baselines. Answer sets, cited domains and the competitive frame differ by language, so mixing them produces an average that describes neither market.
What counts as a citation if the assistant links to a marketplace listing instead of my site?
Score it as a mention without citation and record the cited domain separately. A pattern of marketplace citations usually points at a content or structured-data gap on your own pages rather than at brand awareness.
We design and operate AI visibility panels as part of our growth and technical SEO work: panel design, wave collection across assistants and languages, scoring calibration, and the analytics segmentation that ties it to revenue. Talk to us about a first baseline wave.
Sources
- Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 10 September 2025
- Brown, Cai and DasGupta, Interval Estimation for a Binomial Proportion, Statistical Science 16(2), 2001
- OpenAI Help Center, Temporary Chat FAQ
- OpenAI Help Center, Memory FAQ
- Search Engine Roundtable, Google Expands AI Mode To 53 New Languages, 19 February 2026
- Google Search Help, AI Mode supported languages
- Zeo, Google AI Mode and AI Overview Live in Türkiye
- Stradiji, What AI Overviews Did to Search in Türkiye
- Similarweb, AI search statistics, June 2025 to May 2026
