How do AI visibility tools work, and can you trust the numbers?

Google gives you Search Console. ChatGPT, Perplexity and Google's AI Mode give you nothing: no report showing when your brand comes up in an answer. AI visibility tools fill that gap, and dozens of them now sell a score that says how often AI recommends you. Before you act on that score, it helps to know where it comes from.
What these tools actually do
Every AI visibility tool works roughly the same way:
- Pick prompts. You (or the tool) write a list of questions a buyer might ask, such as "best accounting software for a small bakery". Nobody outside the AI companies has real data on what people type into chatbots, so this list is an educated guess, as Laura de Castro points out.
- Ask the AI. The tool sends each prompt to one or more AI products on a schedule, daily or weekly, and saves the full answer.
- Read the answers. It looks for your brand and your competitors, notes the order they were mentioned in, which pages were cited as sources, and whether the tone was positive.
- Roll it up. Those results become metrics like share of voice, average position and a visibility score.
Keyword.com's help page is a clear example: its "detection rate" is simply the percentage of runs where your brand appeared, and "position 1" means you were the first brand named in the text. There's no fixed ranked list in an AI answer, so position is mention order, not a search ranking.
The big difference: API or real app
The step that varies most between tools is how they ask the AI. There are two ways.
Through the API. An API is the developer connection AI companies sell so software can send questions and get answers. It's cheap, fast and easy to run at huge volume. But it isn't the same thing a person sees in the ChatGPT app. The API may run a different model version with different instructions, may not search the web unless asked, and doesn't show the source panels or shopping boxes the app does. Some products, including Google's AI Overviews and AI Mode, have no official API at all, as Growth Method's comparison notes.
Through the real interface. The tool loads the consumer app, types the prompt like a user would and records what appears on screen, sources included. This is closer to reality, but slower, more expensive and harder to keep running. Many tools buy this data from scraping providers instead of collecting it themselves, and most don't say so.
How much does the choice matter? Surfer ran 1,000 prompts through both routes across five AI products. On ChatGPT, the API answers named about 13.8 brands on average against 7.9 in the app, ran about twice as long, and cited about 3 sources against 12 in the app. Across products, only about 15% to 24% of the brands named overlapped between the two methods. Surfer sells a tool built on interface data, so it has a stake in the result, and it ran each prompt only once. Still, the gap is large enough that an API-based report may count brands real users never see.
Neither method is wrong on its own. Metehan Yesilyurt argues that APIs are fine for broad research, as long as the vendor tells you which method it used for each AI product. The problem is when a tool mixes them without saying so.
Why your score jumps around
Ask an AI the same question twice and you can get two different answers. Even with its randomness setting turned all the way down, researchers at Thinking Machines Lab got 80 different completions from 1,000 runs of one prompt. Consumer apps add live web search, location and silent model updates on top of that. Your brand can show up at 9am and disappear by 10am with nothing changed on your site.
That makes a visibility score a sample, not a measurement, and small samples are noisy. The HOTH works through the math: if a tool says you appear in 20% of answers, the real figure could be about 11 points either way with 50 prompts, and about 8 points either way with 100. A jump from 20% to 25% on a 50-prompt list can easily be noise.
The prompt list itself can skew things too. Prompts that include your brand name inflate the score, and quietly adding or removing prompts can look like a gain or a drop.
How to read the numbers sensibly
- Watch trends, not single runs. Look at weeks or months of history, and ignore day-to-day swings.
- Keep the prompt list fixed. Use a large enough set (100 or more is a common floor) and report branded and unbranded prompts separately.
- Run each prompt more than once per period, so the tool reports a rate, not a coin flip.
- Look at each AI product on its own. ChatGPT, Perplexity and Google's AI features pull from very different sources, so a single blended number hides a lot.
- Track competitors too. If everyone moves at once, the AI changed, not you.
- Check against real traffic. Visits from chatgpt.com or perplexity.ai in your analytics are a count, not an estimate. When they disagree with the score, trust the traffic.
Questions to ask a vendor
Before you pay, ask in writing:
- Do you collect through the API, the real interface, or both, for each AI product?
- How many times do you run each prompt, and from which country and language?
- Can I see the raw answers behind every data point, and export them?
- Do you flag it in my history when your method changes?
A vendor that can't answer these, or promises guaranteed placement in AI answers, is a red flag.
Tracking the output vs. fixing the input
Visibility tools measure the result: what AI says about you. They don't tell you whether AI can read your site in the first place. If AI crawlers are blocked, or your content only appears after JavaScript runs, no amount of tracking will move the number. That's the part Jrooky checks, using the 13 points that make a site AI-ready.
Run a free scan to see whether AI tools can reach and read your site before you start measuring how often they mention it.