How much of your “AI crawler” traffic is real?
Anyone can send User-Agent: GPTBot from a laptop, so every “AI traffic” chart built on user-agent strings is counting claims, not crawlers. Drop in an access log and this tool checks each one against the operator by forward-confirmed reverse DNS — the whole file at once, in your own browser.
Drop an access log here
nginx, Apache, Vercel or Cloudflare · .log, .txt, .json or .gz · up to 500MB
Your log is read inside this browser tab and never uploaded. The only thing sent to us is the list of IP addresses that claim to be crawlers, because reverse DNS needs a resolver and browsers have none. No paths, no timestamps, no visitor addresses. Open your network tab and check.
Paste a log instead
Useful when the log is on a server you are already SSH’d into: grep -Ei 'gptbot|claudebot|perplexitybot|oai-searchbot' access.log | tail -5000
How the check works
There is no single method, because the operators do not agree on one. The report tells you which was used for each crawler, and links you to the same source, so you can redo any line of it yourself.
- 1
Match the IP against the operator’s published list
OpenAI, Anthropic, Perplexity and Bing each publish the prefixes their crawlers use. Inside the list is the operator vouching for that address; outside it is the operator disowning it. For OpenAI this is the only method that works at all — their crawlers carry no reverse DNS.
- 2
Or reverse-lookup the IP and confirm the hostname
Google, Bing and Apple support the older convention: the address resolves to a hostname under googlebot.com, search.msn.com or applebot.apple.com. A hostname anywhere else is a red flag, not a pass.
- 3
Forward-lookup it back to the same IP
The step most checks skip. Reverse records are controlled by whoever owns the IP block, so only the round trip proves the claim.
If any step is inconclusive the hit is reported as unresolved, never as fake. An inconclusive check must not be presented as a caught liar — that is how a verification tool ends up doing more damage than the spoofers.
Why we don’t want your log file
An access log contains the IP address of every human who visited your site. Under GDPR that is personal data, and asking you to hand it to a third party so they can count crawlers is a liability for both of us — an unnecessary one, because the counting does not need a server.
So the parsing happens in a Web Worker in your browser. The only request this page makes carries the unique IP addresses that claim to be crawlers, and which operator each claims to be — machines whose entire job is to identify themselves publicly. Typically a few dozen addresses. Nothing is written to a database, and there is no account to create.
Don’t take our word for it: open DevTools, watch the Network tab while it runs, and look at what is actually in the one request.
Questions
How can I tell if GPTBot traffic is real?
Check the IP, not the name — and for OpenAI specifically, check it against their published list. GPTBot, OAI-SearchBot and ChatGPT-User run on Azure address space with no reverse-DNS record under openai.com, so a reverse lookup of a genuine OpenAI crawler returns nothing at all. OpenAI publishes a separate prefix list per crawler (openai.com/gptbot.json, /searchbot.json, /chatgpt-user.json); an address inside the right one is the operator vouching for it, and an address outside it is the operator saying it is not theirs.
How much AI crawler traffic is actually fake?
Nobody knows the industry figure, and anyone quoting one is guessing. The most-cited measurement is Duane Forrester’s: on a brand-new domain over 14 days, 27 of 33 requests claiming to be AI assistants failed verification (81.8%), and of 799 requests claiming to be Googlebot only 107 verified. He is explicit that this is one small sample and a baseline rather than a benchmark — which is exactly why the number that matters is the one from your own log.
Why does reverse DNS not work for OpenAI or Anthropic?
Because they do not publish reverse-DNS records for their crawlers. Reverse DNS is the older convention and Google, Bing and Apple still support it — Googlebot resolves under googlebot.com, Applebot under applebot.apple.com. The AI-native operators went the other way and publish prefix lists instead: OpenAI per crawler, Anthropic at claude.com/crawling/bots.json, Perplexity at perplexity.ai/perplexitybot.json. A verifier that only does reverse DNS therefore reports "unknown" for every OpenAI hit, which is worse than useless — it looks like an answer. This tool uses whichever method the operator actually supports, and tells you which one it used.
Can I just block fake crawlers by user-agent?
No. The user-agent string is the thing being faked, so a rule matching on it stops only the honest crawlers. Blocking works at the IP layer: verify by reverse DNS, or allow only the IP ranges the operator publishes and rate-limit the rest.
Does this tool upload my access log?
No. The file is read and parsed inside your browser tab by a Web Worker, and never leaves your machine. The only thing sent to our server is the list of IP addresses that claim to be crawlers, because reverse DNS needs a resolver and browsers do not have one. No paths, no timestamps, no referrers, no visitor addresses — and nothing is stored. You can confirm all of this in your browser’s network tab.
Why are so many hits marked unresolved?
Unresolved means the check was inconclusive, not that the hit was fake. It happens when a DNS lookup times out, when a published list could not be fetched, when the IP was outside the run’s lookup budget, when the log line carried no IP at all, or when the operator publishes nothing to check against — Common Crawl and ByteDance publish neither a prefix list nor a reverse-DNS convention, so hits claiming them cannot be verified by anyone. Unresolved is counted as its own column and never folded into either side.
Does blocking GPTBot stop ChatGPT citing me?
No. GPTBot collects training data only. The crawler behind ChatGPT citations is OAI-SearchBot, and live user questions are fetched by ChatGPT-User. They are independently controllable, which is why this report shows each crawler’s role rather than lumping them together as "AI traffic".
What log formats does it read?
Combined Log Format — the nginx and Apache default — including the vhost variant, and JSON lines as produced by Vercel, Cloudflare and structured nginx. Mixed files are fine. Gzipped logs (.gz) are decompressed in the browser, so you can drop a rotated file straight in. Lines it cannot read are counted and reported rather than silently dropped.
Can they reach you in the first place?
This tool checks the crawlers that already arrived. The visibility check reads your robots.txt crawler by crawler and tells you which ones are being turned away.
Run the visibility checkOne log is a snapshot
It cannot tell you which of your published pages a retrieval crawler has never fetched, or when one stopped coming. That needs a record kept over time.
See what ActiveGeo tracks