How to See Which AI Bots Visit Your Site (and Check They're Real)

Your access log already records each AI crawler request that reaches your server, and declared crawlers name themselves in the user-agent field. A few lines of awk show how often GPTBot, ClaudeBot and PerplexityBot come and what they read. The catch is that the user agent is text the client chooses. A script on a rented server can call itself GPTBot, and your log counts it like the real one.

This guide covers both halves: recipes that count AI bot hits by bot, day, path and status, then a check of each hit's address against the ranges OpenAI, Anthropic, Perplexity, Common Crawl, Apple and Google publish. You end with hits you can trust and hits you can't.

Where your host keeps request logs

These details come from each vendor's documentation, read in October 2026.

Vercel drains deliver to an HTTP endpoint you run or a Marketplace integration. Choose NDJSON, save the lines, and one jq command rewrites them as combined-format lines. Each console.log line is its own entry, so busy routes are over-counted, but the set of outside addresses is not.

jq -r 'select(.proxy) | "\(.proxy.clientIp) - - [\(.proxy.timestamp / 1000 | floor | strftime("%d/%b/%Y:%H:%M:%S")) +0000] \"\(.proxy.method) \(.proxy.path) HTTP/1.1\" \(.proxy.statusCode) \(.proxy.responseByteSize // 0) \"-\" \"\(.proxy.userAgent | join(" "))\""' drain.ndjson > access.log

Note: Behind a CDN or proxy, your origin log shows the proxy's address, so every bot looks fake: nearly all hits outside, with the same top address. Cloudflare passes the visitor's address in CF-Connecting-IP; in nginx, the realip module swaps $remote_addr using set_real_ip_from and real_ip_header. Fix this first.

Five awk recipes: hits by bot, day, path, status and robots.txt fetches

An nginx combined line, which Apache's combined format matches, looks like this (an invented line using an address from OpenAI's current list):

20.171.206.5 - - [14/Oct/2026:02:10:00 +0000] "GET /robots.txt HTTP/1.1" 200 16495 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

Splitting on double quotes with -F'"' puts the request in $2, the status in $3 and the user agent in $6; address and time sit in $1. Each recipe lowercases the user agent so capitalization never hides a hit. nginx -T | grep access_log shows your log path; Apache's is the CustomLog line. If an Apache line starts with host:port (vhost_combined), recipes 2 and 5 need f[5] instead of f[4] for the date.

BOTS='gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user|ccbot|applebot|googlebot'

# 1. Hits per bot
awk -F'"' -v re="$BOTS" 'match(tolower($6), re) { print substr(tolower($6), RSTART, RLENGTH) }' access.log | sort | uniq -c | sort -rn

# 2. Hits per day and bot
awk -F'"' -v re="$BOTS" 'match(tolower($6), re) { split($1, f, " "); print substr(f[4], 2, 11), substr(tolower($6), RSTART, RLENGTH) }' access.log | sort | uniq -c

# 3. Paths one bot requested, query strings removed
awk -F'"' 'tolower($6) ~ /gptbot/ { split($2, r, " "); sub(/\?.*/, "", r[2]); print r[2] }' access.log | sort | uniq -c | sort -rn | head -20

# 4. Status codes one bot received
awk -F'"' 'tolower($6) ~ /gptbot/ { split($3, s, " "); print s[1] }' access.log | sort | uniq -c

# 5. robots.txt vs content fetches per day, minus outside addresses (one -e each)
grep -vFw -e 203.0.113.7 access.log | awk -F'"' 'tolower($6) ~ /gptbot/ { split($1, f, " "); split($2, r, " "); print substr(f[4], 2, 11), (r[2] ~ /^\/robots\.txt/ ? "robots.txt" : "content") }' | sort | uniq -c

Vendors add tokens, so also list whatever else looks like a bot. This over-matches on purpose, and shows tokens such as Amazonbot that the script below has no range file for:

awk -F'"' 'tolower($6) ~ /bot|crawl|spider|-user|agent|fetch|externalhit|preview/ { print $6 }' access.log | sort | uniq -c | sort -rn | head -20

For rotated logs, pipe zcat -f access.log* into the same awk. Totals count requests, not pages read. No hits can mean a short log window, a CDN answering before your origin, or no visit yet, so check retention above before concluding there is no AI traffic.

Why a matching user agent proves nothing

Common Crawl's CCBot page says it is aware of crawlers falsely identifying themselves as CCBot; Google and Apple tell you to verify by address and DNS, not by name.

A Duke team studied 40 days of traffic to 36 of its university's websites (ACM Internet Measurement Conference, 2025). It flagged 18 well-known bots when at least 90% of a bot's requests came from one network and some came from others: under five accesses for most, but 381 of 15,132 for Baiduspider and 33 of 9,103 for Googlebot. The authors say that cannot prove spoofing, though it strongly suggests it, and that their institution's IP allowlisting may have kept the numbers low.

Undeclared crawling is a different problem. In August 2025 Cloudflare reported crawlers presenting as Chrome on macOS from addresses outside Perplexity's published range; Perplexity disputed that, saying its assistants fetch pages only in response to user questions. An address check unmasks a false name but cannot find a crawler that never claims one.

What each vendor lets you verify against

Reverse DNS maps an address to a hostname; forward confirmation resolves that hostname and expects the same address back.

Every file carries a prefixes array of CIDR blocks (ipv4Prefix or ipv6Prefix) and a creationTime. When we fetched them on 11 October 2026, creation times ran from February 2025 (PerplexityBot) to 9 October 2026 (Google). Download on a schedule and keep dated copies, so an old hit is matched against the list that was current then.

That day we also looked up the first address of each IPv4 prefix. Google's gave googlebot.com names for 165 of 170 (the rest were bc.googleusercontent.com) and Common Crawl's gave crawl.commoncrawl.org. None of OpenAI's 293 had a reverse record. Of Anthropic's 38, 29 gave generic bc.googleusercontent.com names, 3 gave Amazon EC2 names, and the six multi-address blocks, including 216.73.216.0/22, gave none. Perplexity's gave generic amazonaws.com names. Treat the files as the main check and DNS as the second.

One trap: Google's page lists googlebot.com, google.com and googleusercontent.com as acceptable domains, and a loose googleusercontent.com test passed some of Anthropic's addresses in our check. Use the vendor's own suffix, as this forward-confirmed lookup does (IPv4 only):

fcrdns() {  # usage: fcrdns IP SUFFIX; prints the hostname on success, nothing on failure
  name=$(host "$1" | awk '/pointer/ {print $NF; exit}')
  case "$name" in *".$2.") host "$name" | grep -q "has address $1\$" && echo "$name";; esac
}
fcrdns 66.249.66.1 googlebot.com    # prints crawl-66-249-66-1.googlebot.com.

A script that checks every hit against the published ranges

Download the files once, then run the script on any combined-format log.

mkdir -p ranges && cd ranges
curl -sSfL --remote-name-all \
  "https://openai.com/{gptbot,searchbot,chatgpt-user,adsbot}.json" \
  "https://www.perplexity.com/{perplexitybot,perplexity-user}.json" \
  https://index.commoncrawl.org/ccbot.json https://search.developer.apple.com/applebot.json
curl -sSfLo claude.json https://claude.com/crawling/bots.json
curl -sSfLo googlebot.json https://developers.google.com/static/crawling/ipranges/common-crawlers.json

Save this as verify_bots.py and run python3 verify_bots.py access.log ranges. Bot-looking lines it cannot parse are counted and reported at the end, so an empty table is not mistaken for no bots.

import ipaddress, json, re, sys
from collections import Counter, defaultdict

FILES = {  # lowercase user-agent token -> range file
    "gptbot": "gptbot.json", "oai-searchbot": "searchbot.json",
    "oai-adsbot": "adsbot.json", "chatgpt-user": "chatgpt-user.json",
    "claude": "claude.json", "perplexitybot": "perplexitybot.json",
    "perplexity-user": "perplexity-user.json", "ccbot": "ccbot.json",
    "applebot": "applebot.json", "googlebot": "googlebot.json",
}
# Combined format, with an optional Apache "host:port " prefix (vhost_combined)
LINE = re.compile(r'^(?:[\w.-]+:\d+ )?(\S+) \S+ \S+ \[[^\]]+\] "[^"]*" \d+ \S+ "[^"]*" "([^"]*)"')

def load(name):
    prefixes = json.load(open(f"{sys.argv[2]}/{name}"))["prefixes"]
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]

nets = {t: load(f) for t, f in FILES.items()}
hits, outside, skipped = Counter(), defaultdict(Counter), 0
for line in open(sys.argv[1], errors="replace"):
    m = LINE.match(line)
    if not m:
        skipped += bool(re.search(r"bot|crawl|spider", line, re.I))
        continue
    token = next((t for t in FILES if t in m.group(2).lower()), None)
    if not token:
        continue
    ip = m.group(1)
    try:
        inside = any(ipaddress.ip_address(ip) in n for n in nets[token])
    except ValueError:
        inside = False
    hits[token] += 1
    if not inside:
        outside[token][ip] += 1

print(f"{'token':16}{'hits':>5}{'inside':>8}{'outside':>9}  top outside addresses")
for token, n in hits.most_common():
    out = sum(outside[token].values())
    top = ", ".join(f"{ip} ({c})" for ip, c in outside[token].most_common(3))
    print(f"{token:16}{n:>5}{n - out:>8}{out:>9}  {top}")
if skipped:
    print(f"\n{skipped} bot-looking lines were not in combined format and were skipped")

On a 58-line log we invented, where we planted fake GPTBot, ClaudeBot and Googlebot traffic from documentation-range addresses, it prints:

token            hits  inside  outside  top outside addresses
gptbot             16       9        7  203.0.113.7 (7)
claude              8       4        4  198.51.100.23 (4)
googlebot           7       5        2  192.0.2.44 (2)
oai-searchbot       4       4        0
chatgpt-user        3       3        0
perplexitybot       2       2        0

Reading the result: which hits are real and what each job means

Sort each outside address

Take 203.0.113.7, the outside GPTBot address above. Its seven hits asked for /wp-login.php, /.env and other paths nothing links to, and OpenAI documents no reverse DNS, so it lands in row 3. Another hint is the network behind an address, which Team Cymru's DNS lookup returns for IPv4:

ip=203.0.113.7
dig +short TXT "$(echo $ip | awk -F. '{print $4"."$3"."$2"."$1}').origin.asn.cymru.com"

The first field is the origin ASN, which identifies the network. In the Duke study, 99.76% of GPTBot traffic came from one Microsoft network and 99.62% of Googlebot's from Google Cloud's, so an outside address in an unrelated hosting network is a stronger tell, though not proof.

What a verified hit means, by job

For hits that pass, the bot's job tells you what they mean. Netlify's definitions help with volume: crawlers read a site in bulk, while AI agents fetch on demand and are "typically not high-volume".

What to do with the findings, starting this week

Verification establishes identity, not behaviour. Policy for each token belongs in robots.txt and at the edge; our guide to allowing or blocking AI crawlers has parser-checked robots.txt recipes and edge rules.

For impostors, enforce the pairing your check proved. Perplexity's docs show a Cloudflare rule that combines a user-agent match with an address condition to allow its bots; inverted, it refuses requests that claim a bot name from outside the vendor's list. To stop real bots, use robots.txt, not range blocks: Anthropic warns that blocking its addresses may stop its crawlers reading your robots.txt.

Give a robots.txt change time before judging it, then run recipe 5. A bot that fetched robots.txt after your edit and still requested disallowed paths (recipe 3 lists them) is ignoring the file. One that never re-fetched may not have seen it: Perplexity says changes can take up to 24 hours to apply, and in the Duke study fewer than 40% of AI assistants and AI search crawlers re-checked robots.txt within 168 hours.

Compliance also differs by kind. In that study's disallow-all test on one site, weighted by accesses, AI assistants scored 1.000, AI data scrapers 0.766, AI search crawlers 0.348 and search engine crawlers 0.189. Spend edge rules on the kinds that keep reading.

1. Export seven days of logs, using the first table.
2. Download the ranges, then run recipe 1, the discovery line and the script.
3. Sort each outside address with the verdict table; if all are outside, fix real-IP handling first.
4. Repeat weekly, keeping each week's range files.

What Is llms.txt? covers the format and what log data says about its readers, and the AI search topic hub lists related guides.

Frequently asked questions

Does an address inside the vendor's list prove the bot respects my rules?

No. It proves who sent the request, not what the sender promises to obey. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

Why does a Googlebot hit sit outside Google's list?

Rule out three causes before assuming a spoofer: your log may show a proxy's address, your copy of the file may be stale (Google's carried a 9 October 2026 creation time), or you may be comparing another Google fetcher against the crawler file. Then run the forward-confirmed lookup; if it still fails, treat the hit as an impostor.

Why do I never see Google-Extended or Applebot-Extended in my logs?

Neither is a crawler. Google says Google-Extended is a standalone robots.txt token with no user-agent string of its own; it controls whether content Google crawls may be used to train future Gemini models. Apple says Applebot-Extended does not crawl webpages and only decides how Applebot's data is used. The hits you do see come from Googlebot and Applebot, so verify those.

Sources and further reading

• Overview of OpenAI Crawlers — OpenAI, accessed October 2026. Tokens, robots.txt behaviour, range files.
• Does Anthropic crawl data from the web, and how can site owners block the crawler? — Claude Help Center, accessed October 2026. Address list, robots.txt and IP-blocking guidance.
• Perplexity Crawlers — Perplexity, accessed October 2026. Range files, robots.txt timing, rule advice.
• Verify requests from Google crawlers and fetchers — Google Search Central, updated March 2026. DNS verification steps and range files.
• Runtime Logs — Vercel docs, accessed October 2026. Log types, filters, retention.
• User agent category — Netlify docs, accessed October 2026. Crawler and ai-agent definitions.
• Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study — Kim et al., ACM IMC '25, DOI 10.1145/3730567.3764471. Spoofing heuristic, re-check windows, compliance data.
• Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives — Cloudflare, 2025. Undeclared crawling as observed.

AgentReady generates an llms.txt and Markdown mirrors of your pages, hosts both at a public URL, and logs every hit from recognised AI crawlers, broken down by bot and by day. The method above works without it.