Tag
AI crawlers
The bots that read websites for AI systems: who they are, what they do, and how to allow, block, or check them. 4 guides use this tag, newest first.
How AI Search Picks Sources: What Providers Actually Document
A provider-by-provider map of what Google, OpenAI, Perplexity, Anthropic, Microsoft and Apple document about crawling, citing and controls, plus how to measure.
How to See Which AI Bots Visit Your Site (and Check They're Real)
Count AI crawler hits in your access logs, then check each address against vendor IP lists and reverse DNS to catch spoofed user agents.
AI Crawlers Explained: Allow or Block GPTBot, ClaudeBot and More
Sort AI bots into training, search and user-triggered fetchers, then copy parser-checked robots.txt recipes and edge rules for the bots that ignore the file.
What Is llms.txt? The Format, the Evidence, and an Example
llms.txt explained: the v2 format, how it differs from robots.txt and sitemap.xml, what server logs say about who reads it, and a template to copy.