Tag
robots.txt
Using robots.txt to state crawler preferences, including for AI crawlers, and what it cannot enforce. 4 guides use this tag, newest first.
Websites & AI Crawlers
How AI Search Picks Sources: What Providers Actually Document
A provider-by-provider map of what Google, OpenAI, Perplexity, Anthropic, Microsoft and Apple document about crawling, citing and controls, plus how to measure.
Websites & AI Crawlers
How to See Which AI Bots Visit Your Site (and Check They're Real)
Count AI crawler hits in your access logs, then check each address against vendor IP lists and reverse DNS to catch spoofed user agents.
Websites & AI Crawlers
AI Crawlers Explained: Allow or Block GPTBot, ClaudeBot and More
Sort AI bots into training, search and user-triggered fetchers, then copy parser-checked robots.txt recipes and edge rules for the bots that ignore the file.
Websites & AI Crawlers
What Is llms.txt? The Format, the Evidence, and an Example
llms.txt explained: the v2 format, how it differs from robots.txt and sitemap.xml, what server logs say about who reads it, and a template to copy.