Tag
#ai-crawlers
3 posts
llms.txt in 2026: a cargo cult most tools are afraid to call out
Google does not use llms.txt, and a 300k-domain study found zero citation lift. Who actually reads it, what belongs in it, and why a red MISSING is wrong.
#llms-txt#ai-crawlers#ai-overviews
Paul ·
The Google-Extended myth: it doesn't opt you out of AI Overviews
Google-Extended blocks Gemini training, not AI Overview citation. The real opt-out is nosnippet - and it removes your normal search snippet too.
#ai-overviews#google-extended#ai-crawlers
Paul ·
Anthropic's 3-bot identity problem: blocking ClaudeBot doesn't block Claude
Anthropic runs ClaudeBot, Claude-User and Claude-SearchBot as three robots.txt identities. Block only ClaudeBot and the other two still read your site.
#ai-crawlers#anthropic#robots-txt
Paul ·