Tag
#ai-crawlers
3 posts
llms.txt in 2026: a cargo cult most tools are afraid to call out
Google does not use llms.txt. The 300k-domain study found zero citation lift. Here is who actually reads it, what it should contain, and why every SEO tool that shows MISSING llms.txt as a red error is participating in misinformation.
Paul Lukic ·
The Google-Extended myth: it doesn't opt you out of AI Overviews
Almost every SEO post gets this wrong. Google-Extended blocks Gemini training data, not AI Overview citation. The actual AI Overview opt-out is nosnippet - and it kills your regular SERP snippet too. Here's the breakdown.
Paul Lukic ·
Anthropic's 3-bot identity problem: blocking ClaudeBot doesn't block Claude
Anthropic now operates ClaudeBot, Claude-User, and Claude-SearchBot as three separate robots.txt identities. Sites that only blocked ClaudeBot still leak content via the other two. Here's the full pattern - and the equivalent OpenAI two-bot trap.
Paul Lukic ·