Tag
#robots-txt
1 post
Anthropic's 3-bot identity problem: blocking ClaudeBot doesn't block Claude
Anthropic now operates ClaudeBot, Claude-User, and Claude-SearchBot as three separate robots.txt identities. Sites that only blocked ClaudeBot still leak content via the other two. Here's the full pattern - and the equivalent OpenAI two-bot trap.
#ai-crawlers#anthropic#robots-txt
Paul Lukic ·