Reference · July 2026
Every AI crawler, one list.
The 15bots we see fetching pages for AI platforms, what each one feeds, and whether it obeys robots.txt. We welcome all of them on CiteMP, because a blocked bot can't cite you.
The crawler list
| User agent | Company | What it feeds | Obeys robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Yes |
| OAI-SearchBot | OpenAI | ChatGPT search index | Yes |
| ChatGPT-User | OpenAI | Live answers on request | Yes |
| ClaudeBot | Anthropic | Model training | Yes |
| Claude-SearchBot | Anthropic | Claude search index | Yes |
| Claude-User | Anthropic | Live answers on request | Yes |
| PerplexityBot | Perplexity | Search index | Yes |
| Perplexity-User | Perplexity | Live answers on request | Partial |
| Google-Extended | Gemini training (token) | Yes | |
| GoogleOther | Research crawls | Yes | |
| Applebot-Extended | Apple | Apple AI training (token) | Yes |
| Meta-ExternalAgent | Meta | AI training | Yes |
| Amazonbot | Amazon | Alexa and AI answers | Yes |
| CCBot | Common Crawl | Open training dataset | Yes |
| Bytespider | ByteDance | AI training | Often no |
Copy-paste robots.txt rules
Welcome AI bots (our setup)
# robots.txt — welcome AI crawlers (what we use) User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
Block training bots only
# robots.txt — block AI training crawlers only User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: /
Want the citations without the crawl budget? Pair open robots with an llms.txt from our free generator.