What is an AI crawler?
An AI crawler is a bot that fetches web content on behalf of an AI system — for training, for live retrieval when a user asks a question, or both. The ones that matter most for visibility are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended (Google’s AI training control). Each identifies itself by user-agent and respects robots.txt.
Access is a genuine strategic choice, but many sites make it by accident: a blanket bot-blocking rule, a CDN default, or a robots.txt copied from a template quietly locks out every AI crawler — and a site AI systems cannot read is a site they will never cite. The first step of any AEO effort is simply confirming that the engines you want citations from are allowed in.
Retrieval-time crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) deserve different treatment from training-only ones: blocking them does not keep your content out of a model — it keeps your name out of the answers. The toolkit’s audit checks your robots.txt directives against each of the key AI bots and flags accidental blocks.