An AI crawler is an automated agent that fetches web pages for an AI company, either to answer users in real time or to collect training data.
There are three kinds. Search crawlers (such as OAI-SearchBot, PerplexityBot and Claude-SearchBot) index pages so assistants can cite them. User-triggered fetchers (such as ChatGPT-User) load a page when a user's question needs it. Training crawlers (such as GPTBot, ClaudeBot and Google-Extended) collect data for future models.
Each can be allowed or blocked separately in robots.txt. Blocking search crawlers, often by accident through CDN bot protection, keeps a site out of AI answers. Many AI crawlers don't run JavaScript, so content that only loads in the browser may be invisible to them.