saidani.net

What AI crawlers are, and why they visit your website

Before an AI assistant can talk about a website, a bot has to read it.

Companies like OpenAI, Anthropic, Perplexity and Google send automated programs, called crawlers or bots, to read public web pages. They come for three different reasons.

1. Training bots

They collect pages to train AI models. What they read can shape what a model knows in general. Examples: GPTBot, ClaudeBot, CCBot.

2. Search bots

They build an index, a kind of library catalogue, that an assistant searches when it answers a question. Examples: OAI-SearchBot, PerplexityBot.

3. User-request bots

They fetch one page right now, because a person just asked an assistant about it. Examples: ChatGPT-User, Claude-User, Perplexity-User.

Why it matters

If a bot cannot read a page, the assistant behind it cannot use that page. Knowing which bots visit, and how often, is the first step to understanding how visible a website is in AI answers.