AI service bots crawl websites and collect content to train models or to answer users' questions. Well-known ones include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot. In robots.txt you can allow or block each of them. If you block all of them, your website will not appear in the answers of AI assistants.
FAQ / Glossary