FLPL
Dibyendu MondalShort answer
If you want to appear in AI search answers, allow the search crawlers: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, and Googlebot for Google's AI Overviews. Training crawlers like GPTBot are a separate choice that doesn't control search visibility.
Search crawlers vs training crawlers
AI companies run different bots for different jobs, and robots.txt treats each one separately. The important split:
| Bot | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search. Blocked sites won't be shown in ChatGPT search answers. |
| GPTBot | OpenAI | Crawls content that may be used to train models. |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT to. Not used to decide what appears in search. |
| PerplexityBot | Perplexity | Helps sites appear and get linked in Perplexity search results. |
| Perplexity-User | Perplexity | Fetches a page when a user asks a question. |
| Googlebot | Crawls for Search, including AI Overviews and AI Mode. | |
| Google-Extended | Controls use of content for AI training and grounding in some other Google systems. Doesn't affect Search. |
A sensible default for a SaaS marketing site
Most SaaS companies want to be found and recommended. For a public marketing site, a reasonable starting point is to allow everything:
User-agent: *
Allow: /
Sitemap: https://www.yoursite.com/sitemap.xmlIf you'd rather not have your content used for training, you can block the training bots and keep the search bots:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Check more than robots.txt
Bots are often blocked somewhere else: CDN bot-fight modes, firewall rules or rate limits. If you use Cloudflare or similar, check whether AI crawlers are being challenged or blocked, because robots.txt can say "allow" while the CDN says "no".
Don't block your app by accident, or open it by accident
Keep logged-in areas, admin pages and customer data out of crawling with proper authentication, not robots.txt alone. robots.txt is a request, not a lock.
Questions people also ask
If I block GPTBot, will I disappear from ChatGPT?
Not from ChatGPT search, according to OpenAI's documentation: each setting is independent, and search visibility depends on OAI-SearchBot. Blocking GPTBot signals that content shouldn't be used for training.
Does Google-Extended affect AI Overviews?
No. Google says Search crawling, including AI Overviews, is controlled by Googlebot. Google-Extended is about use in some other Google AI systems.
Do user-triggered bots follow robots.txt?
Not necessarily. OpenAI and Perplexity both say their user-triggered agents may not follow robots.txt, because a person asked for the page.