Most website owners have never opened their own robots.txt file. There's rarely a reason to - it's a small text file that tells crawlers which parts of a site they're allowed to visit, and for years the only thing that touched it was Googlebot. That's changed. There's now a small population of AI-specific bots reading that same file, and depending on what it says, some of them turn around and leave.
What robots.txt actually controls
Every site has one, sitting at yourdomain.com/robots.txt. A well-behaved bot reads it before crawling anything and respects whatever it says. A badly-behaved one can ignore it, and that matters more here than it used to.
The file works in "User-agent" and "Disallow" pairs. A line reading User-agent: GPTBot followed by Disallow: / tells that specific bot to stay off the entire site. Leave the Disallow line out, or leave the bot unmentioned altogether, and it's free to crawl.
Not all AI bots do the same job
This is the part that trips people up. There isn't one "AI bot" to allow or block - there are several, from different companies, doing genuinely different things, and treating them as one decision is where most of the damage happens.
OpenAI runs two separate bots and says so directly in its own documentation: GPTBot gathers content to train its models, while OAI-SearchBot decides whether a page can be surfaced and cited inside ChatGPT's search answers. Blocking GPTBot only opts out of training. Blocking OAI-SearchBot removes a page from ChatGPT's search results. Perplexity draws the same line - PerplexityBot indexes pages for citation, and Perplexity says explicitly it isn't used to train its underlying model at all.
So there are two families here. Training bots - GPTBot, ClaudeBot, Google-Extended, Applebot-Extended - only affect whether a model learns from your content during training. Search and retrieval bots - OAI-SearchBot, PerplexityBot, ChatGPT-User, Perplexity-User - are the ones deciding whether your page can show up as a citation right now. Blocking the first group doesn't touch AI search visibility. Blocking the second group does, directly.
What most sites are actually doing
BuzzStream analysed the robots.txt files of the top 50 news sites in the UK and top 50 in the US and found 79% block at least one AI training bot, and 71% block at least one AI search or retrieval bot. PerplexityBot specifically - the one that indexes pages for citation, not training - is blocked by 67% of them. Only 14% block every AI bot going. 18% block none at all.
That's a sample of major news publishers, not small B2B sites - the percentages describe how the biggest players behave, not what every website does. But the pattern behind it holds regardless of size: a large share of that 67% almost certainly didn't sit down and decide to opt out of AI citations specifically. Blocking "AI bots" is usually one bundled decision, set by a security plugin, a hosting default, or a developer years ago who was blocking scrapers generally. PerplexityBot gets swept up in it without anyone weighing the trade-off.
The part that surprised me: blocking doesn't stop citations either
BuzzStream ran a second study checking whether the sites blocking these bots were actually being excluded from AI citations - using Citation Labs' citation-tracking tool across 4 million citations and 3,600 prompts. They weren't. 88.2% of sites blocking GPTBot still turned up in citations. 92.3% of sites blocking Google-Extended still turned up. Even 70.6% of sites blocking ChatGPT-User's live retrieval - the bot fetching a page in the moment someone asks about it - still got cited.
Some of this content was crawled and indexed long before a site ever added the Disallow line, so the block came too late to matter. Some of it comes through Common Crawl, the open web archive several AI models train on, which sits outside any individual site's robots.txt. And some AI crawlers fetch pages regardless of what robots.txt says - Reuters reported evidence of this directly, and Cloudflare separately found Perplexity using undeclared crawlers that bypass no-crawl instructions altogether.
None of that makes robots.txt pointless. It's still the correct, low-effort way to state a preference, and well-behaved bots do follow it. It just means a blank robots.txt file isn't a guarantee you'll get cited, and a restrictive one isn't a guarantee you won't. Treat it as one signal among several, alongside everything else that decides whether a page gets cited.
What to actually check on your own site
Type yourdomain.com/robots.txt into a browser. If nothing loads, there's no file, which by default means every bot is welcome. If something does load, scan it for these five names: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. If any of them sit above a line reading Disallow: /, that bot is blocked from the whole site.
Pay closest attention to OAI-SearchBot and PerplexityBot specifically, since those two do the actual citation work and the others only affect training. If either is blocked and you didn't put it there on purpose, the cause is usually a security plugin or a "block AI scrapers" toggle that bundled every AI-related bot into one switch. Turning that toggle off for search bots while leaving training bots blocked is a five-minute fix once you know where to look - usually inside the same plugin, or by editing the robots.txt file directly if the site doesn't use one.
A few common follow-up questions
Does blocking GPTBot hurt my chances of being cited in ChatGPT?
No. GPTBot only affects training data. OAI-SearchBot governs ChatGPT's search and citation feature, and OpenAI documents the two as separate settings.
If blocking doesn't stop citations anyway, why bother checking robots.txt at all?
Because the direction of the mistake usually runs the other way. The real risk is a search bot getting blocked by accident, when only a training bot was meant to be. Checking costs five minutes and only ever removes an accidental restriction, never adds a real one.
How long does it take for a robots.txt change to take effect?
OpenAI states it can take around 24 hours for a robots.txt update to reach its systems. Perplexity gives the same estimate. Neither is instant, so a change made today won't show up in tomorrow morning's citations.