What Is the Difference Between GPTBot and OAI-SearchBot, and Which Should You Block?

What does each OpenAI agent do?

OpenAI says changes to robots.txt take about 24 hours to reach its search systems, and publishes IP address lists for each agent.

What does OAI-SearchBot do, and should websites allow it?

OAI-SearchBot is the crawler behind ChatGPT's search features. OpenAI says it is "used to surface websites in search results in ChatGPT's search features". It recommends allowing OAI-SearchBot in robots.txt, and allowing requests from its published IP ranges, if you want your site to appear. Sites that opt out are not shown in ChatGPT search answers.

For almost every business that wants customers to find it through ChatGPT, the answer is to allow it. The exceptions are sites that do not want to appear in AI answers at all, such as some publishers negotiating licences, or private content that should not be public anyway.

Which OpenAI bots should robots.txt allow for ChatGPT search visibility?

Allow OAI-SearchBot and ChatGPT-User. Whether to allow GPTBot is a separate choice about training, not search. A typical file for a business that wants search visibility but not training use:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

A business that wants the widest possible presence, including in what future models know without searching, can allow GPTBot too. Rules for other companies' crawlers are covered in how to allow search AI bots while blocking training crawlers, and Anthropic's equivalent agents in the difference between ClaudeBot, Claude-User and Claude-SearchBot.

Does blocking GPTBot stop live ChatGPT retrieval or only future crawling?

Only future training crawling. OpenAI says each setting is independent, and gives the example of a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training. Blocking GPTBot does not stop OAI-SearchBot indexing your pages for search, and does not stop ChatGPT-User opening them when a user asks. It also does not remove anything from models already trained.

Can I disallow GPTBot without disappearing from ChatGPT recommendations?

Yes, for answers that use search. When ChatGPT searches, it relies on its search index and providers, which OAI-SearchBot and ChatGPT-User feed, not GPTBot. The trade-off is in answers without search. ChatGPT answers many questions from what it learned in training, and blocking GPTBot keeps your newer content out of future training data. Over time, models may know less about you unless they search. For most businesses, that is a small cost next to losing search visibility, but it is worth knowing. When ChatGPT decides to search is covered in how ChatGPT decides when to use its search tools.

ChatGPT-User vs ChatGPT search: what is the difference for website owners?

ChatGPT search draws on an index built in advance, which OAI-SearchBot helps build, together with third-party search providers. ChatGPT-User is different: it visits a specific page at the moment a user's request needs it, for example when someone pastes your URL or ChatGPT decides to read a page it found. OpenAI says ChatGPT-User is not used to crawl the web automatically, and is not used to decide whether content appears in search. Because the visit is initiated by a user, robots.txt rules may not apply.

For site owners, that means OAI-SearchBot decides whether you can be found, and ChatGPT-User decides whether a page can be read on demand. Block ChatGPT-User only if you have a strong reason, and use a firewall if you do, since robots.txt may not stop it.

How can I get the OpenAI crawler to visit my website?

OpenAI offers no URL submission tool. To be discovered and indexed:

• Allow OAI-SearchBot in robots.txt and in your firewall, and make sure pages return a normal 200 response.
• Be indexed in Bing. ChatGPT search uses third-party search providers, and Bing has long been one of them. Verify your site in Bing Webmaster Tools, submit your sitemap and use IndexNow to announce new pages. More is in whether ChatGPT search uses Google, Bing or its own index.
• List your sitemap in robots.txt and link important pages from your navigation.
• Earn links and mentions from sites that are already crawled often.
• Ask ChatGPT about a specific URL. That triggers a one-off ChatGPT-User visit, which confirms access but does not guarantee indexing.

How do I allow OAI-SearchBot through Wordfence safely?

Wordfence, a common WordPress security plugin, can block or throttle crawlers even when robots.txt allows them. Its rate limiting treats automated visitors as crawlers and applies separate limits to them, with a special exception only for verified Google crawlers. To keep OpenAI's search agents working:

1. Check Wordfence's Live Traffic for OAI-SearchBot or ChatGPT-User requests that were blocked or throttled.
2. Review the Rate Limiting settings, especially the limits for crawler page views and 404s. Choose throttling rather than blocking for crawlers, and set limits your server can handle.
3. Consider allowlisting OpenAI's published IP ranges for OAI-SearchBot and ChatGPT-User in Wordfence's "Allowlisted IP addresses that bypass all rules" option. Use it carefully: those addresses skip every Wordfence check, so add only ranges from OpenAI's official lists, and update them when OpenAI changes them.
4. Do not allowlist by user agent alone, since anyone can claim to be OAI-SearchBot.
5. Recheck after changes by fetching a page with the crawler's user agent and watching Live Traffic.

We have not tested Wordfence configurations with OpenAI's crawlers ourselves; these steps are based on Wordfence's settings as described in its documentation and OpenAI's crawler documentation.

How do you check it is working?

• Read your live robots.txt for groups naming GPTBot, OAI-SearchBot or ChatGPT-User, and any * group that might catch them.
• Search your server logs for OAI-SearchBot visits and their response codes.
• Verify IPs against OpenAI's published lists.
• Ask ChatGPT with search about your topic after about a day, and see whether your pages are cited.

If an earlier block cost you visibility, see how to recover AI search visibility after a robots.txt block.

What might this look like in practice?

Consider an illustrative photography studio, Harlstead Studio, on WordPress. The owner blocked GPTBot to keep portfolio text out of training, but ChatGPT still never cited the studio. Wordfence's Live Traffic showed OAI-SearchBot requests being blocked for exceeding the crawler page view limit during a crawl of the gallery. The owner changed the crawler action to throttle, raised the limit, and allowlisted OpenAI's published OAI-SearchBot ranges. Within a few weeks, ChatGPT began citing the studio's pricing page for local portrait questions, while GPTBot stayed blocked.

What do people ask about OpenAI's crawlers?

Does blocking GPTBot remove me from ChatGPT search?

No. ChatGPT search uses OAI-SearchBot. GPTBot is for training.

Should I block OAI-SearchBot?

Only if you do not want to appear in ChatGPT search answers. Sites that block it are not shown there.

Does ChatGPT-User obey robots.txt?

OpenAI says robots.txt rules may not apply, because a user initiated the visit.

How long do robots.txt changes take to affect ChatGPT search?

OpenAI says about 24 hours.