How Do I Recover AI Search Visibility After robots.txt Blocked ChatGPT, Perplexity, and Google Crawlers?

What did the block actually affect?

Start by mapping the blocked names to what they do. The same "block AI" list can cause very different damage:

If you only blocked training tokens, there may be nothing to recover in AI search. If you blocked Googlebot or Bingbot, the problem is bigger than AI and needs treating as a search indexing incident. Recovering from a wider drop is covered in how to recover a deindexed website.

Why does my robots.txt file block AI crawlers, and how do I fix it?

The usual causes:

• A copied "block all AI bots" list that included search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot along with training crawlers.
• A staging rule shipped to production, typically "User-agent: * Disallow: /".
• A plugin or CDN feature that adds AI blocks, such as an SEO or security plugin's one-click setting, or a CDN's managed robots.txt.
• A named group that replaced the defaults. A crawler with its own group ignores "User-agent: *", so an overly broad named group can block more than intended.
• Other hosts. A separate subdomain or country site with its own restrictive file.
• Server errors on robots.txt. A file that returns a 5xx status is treated by compliant crawlers as a complete block.

WordPress's "Discourage search engines" setting is no longer a robots.txt issue: since version 5.3 it adds a noindex meta tag instead. It still removes pages from search, so check it too.

To fix the file: remove disallows for the search crawlers and fetchers you want, keep training blocks if you still want them, and make sure /robots.txt returns 200 with plain text on every host. The rules each crawler follows are explained in how AI crawlers handle robots.txt.

What should you check besides robots.txt?

The firewall and CDN. Many "robots.txt" problems are really edge blocks that return 403 to AI crawlers whatever robots.txt says. Check your server or CDN logs for OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot and Googlebot, and look at the status codes they receive after the fix. If they still get 403, 429 or challenge pages, the WAF is the problem, as covered in how to find and fix CDN or firewall rules that block AI crawlers. Also check for noindex tags or headers added during the same change, since they remove pages from Google and Bing even when crawling is allowed.

How do you tell each platform the block is gone?

• Google. Open Search Console's robots.txt report, which shows the files Google found for your top 20 hosts and lets you request a recrawl of robots.txt in urgent cases. Then use URL Inspection to request indexing for your most important pages, and resubmit your sitemap.
• Bing and Copilot. In Bing Webmaster Tools, submit key URLs, resubmit your sitemap, and send changed URLs through IndexNow.
• ChatGPT. There is no submission tool. OpenAI says robots.txt changes for OAI-SearchBot take about 24 hours to be reflected, after which its crawler can return. Keep sitemaps current and internal links clean so it finds your pages.
• Perplexity and Claude. No submission tools either. Make sure their crawlers are allowed in robots.txt and in the WAF, using the IP lists they publish.

How long does recovery take?

In stages. Robots.txt itself is picked up quickly: RFC 9309 says crawlers should not cache it for more than 24 hours, and Google says it generally caches the file for up to a day. Recrawling your pages takes longer, depending on site size and how often each crawler visits. Answers then reflect the recrawled pages as each system refreshes its index.

Pages that were cited heavily before the block, and that other sites link to, usually return first. Deep pages with few links can take much longer.

How do you confirm visibility is coming back?

Watch four signals:

1. Crawler requests in your logs, with 200 responses, for each bot you unblocked.
2. Google and Bing coverage: indexed pages in Search Console and Bing Webmaster Tools, and AI impressions in Search Console's generative AI report.
3. Bing's AI Performance report, where Copilot citations and grounding queries should reappear.
4. Referrals and manual checks: AI Assistant sessions in GA4, and the answers to 20 key questions in ChatGPT, Perplexity, Copilot and Google.

Keep a dated log of the incident and the fix. It helps explain the dip in reports, and it is the evidence you need if recovery stalls.

How do you stop it happening again?

• Keep robots.txt in version control, with a review step for changes.
• Separate staging and production rules, so a staging file cannot be deployed live.
• Name training crawlers explicitly instead of using broad "block all AI" lists.
• Monitor it: an automated check that alerts when robots.txt changes, or when key crawlers receive errors.
• Audit plugins and CDN settings after updates.

A wider list of quick technical wins is in which technical SEO fixes improve AI search visibility first.

What might this look like in practice?

Consider an illustrative accountancy software firm, Ledgerwick, whose ChatGPT and Perplexity referrals fell to almost nothing in a month. Its robots.txt had been replaced by a security plugin update with a list that disallowed GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Claude-SearchBot. Googlebot was untouched, so Google traffic held. Ledgerwick restored its previous file, kept GPTBot and ClaudeBot blocked by choice, and its logs showed that OAI-SearchBot and PerplexityBot received 200 responses within two days. ChatGPT referrals began returning within a week and were back to earlier levels after about five weeks, led by its most-linked comparison pages.

What do people ask about recovering from a robots.txt block?

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls Gemini training and grounding in Gemini apps. AI Overviews use Googlebot.

Can I speed up ChatGPT's recrawl?

Not directly. Allow OAI-SearchBot, keep sitemaps and internal links current, and make sure the firewall is not blocking it.

How quickly do crawlers notice a robots.txt fix?

Usually within a day, since crawlers should not cache robots.txt for longer than 24 hours.