How Do I Test Whether GPTBot, PerplexityBot and ClaudeBot Can Crawl My Site?

Which crawlers should you test?

Test the ones that matter for your goals, and know what each does:

People often test GPTBot and ClaudeBot and conclude they are visible or invisible in AI answers. But those two are training crawlers. For ChatGPT, Perplexity and Claude answers, the search crawlers and user-triggered fetchers matter more.

How do I check whether robots.txt allows AI crawlers like GPTBot, PerplexityBot and ClaudeBot?

Open https://yourdomain.com/robots.txt and read it with three rules in mind:

1. A crawler uses only the group that names it. If there is a "User-agent: GPTBot" group, GPTBot ignores "User-agent: *" entirely. A named group containing only "Allow: /" therefore drops your default disallows for that bot.
2. Unnamed crawlers use the default group. If OAI-SearchBot is not named, "User-agent: *" decides.
3. The longest matching rule wins, and an Allow wins a tie.

Check every host separately, such as www, shop, docs and help subdomains, since each serves its own file. Also check the status code: a 5xx error on robots.txt makes compliant crawlers treat the whole site as blocked, while a 404 means no restrictions.

Be careful with automated checks. When we tested Python's built-in robots.txt parser with a rule such as "Disallow: /*?sort=", it ignored the wildcard and reported the URL as allowed. Wildcards are part of the internet standard for robots.txt, RFC 9309, so use a parser that implements it, such as Google's open-source robots.txt library, or read the rules yourself. The full rule set is covered in how AI crawlers handle robots.txt.

How to use curl and AI bot user agents to test whether your site blocks AI crawlers

Request the same page with a normal browser user agent and with each bot's token, and compare:

# Baseline: a normal browser
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yourdomain.com/pricing

# AI crawlers (tokens; operators publish the full strings)
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yourdomain.com/pricing
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://yourdomain.com/pricing
curl -s -o /dev/null -w "%{http_code}\n" -A "ClaudeBot" https://yourdomain.com/pricing

# Is the content in the HTML the crawler receives?
curl -s -A "OAI-SearchBot" https://yourdomain.com/pricing | grep -i "your plan name"

What the results mean:

• 200 for all, and your text appears: no user-agent block, and content is server-rendered.
• 403, 429 or 503 only for bot user agents: a firewall, CDN or security plugin rule is matching the user agent.
• 200 but your text is missing: the page is built with JavaScript. Most AI crawlers do not run JavaScript, so they see an empty shell.
• A challenge page in the HTML: bot protection is intercepting the request.

The trap is that many CDNs verify bots by IP address, reverse DNS or signatures, not by user agent. Your laptop claiming to be OAI-SearchBot is an impostor. If it is blocked, that may mean your rules correctly block spoofers, while the real OAI-SearchBot is allowed. If it is allowed, the real bot could still be blocked by a rule that matches verified bot categories. Use curl to find user-agent rules and rendering problems, then confirm with logs.

How do you confirm what the real crawlers receive?

From your server or CDN logs. Filter requests by user agent for each bot, then check:

1. Status codes: mostly 200, with 304 for unchanged pages, and no runs of 403, 429 or 503.
2. Paths: whether the crawler reaches the pages that matter, such as product, pricing and guides.
3. Identity: match the source IPs against the lists OpenAI, Perplexity and Anthropic publish. Requests that claim a bot name from other IPs are impostors and can be ignored or blocked.
4. Frequency: a crawler that visited last month and has stopped may have hit a new block.

On Cloudflare, Security Analytics shows which rule or service acted on each request, and AI Crawl Control shows AI crawler activity and robots.txt compliance. How to read this traffic over time is covered in how to see and measure AI bot traffic.

What free tools can check whether AI bots are blocked from a website?

• Your browser and curl, as above, for robots.txt and user-agent tests.
• Search Console's robots.txt report, for how Google reads your file, and URL Inspection for rendering, though both cover Google only.
• Bing Webmaster Tools, whose robots.txt tester checks rules for Bing's crawlers.
• Screaming Frog SEO Spider, whose paid licence can crawl with a custom user agent such as GPTBot and respect robots.txt as that agent; the free version, up to 500 URLs, crawls only with its default settings.
• Your CDN's dashboard, such as Cloudflare's Security Analytics and AI Crawl Control, which show real crawler requests and the actions taken.
• The assistants themselves: ask ChatGPT, Claude or Perplexity to open and summarise a specific page, then look for ChatGPT-User, Claude-User or Perplexity-User in your logs. This tests user-triggered fetchers, not the search crawlers.

Online robots.txt checkers that let you choose a user agent are also useful for a quick look, but check that they support wildcards.

What do the results usually reveal?

In practice, blocks come from a handful of places: a copied "block all AI" list in robots.txt that also names search crawlers; a security plugin or CDN setting that challenges or blocks bots; rate limits tuned for humans; and pages whose content only appears after JavaScript runs. How often sites block AI crawlers without meaning to is discussed in how many websites accidentally block AI crawlers, and fixing CDN and firewall causes in how to find and fix CDN or firewall rules that block AI crawlers.

How often should you test?

After any change to robots.txt, CDN or security settings, themes or hosting, and otherwise monthly. A simple scheduled script that fetches robots.txt, runs the curl checks against five key pages and alerts on changes catches most accidental blocks within a day.

What might this look like in practice?

Consider an illustrative garden furniture retailer, Norvmere Outdoor. Its robots.txt allowed every AI crawler, yet ChatGPT never cited its product pages. A curl request with the OAI-SearchBot token returned 200, but a grep for the product name found nothing: the product details loaded by script. Its logs showed the real OAI-SearchBot receiving 200 responses with the same empty template. Norvmere switched its product templates to server-side rendering, and its next curl test showed the product name, price and dimensions in the HTML. Within weeks, ChatGPT's answers about weatherproof garden sofas began citing Norvmere's pages.

What do people ask about testing AI crawler access?

Does a 200 response mean an AI crawler can read my page?

Not necessarily. Check that your content appears in the HTML, since most AI crawlers do not run JavaScript.

Is a curl test with GPTBot's user agent reliable?

It shows how your rules treat that user agent, but CDNs often verify real bots by IP, so confirm with logs.

Which crawler matters for ChatGPT search?

OAI-SearchBot. GPTBot is for training, and ChatGPT-User fetches pages when a user asks.

Can Python check robots.txt for me?

Its built-in parser ignored wildcard rules in our test, so use an RFC 9309-compliant parser or check by hand.