How Hosting Providers Block AI Bots and How to Check for Crawler Restrictions

How do hosts restrict AI bots?

Reported examples include IONOS, which, according to SEO Südwest, returned 429 responses to training bots such as GPTBot and ClaudeBot on shared hosting while letting OAI-SearchBot and ChatGPT-User through.

Can default CMS settings block AI crawlers and limit AI visibility?

Yes, but the common platforms mostly leave AI blocking off by default:

• Squarespace has a “Block known artificial intelligence crawlers” option under Settings, then Crawlers. It is not checked by default. When on, it blocks a long list through robots.txt, including GPTBot, ClaudeBot and Google-Extended.
• WordPress has no AI toggle in core, but its “Discourage search engines from indexing this site” reading setting blocks indexing for everyone. SEO plugins such as Yoast have offered switches to block AI bots such as GPTBot, CCBot and Google-Extended, and security plugins can block or rate-limit bots by user agent.
• Framer generates robots.txt automatically, with custom files on higher plans.
• Cloudflare, used in front of many CMS sites, changed its defaults on September 15, 2026: new domains block Training and Agent bots on pages that show ads, while Search bots stay allowed. Mixed-purpose crawlers are blocked by any training block.

The risk is less the default than an old choice nobody remembers: a plugin setting switched on years ago, or a “block AI” option that also catches the AI search crawlers you want. Which bots to allow is covered in how to allow search AI bots while blocking training crawlers.

How do you test what AI crawlers receive?

1. Read robots.txt at your domain and check each AI user agent.
2. Request a page with AI user agents, for example curl -I -A "OAI-SearchBot" https://yoursite.com/page, and compare with a browser user agent. A 200 for the browser and a 403, 429 or challenge page for the bot means something is filtering.
3. Treat spoofed tests with care. Your test does not come from OpenAI's or Anthropic's addresses. Pressable, for example, says it may return a 403 challenge when a request claims a crawler identity but its source cannot be verified, and that this does not by itself mean the genuine crawler is blocked.
4. Check real logs. Look for requests from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot, and the status codes they received, as covered in how to measure AI bot traffic in server logs.
5. Check the CDN or firewall dashboard for blocked or challenged bot requests, as covered in how to find CDN or firewall rules that block AI crawlers.

How to find CAPTCHA and mod_security rules blocking AI crawlers on SiteGround

SiteGround runs several layers. Its policy, published in September 2025, blocks crawlers “intended for AI model training” at server level to protect client content and save resources, and its list of allowed AI crawlers includes OpenAI's ChatGPT-User and OAI-SearchBot and Anthropic's Claude-User and Claude-SearchBot. Separately, its Anti-Bot AI system challenges suspicious requests with a CAPTCHA rather than blocking IP addresses, and server firewall rules, such as the ModSecurity web application firewall many hosts run, can block requests that match attack patterns.

To find which layer is responsible:

1. Look at the response body, not just the status. A CAPTCHA or challenge page means the anti-bot system; a plain 403 suggests a firewall rule; a block only for training bots such as GPTBot matches SiteGround's stated policy.
2. Check the access log for the crawler's requests and their status codes.
3. Open a support ticket with timestamps, user agents and status codes, and ask which system blocked the request. Firewall rules and CAPTCHA exceptions are managed by SiteGround, not by a switch in your dashboard, and a change to one layer does not change the other.
4. Decide on training crawlers deliberately. Reports indicate SiteGround will consider changing the default for a site on request; if you want GPTBot or ClaudeBot allowed, ask explicitly.

How WordPress site owners can unblock AI crawlers and improve visibility

1. Check Settings, then Reading, and make sure the setting that discourages search engines is off on your live site.
2. Review SEO and security plugins for AI or bot options. Turn off blanket AI blocks if you want to be cited, or narrow them to training crawlers.
3. Review the robots.txt your plugins generate. Several plugins can each add rules.
4. Check your host and CDN, since blocks there override anything in WordPress.
5. Allow the crawlers that power answers: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, and PerplexityBot and Perplexity-User for Perplexity. Google's AI features use Googlebot.
6. Retest and watch logs over the following weeks for successful crawls.

Unblocking lets AI systems read your pages; it does not by itself make them cite you. Content still has to answer the questions people ask.

How to manage AI bot blocking on WP Engine when there is no customer-facing toggle

WP Engine says it deprioritises self-identified bots at platform level, and customers on its Global Edge Security (GES) network get a traffic view that sorts visits into human, verified bot, automated and likely automated, with verified bot types, traffic mitigation rules and an Under Attack Mode. WP Engine also writes that it is increasingly important to allow trusted LLM crawlers to index sites.

If AI crawlers receive 429 or error responses while browsers get 200:

1. Gather evidence from logs: user agents, times, paths and status codes.
2. Ask support which platform or edge rules apply to those user agents, and request that verified AI search and user-action crawlers be allowed.
3. Use GES rules if you have GES, to allow the specific bots you want.
4. Retest after each change with real log data, not only spoofed requests.

Do Kinsta, Pressable, and Pantheon block AI bots?

• Kinsta: not by default. Its bot protection blocks known malicious traffic and allows bots on Cloudflare's verified list, which Kinsta says includes AI bots. A separate Block AI crawlers setting, off unless you turn it on, blocks AI and training crawlers without affecting Googlebot or Bingbot. Stricter challenge modes challenge excessive-rate AI crawlers.
• Pressable: most bots are allowed on live sites, and staging sites use a robots.txt that disallows everything. It may challenge requests that claim a crawler identity it cannot verify.
• Pantheon: its documentation does not describe a platform-wide AI crawler block. It recommends site-level tools, such as rules on its Advanced Global CDN, a WAF and Drupal modules, to control aggressive crawlers.

Whether blocking is right for your business at all is covered in whether businesses should allow or block AI crawlers, and how common accidental blocks are in how many websites accidentally block AI crawlers.

We have not tested each host's behaviour with genuine AI crawler traffic. This page is based on the hosts' published documentation and policies, Cloudflare's documentation, and reporting on host policies.

What might this look like in practice?

Consider an illustrative physiotherapy clinic, Trellisk Physio, on managed WordPress hosting. Its robots.txt allowed everything, yet ChatGPT never cited its treatment pages. Server logs showed OAI-SearchBot receiving 429 responses and ClaudeBot receiving 403. A security plugin had a “block AI bots” option switched on, and the host rate-limited self-identified bots. The team turned off the plugin option, kept training crawlers blocked in robots.txt, and asked the host to allow verified search and user-action crawlers. Within weeks the logs showed 200 responses to OAI-SearchBot.

What do people ask about hosts and AI crawlers?

Does SiteGround block ChatGPT?

SiteGround says it blocks AI training crawlers but allows ChatGPT-User and OAI-SearchBot, which ChatGPT uses to answer users.

Is a 403 in my curl test proof of a block?

No. Hosts may challenge requests that claim to be a crawler but come from unverified addresses. Check real logs.

Does Kinsta block AI bots?

Not by default. It offers an optional Block AI crawlers setting.

Can a CDN block AI crawlers even if my host allows them?

Yes. CDN rules apply before the request reaches your host, so check both.