How to Manage robots.txt for AI Crawlers on Webflow, Framer and Next.js

What rules should AI crawlers get?

Most sites want to be found in AI search but may not want their content used for training. A common setup:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Google-Extended and Applebot-Extended are tokens you can add to either group: they control training and, for Google, grounding in the Gemini app, without affecting Google or Apple search. Why each crawler matters, and the trade-offs, are covered in how to allow search AI bots while blocking training crawlers. A crawler follows only the most specific group that names it, so a bot listed in a named group ignores the rules under User-agent: *. Repeat private-path blocks in every named group that allows anything.

How to control AI bots in Webflow site settings

Webflow publishes a robots.txt for every site and lets you add your own rules:

1. Open Site settings, then SEO, then the Indexing section.
2. Enter your rules in the robots.txt field, such as the example above with your own sitemap URL.
3. Save and publish the site. Rules take effect only after publishing.

Webflow also has related controls in the same area:

• Staging indexing. Turning it off publishes a separate robots.txt for your webflow.io staging subdomain that keeps it out of search, so your staging copy does not compete with your real domain in AI answers.
• Page-level indexing. Unchecking “Index this page” in a page's settings adds noindex, which removes that page from search results rather than just blocking crawling.

Webflow sites are served through Cloudflare. Webflow has said AI crawler traffic to its hosted sites rose by more than 125% in the first part of 2025 and that it is working with Cloudflare on further AI crawler controls, so check its site settings for new options.

How to manage robots.txt for AI crawlers on Framer sites

Framer automatically generates a robots.txt for every published site, which you can view at yourdomain.com/robots.txt. To change it:

1. Check your plan. Custom robots.txt uses Static Files, available on Pro and Enterprise plans.
2. Create a robots.txt file with your rules on your computer.
3. Upload it in the project dashboard: select your domain, open the Files tab, and upload the file with the path set to /.
4. Publish and check the live file.

Framer's help centre says an uploaded file replaces the automatic one entirely, so include your sitemap line and any rules you still need from the default. On other plans you cannot edit robots.txt, but you can still control indexing page by page with Framer's SEO settings. The same Static Files feature serves llms.txt, as covered in how to add llms.txt on Webflow, Framer, Next.js or any CMS.

How to allow AI crawlers in a Next.js robots.txt file

With the App Router, Next.js offers two ways:

• A static file: place robots.txt in the root of the app directory, with plain text rules.
• A generated file: add app/robots.ts that returns a Robots object. Next.js turns it into robots.txt.

A generated version of the rules above:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: ['OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot',
                    'Claude-User', 'PerplexityBot', 'Perplexity-User'],
        allow: '/',
        disallow: '/admin/',
      },
      { userAgent: ['GPTBot', 'ClaudeBot', 'CCBot'], disallow: '/' },
      { userAgent: '*', disallow: '/admin/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Next.js caches robots.ts by default unless it uses request-time data, which suits a file that rarely changes. The generated version is useful when rules differ by environment, for example blocking everything on preview deployments. With the Pages Router, put a plain robots.txt in the public folder instead.

Should you block AI training crawlers?

It is a business decision, not a technical requirement. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot keeps new content out of future model training, but it does not remove you from AI search, which uses separate crawlers. Allowing them may help models learn your brand and basics over time, which matters for questions assistants answer without searching. Publishers often block training while allowing search; many product companies and service businesses allow both because being known helps them. Whatever you choose, write it down, apply it the same way on every domain and subdomain you run, and revisit it when AI companies add or rename crawlers, which happens several times a year.

What should you check after publishing?

• Open /robots.txt on the live domain and read it. Look for leftover staging rules such as “Disallow: /” under User-agent: *.
• Test specific bots with a robots.txt tester, or by checking server or CDN logs for requests from OAI-SearchBot, Claude-SearchBot and PerplexityBot.
• Check that the sitemap URL is correct and reachable.
• Re-check after redesigns or platform migrations, when defaults often come back.

What else can block AI crawlers?

robots.txt is only a request. Bots can also be stopped, or let through, by other layers:

• Hosting and CDN bot protection, which may challenge unfamiliar crawlers even when robots.txt allows them, as covered in how hosting providers block AI bots.
• Firewall rules that block by user agent or country.
• Login walls, which no crawler gets past, and JavaScript-only content, which most AI crawlers, including those used by ChatGPT, Claude and Perplexity, cannot read because they do not run scripts; Googlebot and Applebot do.

How robots.txt and firewall rules work together is covered in how to use robots.txt and a WAF to allow useful AI bots, and common robots.txt mistakes in how AI crawlers handle robots.txt.

We have not tested every plan on these platforms ourselves. This page is based on Webflow's, Framer's and Next.js's documentation and the AI companies' crawler documentation.

What might this look like in practice?

Consider an illustrative design studio, Harlstead Studio, that moved its site from Webflow to Next.js. Its Webflow robots.txt had allowed AI search crawlers and blocked training bots, but the new site shipped with a template robots.ts that disallowed everything outside production, and the production check used the wrong environment variable. For three weeks, OAI-SearchBot and PerplexityBot received a blanket block. The developer fixed the environment check, rewrote robots.ts with separate groups for search and training crawlers, and added a deployment test that fetches /robots.txt. Within a month, the studio's case studies were being cited again in AI answers about brand design agencies.

What do people ask about robots.txt on these platforms?

Does Framer let me edit robots.txt?

Yes, on Pro and Enterprise plans, by uploading a custom file through Static Files, which replaces the automatic one.

Where is robots.txt in Webflow?

Under Site settings, then SEO, then Indexing. Save and publish the site for changes to take effect.

Should I use robots.txt or robots.ts in Next.js?

Use a static file for simple rules and robots.ts when rules depend on the environment or need to be generated.

Does blocking GPTBot remove me from ChatGPT search?

No. ChatGPT search uses OAI-SearchBot; GPTBot is for training.