How Can I Get My Website Included in AI Training Data?

Which crawlers collect training data?

Check your robots.txt for rules that block them, then check other layers: security plugins, hosting presets and CDN settings often add blocks. Cloudflare's managed robots.txt, for example, disallows GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta's crawler, among others.

Why does training data matter?

Assistants answer many questions from what they learned, without searching. Muck Rack's May 2026 study found Claude cited sources in only 55% of its responses, compared with 96% for ChatGPT. When no search happens, the model's training is the whole answer. Brands that appear widely in that training are named more often for broad questions, which is part of why long-established companies dominate category answers, as covered in why HubSpot dominates AI answers.

But training data is always behind. A model knows only what was collected before its cutoff, and that cutoff is often a year or more before you use it. How cutoffs work is covered in how LLM training cutoff dates work.

How to get a website included in Common Crawl for future AI training runs

Common Crawl is a non-profit that publishes a free copy of a large sample of the web. Mozilla Foundation research in 2024 found that at least 30 of 47 large language models released between 2019 and October 2023 used a filtered version of it, and that it made up more than 80% of the training tokens for OpenAI's GPT-3. Its main crawls, published regularly, each contain billions of URLs.

It is a sample, not the whole web. Common Crawl says it archives a randomly selected subset of each site, not entire websites. According to the Mozilla research, it ranks domains by harmonic centrality, a measure of how easily a domain can be reached through links from other sites, and samples more pages from domains that score higher. So links from other domains matter more than anything you do on your own site.

To give your site the best chance:

1. Allow CCBot in robots.txt. If you block it, it stops crawling your site.
2. Announce your sitemap in robots.txt. Common Crawl says CCBot uses any sitemap listed there.
3. Serve text in plain HTML. CCBot does not run JavaScript or use cookies, so content that loads through scripts is invisible to it.
4. Keep the server responsive. CCBot slows down when it receives 429 or 5xx responses.
5. Earn links from other well-linked sites, since domain centrality drives how much is sampled.
6. Check your pages in Common Crawl's URL index at index.commoncrawl.org, which lets you search each archive for a URL pattern.

To check that a visitor claiming to be CCBot is genuine, Common Crawl publishes its IP ranges, and its crawler addresses have reverse DNS names ending in crawl.commoncrawl.org.

What happens after pages are collected?

Collection is only the first step. AI builders filter crawl data before training: removing duplicates and near-duplicates, boilerplate and navigation, text in unwanted languages, and pages judged low quality. Mozilla notes that Common Crawl's coverage is skewed towards English. Pages most likely to survive are the ones with substantial, original text in the main content: explanations, specifications, guides and data that are not copied elsewhere.

Your own site is also only part of what a model learns about you. Mentions in widely used sources, such as news coverage, reference sites, public documentation and well-known directories, reach training data too and are seen by more crawls. Those are worth more than any change to your robots.txt.

How long until a model knows about you?

There is no schedule you can rely on. A page must be crawled, kept through filtering, and then included in a training run for a model that is later released. That can take many months, and a given page may never make it. You cannot see the training data behind ChatGPT, Claude or Gemini. A few open models, such as Ai2's OLMo, publish their training data and let you search it, but for the assistants your customers use, the practical test is to ask the model about your business with search turned off, or through a version without web access, and see what it knows. The difference between trained knowledge and searched answers is covered in how to tell whether an AI answer used retrieval or training data.

Should you focus on training data or AI search?

For most businesses, search first. Assistants search the web for current information, prices, local questions and comparisons, which are the questions that lead to customers. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus Googlebot and Bingbot, decide whether you can be found there, and changes show up in weeks rather than model generations.

Allowing training crawlers is still worth doing if you want models to know your name and basics. If you would rather keep content out of training, you can block training crawlers while allowing search crawlers, as covered in how to allow search AI bots while blocking training crawlers. Many publishers choose that split.

We have not tested how quickly a new site enters any model's training data. This page is based on Common Crawl's documentation, crawler documentation from AI companies and Mozilla Foundation research.

What might this look like in practice?

Consider an illustrative specialist bakery equipment supplier, Ovenwright, that found assistants without search had never heard of it. Its robots.txt, set by its CDN, blocked GPTBot, ClaudeBot and CCBot, and its product pages loaded descriptions through a script. The owner allowed training and search crawlers, moved descriptions into the HTML, and listed the sitemap in robots.txt. Common Crawl's URL index showed the product pages in a later archive. Meanwhile, a trade magazine feature and a listing in an industry directory earned links. Assistants with search began naming Ovenwright within weeks; whether future models will know it without searching remains to be seen.

What do people ask about AI training data?

Can I submit my site to ChatGPT's training data?

No. There is no submission process. Allowing GPTBot lets OpenAI collect pages it may use.

Does being in Common Crawl mean AI models use my content?

Not necessarily. AI builders filter Common Crawl heavily, and not every model uses it.

How do I check whether Common Crawl has my pages?

Search for your domain in Common Crawl's URL index at index.commoncrawl.org, choosing a recent archive.

Will allowing training crawlers help me appear in AI search?

Not directly. AI search depends on search crawlers and indexes; training crawlers affect what future models know.