Why Can a Website Migration With 301 Redirects Reduce AI Visibility Temporarily?

What causes the dip?

• Recrawling takes time. Each system must revisit old URLs, see the redirect, fetch the new page and update its index. Large sites and rarely crawled pages take longest.
• Indexes update independently. A page can be current in Google, still old in Bing, and unknown to another index.
• Model memory is out of date. Answers from training knowledge cite the old URLs. The redirect catches the click, but the model's picture of your site does not change until the next model.
• Content changed. Rewritten pages, merged sections, removed FAQs and new templates that load text by JavaScript change what can be quoted. Most AI crawlers do not run JavaScript.
• Signals were lost. Internal links, breadcrumbs, structured data, canonical tags and hreflang are often rebuilt imperfectly.
• Access broke. Staging rules, bot protection or a new CDN default can block AI crawlers on the live site.
• Redirects were imperfect. Chains, loops, redirects to the homepage and missing mappings all lose value. Google warns that redirecting many old URLs to one irrelevant page can be treated as a soft 404.

Google's site move guidance also recommends keeping redirects as long as possible, and generally for at least a year. How redirects, canonicals and country domains affect AI visibility more broadly is covered in how redirects, canonicals and ccTLDs affect AI search visibility.

How to find blanket AI bot blocks left behind after a site migration

Leftover blocks are the most damaging and easiest-to-miss migration error. Check all of these on the live site:

1. robots.txt. Look for User-agent: * followed by Disallow: /, copied from staging, and for groups naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User or PerplexityBot. Blocking training crawlers may be deliberate. Blocking search and user-triggered crawlers makes you much less likely to appear in answers.
2. Robots meta tags and headers. Check page source for <meta name="robots" content="noindex"> and response headers for X-Robots-Tag: noindex. CMS settings such as WordPress's "Discourage search engines" option add these.
3. CDN and firewall settings. Cloudflare's AI bot controls, bot fight modes and WAF rules can block or challenge crawlers regardless of robots.txt. Cloudflare's defaults for new domains have changed several times, and a migration often means a new zone with new defaults.
4. Password or IP protection left on from staging for some paths.
5. Test each crawler. Fetch key pages with each AI user agent and check for a 200 response with real content, not a 403 or a challenge page. Then confirm in your server or CDN logs that real crawler requests succeed.

Files such as ai.txt or ai-plugin.json are not what to look for. No major AI search engine documents using ai.txt, and ChatGPT plugins, which used ai-plugin.json, were retired in 2024. Their absence is not an error.

How do you make a website migration and its content easier for AI to read?

Before launch:

• Map every URL one-to-one. Export all old URLs from the sitemap, analytics, logs and backlink tools, and map each to its closest new equivalent.
• Freeze content during the move where you can, so you are changing one thing at a time.
• Keep text in the HTML. Test new templates without JavaScript to confirm headings, body text, prices and specifications are present.
• Carry over structured data, canonical tags, hreflang, breadcrumbs and alt text.

At launch:

• Deploy redirects, test a sample of each page type, and check for chains.
• Publish a sitemap of new URLs and submit it in Google Search Console and Bing Webmaster Tools. Use IndexNow to notify Bing and other participating engines of changed URLs.
• Use Google's Change of Address tool if the domain changed.
• Update internal links to point to new URLs directly, not through redirects.

After launch:

• Update external references you control: business profiles, social profiles, marketplace and partner listings, and your most-linked pages' main referrers.
• Watch logs for AI crawler requests to old and new URLs, and for errors.
• Track AI answers for your key questions weekly for two months, noting which URLs are cited.

Can Claude and ChatGPT find my new ecommerce URLs before Google has fully indexed the migration?

Yes, sometimes sooner, sometimes later. They do not depend on Google's index:

• ChatGPT's search uses OpenAI's own crawler and third-party search providers. Being indexed in Bing helps, and IndexNow notifies Bing of changes quickly.
• Claude's web search draws on Brave's index, according to Anthropic's list of subprocessors. Brave's crawler discovers pages partly through opted-in users' browsing, so it may take time to find the new URLs.
• Perplexity runs its own index and fetches pages live.
• User-triggered fetches, such as ChatGPT-User and Claude-User, retrieve pages at the moment a user asks. If an answer cites an old URL, the fetch follows your redirect to the new page. That is one reason one-to-one redirects matter so much.

So new URLs can appear in ChatGPT or Claude answers within days, or lag for weeks, depending on each index. Old URLs will keep appearing for months, which is fine as long as they redirect correctly.

Digital merchandising managers auditing AI search visibility after an ecommerce taxonomy migration

A taxonomy change, such as merging categories, renaming them or restructuring navigation, affects AI answers differently from a simple URL move. Category pages are often what answers cite for "best [product type]" questions. When "Trail running shoes" becomes part of "Running footwear", the page that answered those questions may disappear.

Audit in this order:

1. Map old categories to new ones, and check that every old category URL redirects to the closest new category or filtered view, not to the homepage.
2. Check that specific category intent survived. If a specific category was merged into a broad one, consider keeping a dedicated page or filter URL for it, with its own heading and buying guidance.
3. Carry over category content: buying guides, size charts and FAQs that answers quoted.
4. Check product feeds. Update product_type and category mappings in Merchant Center and in the product feeds you send to AI shopping systems so they match the new taxonomy.
5. Check faceted URLs. New filter parameters may create crawlable duplicates or block previously indexed filtered pages.
6. Run your category questions in AI assistants before and after, and record which of your pages are cited.
7. Watch AI-referred landing pages and 404s in analytics for the first two months.

How do I preserve AI search visibility when migrating from WordPress to Webflow?

The main risks are specific to how Webflow builds URLs and what WordPress plugins were doing for you:

• URL structure changes. Webflow's CMS pages follow the pattern /{collection-url}/{item-slug}, such as /blog/my-post. WordPress posts often sit at the root, such as /my-post/, or under date folders. Every post URL may change, so map each one.
• Redirects. Add 301 redirects in Webflow's site settings under Publishing. Test that they work on the custom domain after publishing.
• Staging. Webflow's staging subdomain can be excluded from indexing in the SEO settings. Make sure the setting does not affect the live domain, and check the live robots.txt after launch.
• Plugin features. Yoast or Rank Math may have produced structured data, breadcrumbs, canonical tags and sitemaps. Webflow generates a sitemap, but schema usually needs custom code. List what each plugin did and rebuild it.
• Content fidelity. Check that tables, FAQs, code blocks and embedded content came across as HTML text, not as images or scripts.
• Internal links in old posts often point to WordPress-style URLs. Update them in the CMS rather than relying on redirects.

How long should the recovery take?

For a clean migration, expect most search-based AI visibility to return within weeks to a couple of months, similar to Google's timeline. If citations have not recovered after two to three months, look again at access, content changes and redirect quality before assuming it is just lag. What to do when AI answers send visitors to broken URLs is covered in how AI-generated answers send visitors to 404 pages instead of the right content.

What might an illustrative case look like?

Consider an illustrative outdoor clothing retailer that moved to a new ecommerce platform and merged 40 categories into 22. Google traffic recovered within a month, but ChatGPT and Perplexity answers stopped citing its category pages. Logs showed the new CDN challenging OAI-SearchBot and PerplexityBot, and "waterproof hiking jackets" had been merged into "outerwear" with no dedicated page. The retailer allowed the crawlers and restored a waterproof hiking jackets page as a filtered view with its own heading and guide. It also updated category mappings in its feeds. Six weeks later, both assistants were citing its category pages again.

What do people ask about migrations and AI visibility?

Should we migrate content and design at the same time?

Preferably not. Changing one thing at a time makes problems easier to find and fix.

How long should we keep the redirects?

At least a year, as Google recommends, and ideally indefinitely for pages that still receive visits or citations.

Will AI answers keep showing old URLs?

For a while, especially from model knowledge. Correct redirects make that harmless.

Does changing domain make it worse?

It adds work, since every index must learn the new domain. Use Google's Change of Address tool and update every profile that links to you.