How to Audit Your Website for AI Visibility Gaps From Its URL

What can a URL audit tell you, and what can't it?

It helps to separate the two, because many "free AI audits" only cover the first half and present it as the whole picture.

A site can pass every check on the left and still be invisible, because engines often rely on third-party lists, reviews and directories. Treat the URL audit as the part you can fix quickly, then measure the rest; see how to manually track AI visibility.

Layer 1: can AI crawlers reach your pages?

Start here, because the other fixes pay off much less if the pages cannot be fetched.

Read your robots.txt. Open yoursite.com/robots.txt and look for groups that name AI crawlers, such as OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and Claude-SearchBot, and PerplexityBot, and for a catch-all User-agent: * group with a broad Disallow. OpenAI and Anthropic each run separate crawlers for training, for their search index and for fetching a page when a user asks; Perplexity runs one for its search index and one for user requests. Blocking one does not block the others. Decide per purpose rather than blocking "AI" as a whole; see whether businesses should allow or block AI crawlers.

A common mix-up concerns Google. Google-Extended is only a robots.txt token: it controls whether content Google crawls may be used to train future Gemini models and to ground answers in the Gemini app and in Grounding with Google Search on Vertex AI. Google says it does not affect a site's inclusion in Google Search. AI Overviews are part of Search, so blocking Google-Extended does not remove you from them, but it can keep your content out of grounded answers in the Gemini app.

Test for firewall and CDN blocks. robots.txt is a request; firewalls and bot protection are enforcement, and they often block AI crawlers without anyone deciding to. A quick check from a terminal:

curl -s -o /dev/null -w "%{http_code}" -A "GPTBot" https://yoursite.com/

A 200 means the page was served to that user agent; 403, 429 or a challenge page means something is blocking it. This only tests rules based on the user-agent string: some protections check the requester's IP address, so a pass here does not prove the real crawler gets through. Server logs are the definitive check; see how to see AI bot traffic in your server logs.

Check that key pages are indexable. Look for noindex, nosnippet or max-snippet directives in the page source or HTTP headers of your important pages. They can keep content out of search results and AI answers built on them. If you have Search Console access, also check the Search generative AI setting, which can exclude the whole site from Google's AI Overviews and AI Mode.

Layer 2: can AI read and understand your pages?

Test without JavaScript. Many AI crawlers do not run JavaScript the way a browser does. Fetch a page with curl or disable JavaScript in your browser and check that the main text, prices and key facts are in the raw HTML. Content that only appears after scripts run, in tabs that load on click, or inside images and PDFs is often invisible to them; see how to make JavaScript-rendered pages readable to AI crawlers.

Check the "who are you" signals. Read your homepage and About page as if you had never heard of the business. Within a few sentences, it should be clear:

• what you sell or do, in the plain words customers use;
• who it is for (business size, industry, type of customer);
• where you operate, if location matters;
• what makes you different, stated as facts, not slogans.

Engines describe businesses using the words that appear most clearly and consistently. A homepage that only says "Empowering growth through innovation" gives them nothing to repeat. See what an About page should say.

Check consistency. Your business name, address, phone number, category and main services should match across your site, your Google Business Profile, major directories and social profiles. Conflicting details are a common reason engines mix up businesses or give wrong contact information.

Check structured data. Organization or LocalBusiness markup that matches the visible page helps machines connect your pages to one business. Markup that contradicts the page does the opposite.

How do you audit existing content for AI search readiness?

This is the layer most audits skip, and the one that most often explains why a business is never recommended: the site has no page that answers the question buyers ask.

1. List the questions. Write 20–30 questions buyers ask before choosing a business like yours: category, requirement, comparison, problem and local questions. Use sales calls, support tickets and review text for the wording.
2. Map each question to a page. For each one, find the single page on your site that answers it best. Use site:yoursite.com plus the key words in a search engine if the site is large.
3. Grade each match. A page answers a question well when the first paragraph gives the answer directly, the page includes the specifics a buyer needs (prices or price ranges, requirements, limits, locations, timelines), and the heading uses words close to the question.
4. Mark the gaps. Questions with no page, or only a weak match such as a homepage or a generic service page, are your content gaps.

Check comparison and "alternatives" questions closely, especially if your site avoids naming competitors. Engines answer those questions anyway, using someone else's page. See which website pages to optimize for AI visibility.

Layer 4: is anything on your site wrong or out of date?

Engines repeat what they find, including your old pages. Look for:

• old pricing pages, promotions and plan names still live or still linked;
• service pages for services you no longer offer;
• old addresses, phone numbers or opening hours in footers, PDFs and blog posts;
• two pages that give different answers to the same question.

Retire or redirect pages that are wrong, and make the current fact easy to find on one clear page. Wrong facts are worth fixing before anything else, because they affect every question in which you are mentioned.

Layer 5: what does the URL not show you?

Finish by writing down what the audit could not check, so nobody mistakes a clean site audit for good AI visibility:

• whether engines name you for your buyer questions, and who they name instead;
• which review sites, directories and lists they cite for your category, and whether you are on them;
• what engines currently say about you.

These need you to run the questions in the engines and read the answers. Most URL-only audit tools cannot see them at all; see whether an AI visibility audit needs access to your website or analytics.

How do you turn the audit into a to-do list?

Rank the findings in this order, because each layer depends on the one before it:

1. Access blockers, such as blocked crawlers, error codes and noindex on key pages. Fix these first, since nothing else counts until they are gone.
2. Wrong or outdated facts on your own site.
3. Missing pages for high-value buyer questions, especially comparison and requirement questions.
4. Weak pages that exist but do not answer the question directly.
5. Clarity and consistency issues on the homepage, About page and profiles.

Then rerun your buyer questions in the engines a few weeks after the fixes to see what changed.

What do people ask about URL-based AI audits?

Are free "enter your URL" AI audits worth running?

They are a reasonable starting point for access and on-page checks. Some only check access and on-page signals; others also run a handful of prompts and show who is named, so check which kind you are using, and treat a few one-off prompts as a hint rather than a measurement. Use them for layer 1 and 2 findings and check the rest yourself.

Should I block AI crawlers to protect my content?

That depends on what you want. Blocking training crawlers and blocking search or user-triggered crawlers have different effects on whether you can be cited. Decide per crawler and purpose rather than blocking everything.

Does adding schema markup fix AI visibility?

It helps engines connect your pages and facts to one business, but it does not replace clear visible content or third-party presence. Markup that disagrees with the page can hurt more than it helps.

How often should I repeat a site audit?

After any redesign, migration, CMS change or security change, and otherwise every few months. Firewall and bot-protection settings change quietly, and a single setting can block AI crawlers across the whole site.