How to Check Whether AI Can Read Your Pages: Screen Reader, curl and No-JavaScript Tests

What does "AI can read your page" actually require?

Five things have to be true:

1. The crawler is allowed by robots.txt and not blocked by a firewall, CDN rule or bot protection.
2. The response is real content, not a challenge page, an empty application shell or a consent wall, even if the status code is 200.
3. The key text is in the HTML the server sends, because many AI crawlers do not run JavaScript.
4. The main content can be separated from menus, banners and footers.
5. The structure survives conversion to text: headings, lists and tables keep their meaning.

Each test below checks one or more of these.

How do you use curl to check what AI crawlers can see?

curl fetches a page the way a simple crawler does: no JavaScript, no cookies, no clicks. Use the user agent strings that AI companies publish in their crawler documentation. For example, OpenAI documents a user agent for its search crawler along these lines:

UA='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'
curl -s -A "$UA" -o page.html -w '%{http_code} %{size_download} bytes\n' https://www.example.com/pricing
grep -c 'per user per month' page.html

What to look for:

• Status code: 200 is needed, but not sufficient.
• Size: compare with the size a normal browser user agent receives. A much smaller response suggests a block, challenge or shell.
• Key phrases: search the saved HTML for your product names, prices, service areas and the first sentence of your answer. If they are missing, the crawler is probably missing them too, unless your host serves pre-rendered pages only to verified crawlers; then check what the real crawler received in your server logs.

Repeat with other crawlers' documented user agents, such as PerplexityBot and Anthropic's Claude-SearchBot, and with a normal browser user agent for comparison.

One caution: some bot protection checks IP addresses as well as user agents, so a test from your own machine may be treated differently from the real crawler. Server logs are the final proof; see below.

How do you check AI visibility by disabling JavaScript in Chrome DevTools?

1. Open the page in Chrome and open DevTools.
2. Press Ctrl+Shift+P (Cmd+Shift+P on a Mac) to open the Command Menu.
3. Type "Disable JavaScript" and press Enter.
4. Reload the page, keeping DevTools open.

What remains is roughly what a non-rendering crawler receives. If prices, product details, reviews or whole sections disappear, they depend on JavaScript. You can also use "View page source" to see the raw HTML the server sent. Remember to re-enable JavaScript afterwards. For fixing script-dependent pages, see how to make JavaScript-rendered pages readable to AI crawlers.

Can an HTML 200 response still be inaccessible to AI search crawlers?

Yes. A 200 status only means the server returned something. Common "soft blocks" that return 200:

• Bot challenge pages that ask the visitor to prove they are human.
• Empty application shells that contain little more than a script tag until JavaScript runs.
• Consent or age walls that replace the content until a button is clicked.
• "Please enable JavaScript" messages served instead of content.
• Region or login walls that show a message rather than an error code.
• Rate-limit pages served with a normal status.

The curl size comparison and key-phrase search catch most of these. If the saved HTML contains a challenge script or a consent form but not your content, the crawler sees the same.

Can cookie banners and popups hide content from AI systems?

It depends on how they are built:

• Overlay banners whose HTML sits alongside the page content are usually harmless to crawlers: the content is still in the HTML.
• Consent walls that withhold content until consent is given, served from the server, can mean crawlers receive only the wall.
• Very large banners placed first in the HTML can crowd the start of the extracted text, and some extraction tools may treat them as the main content or cut the page short.
• Interstitial popups loaded by script generally do not affect crawlers that do not run scripts, but can affect those that render pages.

The same applies to engines that use rendered pages, such as Google's systems behind Gemini and AI Overviews: they see what a rendering crawler sees, which does not include clicking "accept". Test by fetching the page with curl and by loading it in a fresh private window, and check that product details appear in the HTML before any interaction. If they only appear after consent, a crawler may never see them.

Can AI read hidden accordion text in HTML?

If the text is in the HTML and merely hidden with CSS, or inside a details element, crawlers that parse HTML can read it. Google has said content hidden in tabs or accordions for usability is indexed normally. Microsoft is more cautious: it lists answers hidden in tabs or expandable menus as a mistake for AI search answers, so keep buying-decision answers visible. The problem cases are accordions whose content is fetched by JavaScript only when a user clicks, or built with scripts after the page loads. Those are invisible to non-rendering crawlers. Check by searching the raw HTML for text from a collapsed section.

Do dedicated pages outperform accordions for AI citations?

For important questions, a dedicated page often works better, although there is no public study that settles it. A dedicated page has its own title, URL, headings and opening answer, all focused on one question, which makes it a strong match for that question and easy to cite with a specific link. An accordion item is one small part of a page about something broader.

A practical rule: use accordions for short, factual answers such as opening hours, delivery times and returns policies. Use dedicated pages, or at least full sections with their own headings, for questions that drive buying decisions and need more than a paragraph.

How can a screen reader help you check AI readability?

Screen readers read a page in the order and structure the HTML provides, announcing headings, lists and table structure. That makes them a useful proxy for how an extraction tool will linearise your page:

• if the screen reader reads the cookie banner and menu for a minute before reaching your content, extractors may also struggle to find the main content;
• if headings are not announced as headings, your outline is not in the markup;
• if a table is read as a string of disconnected numbers, it has lost its labels.

Built-in options include VoiceOver on Mac and Narrator on Windows, and NVDA is a free option on Windows. Chrome DevTools also shows the accessibility tree, a quick way to see the structure without listening. Accessible pages and AI-readable pages share most of the same requirements.

How do you check whether AI can read and extract content from a page end to end?

After the technical tests, check the full chain:

1. Ask an AI assistant that can open links to read the URL and summarise the key facts. Compare its summary with the page.
2. Ask a question only that page answers, in an engine that searches the web, and see whether the page is cited.
3. Check server logs for recent requests from the AI crawlers you care about, with 200 responses and normal response sizes.

If the assistant cannot open the page, the problem is access. If it opens it but gets facts wrong, the problem is extraction or unclear content. If it summarises correctly but never cites the page for real questions, the problem lies further along: retrieval and selection. That later stage is covered in why high-quality content may not show up in AI answers.

How do you validate AI visibility fixes with curl and manual prompts?

Use the same checks before and after every fix:

• Before: save the curl output, size and key-phrase results for each crawler user agent; note log entries; run a small set of prompts and record whether the page is cited.
• Fix: change the robots.txt rule, firewall setting, rendering or banner behaviour.
• Immediately after: repeat the curl tests; they should show the content.
• Over the next one to three weeks: check logs for real crawler visits with good responses, then re-run the prompts several times.

Keep the evidence. It shows exactly when access was restored and whether citations followed.

How do you run the checks across many pages?

A short script can run the curl test for a list of URLs and several user agents, and save a table you can sort. A minimal version:

while read -r url; do
  for bot in browser oai perplexity; do
    case $bot in
      browser)    ua='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0 Safari/537.36';;
      oai)        ua='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot';;
      perplexity) ua='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)';;
    esac
    curl -s -A "$ua" -o /tmp/p.html -w "$url,$bot,%{http_code},%{size_download}\n" "$url"
  done
done < urls.txt > results.csv

Sort the results by size for each URL. Any crawler row far smaller than the browser row is worth opening. Use the exact user agent strings from each company's current documentation, as they change over time. Add a column for a key phrase check, so the table shows not just whether each crawler got a response but whether it got your content.

What do common failures look like?

What does a complete AI readability checklist look like?

What do people ask about testing AI readability?

Is Google's URL Inspection tool enough to check AI readability?

It shows what Googlebot sees, including rendered HTML, which covers Google's AI features. It does not show what non-rendering crawlers from other AI companies receive, so add the curl and no-JavaScript tests.

How often should these tests run?

After any change to templates, hosting, CDN or security settings, and as part of a monthly or quarterly technical check.

Can I test all pages at once?

Yes, with a script that fetches a list of URLs with each user agent and records status, size and key phrases. Start with templates and your most important pages.

What if the curl test passes but AI still ignores the page?

Then access is not the problem. Look at whether the page answers the questions people ask, and whether other sources engines trust mention you.