What AI crawlers actually read on your site (and what they skip)

The most common GEO mistake isn’t bad content; it’s content that the crawler never sees. GPTBot, ClaudeBot, PerplexityBot, Meta, and ByteDance bots execute zero JavaScript. Client-rendered content is invisible to them.

Only Gemini and AppleBot render JavaScript. Every other major AI crawler reads the raw HTML the server returns, and nothing more. (Crawlability is one half of the picture; for why being read still isn’t being cited, see what GEO is.)

What AI crawlers actually read

When an AI crawler visits your page, it receives the HTTP response body and stops. It does not:

• Execute JavaScript
• Wait for component hydration
• Trigger lazy-loaded sections
• Scroll to reveal deferred content
• Click, hover, or interact

A React or Next.js app that renders its core content client-side will return a near-empty HTML shell to the crawler. A methodology page that appears fully rendered in a browser may appear blank to GPTBot. An interactive comparison tool built as a client-side component is, from the crawler’s perspective, a div.

The old world vs. the new world

The web was built for browsers. JavaScript, animations, and progressive enhancement made sense when the reader was human: they’d wait, they’d scroll, they’d interact.

AI agents don’t work that way. The crawler fetches once, reads the response, and moves on. Content that exists only after JavaScript executes isn’t hidden; it’s absent.

This is a structural shift. The question “does this look good in a browser?” is not the same question as “is this readable by the agents that feed AI models?” The first is about human UX. The second is about what the training and retrieval bots find when they arrive.

How to verify what a crawler sees

The simplest check: fetch your own page without JavaScript.

curl https://yourdomain.com/your-page

What you get back is what GPTBot sees. If your core content (the methodology, the comparison table, the answer to the question the page is named for) is not in that output, it is not being read by the no-JS crawlers.

For more precision, check which user-agents your robots.txt allows and whether the specific retrieval bots are permitted. Blocking a training bot (GPTBot) is different from blocking a retrieval bot (OAI-SearchBot, ChatGPT-User). The consequences are different.

What to do about it

• Server-render your core content. The answer the page is named for, the claim that makes the page worth citing, that content needs to be in the initial HTML response.
• Check your framework’s rendering mode. Static generation or server-side rendering passes the content in the HTTP response. Client-side rendering does not.
• Don’t confuse Google-Extended with AI Overviews. Blocking Google-Extended in robots.txt blocks Gemini/Vertex grounding, but AI Overviews run on normal Googlebot and are unaffected. This is a common misconfiguration, covered alongside the other myths in schema markup for AI visibility.

The goal is not to remove JavaScript; it’s to make sure your citable content doesn’t depend on it.

See whether AI assistants can actually read your key pages today: run a free scan; it takes about 90 seconds.

For agents: try this yourself

Check what a no-JavaScript crawler sees on your own site:

• Run curl https://yourdomain.com/your-most-important-page and read the output. Is your core answer in the raw HTML, or only in a near-empty shell?
• “Which major AI crawlers execute JavaScript, and which read only the raw HTML response?”
• “Does blocking Google-Extended in robots.txt remove my site from Google’s AI Overviews?” Verify the answer against the trap described above.

---

Drawn from Collimer’s cited research library and findings. Crawler behavior is documented by vendors and corroborated by network studies; we update this guide when it changes. As of June 2026.