Googlebot has rendered JavaScript for years, so most teams assume every crawler does the same. The bots that feed ChatGPT, Perplexity, and Claude do not. A widely cited analysis by Vercel and MERJ of more than a billion crawler requests found that GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot fetch your raw HTML and never execute the JavaScript that builds a client-rendered page. GPTBot downloaded JavaScript files in about 11.5% of its requests, then ran none of them. If your content only appears after React, Vue, or Angular hydrates in the browser, these crawlers see an empty shell.
That gap is expensive and growing. In the month Vercel measured, the major AI crawlers together reached about 1.3 billion requests, close to a third of Googlebot's volume. Every one of those visits reads HTML the way a browser did in 1998: whatever the server sends, nothing more. This post explains why AI crawlers see a different web, how to check what they receive, and the rendering changes that fix it.
Key Takeaways
- Major AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) parse raw HTML and do not execute JavaScript, so client-side-rendered content is invisible to them.
- Googlebot still renders JavaScript in a headless browser, which is why a site can rank in Google yet be absent from AI answers.
- You can see exactly what a bot sees by requesting the page with the bot's user-agent and reading the raw HTML, before any script runs.
- The durable fixes are server-side rendering, static generation, and shipping critical content and semantic markup in the initial HTML.
AI agents that operate a real browser are different again: many read the page's accessibility tree, so unlabeled buttons and links can stop them. Our guide to web accessibility and SEO explains why.
Why AI Crawlers See a Different Web Than Googlebot
Google documents a three-stage pipeline in its JavaScript SEO basics: crawl, render, then index. In the render stage, Googlebot loads your page in headless Chromium, executes the JavaScript, and indexes the resulting DOM. That is why a client-side-rendered app can still rank: Google ran your scripts and read the output.
AI crawlers skip the render stage entirely. OpenAI's crawler documentation describes GPTBot as a training crawler and OAI-SearchBot as the bot that surfaces sites in ChatGPT search, but neither runs a browser engine. As the Vercel study put it, the bots may download script files but "they don't execute them," so they cannot read client-side-rendered content. Perplexity's crawler documentation points the same way: PerplexityBot works from the HTML it is served. Apple's crawler renders as an exception, but assume that if a fact is not in your raw HTML, most answer engines never see it.
How to Check What Bots Actually See
You do not have to guess. A human sees the rendered HTML (the DOM after JavaScript runs); an AI crawler gets only the raw HTML the server returned, and you can pull that raw view in seconds.
- Request the page as the bot. From a terminal, run curl -A "OAI-SearchBot/1.0" https://yoursite.com/page, swapping in "GPTBot/1.4" or "PerplexityBot/1.0" to match each crawler. That is the same raw HTML those bots parse.
- Search that output for your real content. If your headline, price, or key answer is missing from the response, it is invisible to AI crawlers even though it renders fine in Chrome.
- Compare against the rendered page. "View Source" shows the raw HTML; the DevTools Elements panel shows the rendered DOM. A large gap means your content lives in JavaScript, not the document. Disabling JavaScript and reloading shows the same blank page a crawler sees.
Run this on your highest-value templates first, one curl each, before you spend a sprint on the wrong fix.
The Fixes: SSR, Static Generation, and Hydration Strategy
The cure is to send complete HTML from the server so the crawler never depends on a browser it lacks. Three mainstream approaches do it, all supported by modern frameworks.
- Server-side rendering (SSR). The server builds full HTML for each request and sends it ready to read. Next.js, Nuxt, SvelteKit, Remix, and Astro all support it. The bot gets real content on the first response; the page then hydrates into an interactive app for humans.
- Static site generation (SSG). Pages are pre-rendered to HTML at build time and served from a CDN. It is ideal for content that does not change per request, such as blog posts, docs, and marketing pages, and it is the fastest option for bots and people alike.
- Prerendering for a legacy SPA. If you cannot rebuild a client-rendered app quickly, a prerender service can serve static HTML snapshots to crawlers while humans keep the SPA. It is a bridge, not a destination, but it makes an invisible site visible.
Your hydration strategy is where teams quietly reintroduce the problem. Partial hydration and React Server Components keep most of the page as server HTML and hydrate only the interactive islands, which is what you want. The failure pattern is the opposite: rendering a bare shell on the server and fetching the real content client-side after load. To a crawler, that shell is the whole page. The rendering model is a major build decision, which is why we treat it as core to any website development company that renders server-side engagement, not an afterthought. How hard this is depends partly on your platform, a tradeoff we weigh in headless vs traditional CMS for SEO.
Put Critical Content in the Initial HTML
Rendering strategy decides how the HTML is produced; this decides what goes in it. The rule is simple: anything you need an AI engine to quote, cite, or act on must exist in the first HTML response, not arrive later over an API call. That means your headline and subheads, the direct answer to the page's core question, product names and prices, contact and location details, and FAQ content. Lazy-loading images below the fold is fine. Lazy-loading your actual answer is not.
Semantic markup then tells the parser what each block means. Use a single clear h1, logical heading levels, real paragraph and list elements, and descriptive link text instead of a wall of styled div tags. Add structured data so machines get your facts unambiguously, the same discipline covered in the schema markup every business website needs. Clean HTML plus valid schema turns a page a bot can read into one it can confidently reuse.
Verify Before and After You Ship
Rendering regressions are silent: a framework upgrade or a new client-side data fetch can move content out of the initial HTML without breaking anything a human notices. So verify continuously. Before release, curl each key template as a bot and confirm the important content is present, then periodically run your customers' real questions into ChatGPT, Perplexity, and Google's AI Mode to check whether you are cited accurately. Make it a standing release check, and note that faster time-to-content also lifts your Core Web Vitals in 2026.
AI crawlers will not start rendering JavaScript on your schedule, so the winning move is to stop depending on it for content that matters. Serve complete, semantic HTML, keep critical facts in the first response, and verify what the bots actually receive. If you want to see exactly what GPTBot, PerplexityBot, and ClaudeBot get on your most important pages, request a crawler visibility audit and we will map every page that is currently invisible.
Read More
- Headless vs Traditional CMS for SEO in 2026
- Core Web Vitals in 2026: What Still Moves Rankings
- The Schema Markup Every Business Website Needs
- The Technical SEO Audit Checklist for Growing Sites
Frequently Asked Questions
Do AI crawlers like GPTBot and PerplexityBot run JavaScript?
No. The Vercel and MERJ analysis of over a billion requests found that GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot fetch raw HTML and do not execute JavaScript, so client-side-rendered content is effectively invisible to them. Apple's crawler is an exception that does render, and bot behavior can change, so verify rather than assume.
Why does my site rank in Google but not appear in ChatGPT or Perplexity?
Because Googlebot renders JavaScript in a headless browser and most AI crawlers do not. Google's crawl, render, and index pipeline executes your scripts and indexes the output, so a client-rendered app can still rank. The AI crawler reads only the raw HTML, finds an empty shell, and moves on. The fix is to serve your content in the initial HTML response.
How do I check what an AI crawler sees on my page?
Request the page with the bot's user-agent, for example curl -A "OAI-SearchBot/1.0" followed by your URL, and read the raw HTML that comes back. If your headline, prices, or main answer are missing, the crawler cannot see them. Comparing "View Source" against the DevTools Elements panel shows the same gap.
Will fixing rendering for AI crawlers hurt my Google SEO or site speed?
It usually helps both. Sending complete HTML gives Googlebot content without waiting to render and improves time-to-content, which supports Core Web Vitals. You keep interactivity by hydrating on top of the server HTML, so humans get a fast experience while every crawler gets readable content.
What content absolutely must be in the initial HTML?
Anything you want an AI engine to quote or act on: your h1 and subheads, the direct answer to the page's main question, product names and prices, contact and location details, and FAQ content. Decorative or below-the-fold assets can load later, but core facts should never depend on a client-side fetch.
Want this handled for you? Our website development company can help you build a fast, SEO-ready site that converts.
Aashirvad Kumar
Writes about SEO, digital marketing and growth at Optimize For SEO.