How to see your page the way a crawler that does not run JavaScript sees it

· Edited by Chris Dolan

To see your page the way a crawler that does not run JavaScript sees it, look at the HTML your server sends before any script runs. Open the page source, or fetch the page with a command-line tool, and search it for the words that matter. If they are missing from that first response, a crawler that does not run scripts never receives them. Every statement below was checked against the tool makers' own pages on 9 October 2026.

Two limits apply from the start. None of these checks imitates a particular crawler exactly: they show what any client receives when scripts do not run. And how far each AI crawler runs JavaScript is less well documented than most articles suggest.

Why View Source and Inspect show different pages

A browser shows you two versions of a page, and they are easy to mix up. Chrome's documentation on viewing the DOM puts the difference in one sentence: “HTML represents initial page content and the DOM represents active, current page content.” The first is what the server sent. The second is what exists after scripts have run and changed it.

Google's Search Console help makes the same distinction. Its page on viewing the rendered source says that showing the source “typically shows only the original page code returned to the browser”, before scripts and other resources have loaded. A crawler that does not run JavaScript gets that original code and nothing more. Our guide to what AI search shows about your business explains why the crawlers that fetch your pages are worth watching in the first place.

Check 1: the raw HTML, with View Source or curl

The quickest check needs no tools. In Chrome, right-click the page and choose View page source, then search for a sentence from the main text, a product name, or a price. If the search finds it, the server sent it.

A command-line fetch is stricter, because it involves no browser at all. The curl project's FAQ says of JavaScript in web pages: “curl and libcurl have no built-in support for that”, so what curl saves is the page as it stands before any script runs:

curl -s https://www.example.com/your-page -o page.html
curl -s -A "ExampleBot/1.0" https://www.example.com/your-page -o page-as-bot.html

The second line uses the option the curl manual describes this way: “Specify the User-Agent string to send to the HTTP server.” It lets you see whether your site sends something different when the visitor names itself as a bot. It changes the name only. A site that also treats visitors differently by address may still send the real crawler another page.

Check 2: turn JavaScript off in Chrome

Reading raw HTML shows what is present. Turning scripts off shows what the page looks like without them, which is easier to judge by eye. Chrome's guide to disabling JavaScript gives the steps.

  1. Open Chrome DevTools on the page.
  2. Open the Command Menu with Control, Shift and P on Windows or Linux, or Command, Shift and P on a Mac.
  3. Start typing javascript, choose Disable JavaScript and press Enter.
  4. Reload the page and see what remains.

The guide adds the detail that catches people out: “JavaScript will remain disabled in this tab so long as you have DevTools open.” Close DevTools and the scripts come back. A blank page, or a page with its navigation and no text, means the content is built in the browser.

Check 3: see what Google rendered

Google does run JavaScript, so its view is the other half of the comparison. The same Search Console help page lists two ways to see the rendered HTML as Google produces it.

  • For your own site. Inspect the address in Search Console, click Test live URL, then View tested page. The HTML tab shows the page after rendering.
  • For any public page. Open Google's Rich Results Test and enter the address.

If your content appears here and not in the raw HTML, the page depends on rendering. Passing this check says nothing about a crawler that does not render. The technical accessibility section of our audit checklist lists content that only appears after scripts run among the things to check.

Which crawlers run JavaScript, and how much is known

Google's own guide to JavaScript and search, last updated on 4 March 2026, says Googlebot renders pages with a headless Chromium browser. In the same passage it recommends rendering on the server anyway, because “not all bots can run JavaScript.”

For AI crawlers the best public evidence is one measurement. Vercel and MERJ studied crawler traffic on Vercel's network and reported on 17 December 2024: “none of the major AI crawlers currently render JavaScript”. Their list covered OpenAI's three agents, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. The same study found that Google's Gemini, which uses Googlebot's infrastructure, and Applebot did render.

That is a measurement from one network at one time, and it is nearly two years old. It is not a statement by the operators, so treat it as the likely case and check your own pages.

What to do when the content is missing

If the words that matter are absent from the raw HTML, the fix is to send them from the server. Google's page on dynamic rendering, last updated on 10 December 2025, recommends server-side rendering, static rendering or hydration, and gives the reason in plain terms: “Other search engines may choose to ignore JavaScript and won't see JavaScript-generated content.”

  • Start with the pages that carry facts. Product, pricing, service and contact pages matter most, because their content is what an answer would quote.
  • Do not build a separate version for bots. Google calls dynamic rendering a workaround and not a long-term solution.
  • Repeat the three checks afterwards. The raw HTML should now contain the text, the page should read sensibly with scripts off, and Google's rendered view should match.

Sources

All eight were read on 9 October 2026.

  1. Get started with viewing and changing the DOM, Chrome for Developers, last updated 1 March 2019.
  2. View the rendered source for a page, Google Search Console Help.
  3. curl: Frequently Asked Questions, the curl project.
  4. curl: How To Use (the manual page), the curl project.
  5. Disable JavaScript, Chrome for Developers, last updated 31 January 2019.
  6. Understand JavaScript SEO Basics, Google Search Central, last updated 4 March 2026.
  7. The rise of the AI crawler, Vercel with MERJ, 17 December 2024.
  8. Dynamic Rendering as a workaround, Google Search Central, last updated 10 December 2025.