ResearchAI Crawler Readability Statistics

AI Crawler Readability Statistics 2026: 240 Sites Analyzed

240 B2B homepages tested for AI crawlers: 50 (21%) are empty without JavaScript and 20 (8%) publish llms.txt.

Data last changed Sample: 240Markdown version

Key findings

  1. 50 of 240 B2B homepages (21%) were close to empty without JavaScript: less than 20% of their text was in the page before scripts ran.

    Link to this finding

  2. Fintech sites are more likely than B2B SaaS sites to render empty without JavaScript: 35% vs 13%. Empty homepages by category: B2B SaaS 12 of 96 (13%), fintech 25 of 72 (35%) and cybersecurity 13 of 72 (18%).

    Link to this finding

  3. 40 of 240 B2B sites (17%) blocked at least one of 4 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and Google-Extended) in robots.txt. GPTBot was blocked most often, by 33 of 240 sites (14%).

    Link to this finding

  4. 20 of 240 B2B sites (8%) published an llms.txt file, against 210 (88%) with an XML sitemap.

    Link to this finding

  5. 131 of 240 B2B homepages (55%) carried Organization JSON-LD and 43 (18%) carried FAQPage JSON-LD.

    Link to this finding

  6. Seed and Series A sites are more likely than growth-stage and public company sites to render empty without JavaScript: 22% vs 9% (20 of 93 vs 4 of 47 homepages).

    Link to this finding

By category

CategorySitesShare of sitesEmpty without JavaScriptBlocks an AI crawlerllms.txtXML sitemapOrganization JSON-LDFAQPage JSON-LD
B2B SaaS9640%12 of 96 (13%)7 of 96 (7%)9 of 96 (9%)85 of 96 (89%)51 of 96 (53%)16 of 96 (17%)
Fintech7230%25 of 72 (35%)17 of 72 (24%)7 of 72 (10%)63 of 72 (88%)46 of 72 (64%)16 of 72 (22%)
Cybersecurity7230%13 of 72 (18%)16 of 72 (22%)4 of 72 (6%)62 of 72 (86%)34 of 72 (47%)11 of 72 (15%)

Share of sites uses all 240 sites; every other share uses the sites in that category.

By AI crawler

CrawlerSites blocking it in robots.txt
GPTBot33 of 240 (14%)
ClaudeBot26 of 240 (11%)
PerplexityBot21 of 240 (9%)
Google-Extended24 of 240 (10%)

Each share uses all 240 sites. A site can block more than one crawler, so the rows do not add up.

By funding stage

Funding stageSitesShare of sitesEmpty without JavaScriptBlocks an AI crawlerllms.txt
Seed3615%6 of 36 (17%)6 of 36 (17%)4 of 36 (11%)
Series A5724%14 of 57 (25%)8 of 57 (14%)5 of 57 (9%)
Series B5021%13 of 50 (26%)9 of 50 (18%)3 of 50 (6%)
Series C5021%13 of 50 (26%)9 of 50 (18%)6 of 50 (12%)
Growth2711%0 of 27 (0%)3 of 27 (11%)2 of 27 (7%)
Public208%4 of 20 (20%)5 of 20 (25%)0 of 20 (0%)

Share of sites uses all 240 sites; every other share uses the sites at that stage.

Homepage text visible without JavaScript

Text visible without JavaScriptSitesShare of sites
Under 20% (empty page)5021%
20% to under 50%2410%
50% to under 80%8335%
80% to 100%8335%

Bands do not overlap: each includes its lower bound and excludes its upper bound (the top band includes 100%). Share of sites uses all 240 sites.

FAQ

  • How many B2B websites look empty to AI crawlers that do not run JavaScript?

    50 of 240 B2B homepages (21%) were close to empty without JavaScript: less than 20% of their text was in the page before scripts ran. Source: AI crawler readability audit of B2B sites, May 2026.

    Link to this answer

  • What share of B2B websites block AI crawlers in robots.txt?

    40 of 240 B2B sites (17%) blocked at least one of 4 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and Google-Extended) in robots.txt. GPTBot was blocked most often, by 33 of 240 sites (14%). Source: AI crawler readability audit of B2B sites, May 2026.

    Link to this answer

  • How many B2B websites publish an llms.txt file?

    20 of 240 B2B sites (8%) published an llms.txt file, against 210 (88%) with an XML sitemap. Source: AI crawler readability audit of B2B sites, May 2026.

    Link to this answer

Methodology

  • Source: AI crawler readability audit of 240 B2B sites, May 2026.
  • Method: Each homepage fetched twice: once with JavaScript disabled and once in a headless browser; text share compared. robots.txt parsed for AI user agents. llms.txt, sitemap.xml and JSON-LD checked by direct fetch.
  • Coverage: 240 B2B homepages audited on weekdays from May 4, 2026 to May 22, 2026.
  • Definitions: a page is empty without JavaScript when less than 20% of its rendered text is present before scripts run. A site blocks a crawler when its robots.txt disallows that user agent.
  • Excluded: no rows (no duplicate sites). Only homepages were audited, so inner pages are not covered. robots.txt rules for user agents other than GPTBot, ClaudeBot, PerplexityBot and Google-Extended are not counted. Time to first byte was recorded but is not used in any finding.
  • Denominators: every share uses all audited sites unless the table or finding names a category or funding-stage group.
  • A finding is published only when every share in it has a denominator of at least 20 rows, and gaps under 3 percentage points are reported as about the same.
  • Refresh: a one-off audit with no scheduled re-run. The data on this page last changed on May 26, 2026.
  • Percentages are rounded to the nearest whole percent (halves round up) from exact counts, so a column of shares can add to 99% or 101%. Means are rounded to one decimal place. Medians of an even number of values are the mean of the two middle values. Gaps in percentage points are the difference between the two published whole-percent shares.
  • Self-reported fields: none. Funding stage comes from the dataset records; every other field was measured by the audit.

Dataset

  • Name: AI crawler readability audit of B2B sites, May 2026
  • Description: Whether 240 B2B homepages can be read by AI crawlers: JavaScript dependence, robots.txt rules for AI user agents, llms.txt, sitemaps and JSON-LD. Source: AI crawler readability audit of 240 B2B sites, May 2026.
  • Variables measured: Share of homepage text visible without JavaScript; robots.txt blocks for GPTBot, ClaudeBot, PerplexityBot and Google-Extended; llms.txt present; XML sitemap present; Organization JSON-LD present; FAQPage JSON-LD present
  • Measurement technique: Each homepage fetched twice: once with JavaScript disabled and once in a headless browser; text share compared. robots.txt parsed for AI user agents. llms.txt, sitemap.xml and JSON-LD checked by direct fetch.
  • Markdown version: https://www.theaeocompany.co/research/crawler-audit.md

Suggested citation

The AEO Company. "AI Crawler Readability Statistics 2026: 240 Sites Analyzed." May 2026. https://www.theaeocompany.co/research/crawler-audit

URL: https://www.theaeocompany.co/research/crawler-auditMonth: May 2026