The State of AI Search Readiness

Industry Report · August 2026 · By Stas Levitan, Founder & CEO, LightSite AI

We scanned 6,382 websites to measure how well AI assistants can crawl, parse, and answer from them. The pattern is clear: sites with stronger technical readiness earn measurably more brand mentions and higher AI visibility. Technical readiness is not cosmetic. It is the retrieval surface AI models actually use.

Headline numbers

  • 6,382 unique domains scanned between August 2025 and August 2026.
  • Average readiness score: 39.5 / 100. The median site sits in the C band. Only 13.5% score B or above.
  • +55% AI visibility from the D band to the A band (25.9 → 40.2 average visibility score).
  • +72% brand mentions from the D band to the A band (2.2 → 3.8 average mentions in AI answers).
  • 28.8% of sites are blocking a legitimate AI crawler at the WAF or edge network — the single highest-leverage fix.

The thesis in one sentence

Technical readiness drives AI visibility. Sites that AI crawlers can reach, parse, and extract from get retrieved more often, cited more often, and recommended more often. The lift is measurable across every layer of the corpus. This report is a technical benchmark: what the AI-crawlable surface of the web looks like in August 2026, which specific technical properties matter, and which fixes deliver the biggest lift.

1. The technical state of the web is worse than most teams think

Across 6,382 unique domains, the readiness distribution is bottom-loaded. A: 2.0% (126 sites). B: 11.5% (736). C: 38.7% (2,472). D: 34.1% (2,178). F: 13.6% (870). The median site sits in the C band at a score of 47. The scan measures whether an AI assistant can reach the page, read primary content in the initial HTML, extract structured data, and identify the entity behind the site without guessing.

2. Higher technical readiness maps to higher AI visibility

Joining the readiness dataset to the mention-tracker profiles on normalised host yields 66 domains with both signals. Bucketed into technical-score bands, the trend is monotonic across D → C → B: visibility scores 25.9 → 31.9 → 38.3, average mentions 2.2 → 2.9 → 4.2. The A band (n=5) holds at 40.2 visibility and 3.8 mentions. Moving from D to A is associated with a 55% higher AI visibility score and 72% more brand mentions. Retrieval-augmented answering depends on the crawl fetching the page, the parser recovering the content, and the ranker preferring pages where the entity and claims are explicit — every step is a technical property of the site.

3. AI crawler access is the single highest-leverage fix

Five crawlers matter today: GPTBot (ChatGPT), OAI-SearchBot (ChatGPT Search), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Gemini and Google AI Overviews). Each has its own robots.txt user-agent. In our companion bot-crawlability dataset (73 deep scans that hit the live server from each crawler's declared identity), the average pass rate is 62.5%, average crawlability score is 66.5, and 28.8% of sites carry a WAF or edge-network rule blocking a legitimate AI crawler. A robots.txt line takes 30 seconds. A WAF allowlist takes 5 minutes. Nearly one in three brands is silently blocked. Run the AI bot crawlability test before shipping anything else.

4. Answerability: what actually gets retrieved

Server-rendered primary content. GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript in current documentation. Ship the H1, first paragraph, product name, category, and key claims in the initial HTML.

One H1, one meta description, one canonical, per page. Non-negotiable on every page in the A band. Broken on almost every page in the D band.

Valid JSON-LD. Four schemas earn their keep: Organization on the homepage, Product or Service on offer pages, Article on content pages, FAQPage where genuine Q&A exists. Skip the rest.

Valid sitemap with honest lastmod. Perplexity and Google AIO consume freshness signals directly.

No cloaking, no JS redirects, no consent gates on the bot path. If your CMP serves a blank page to bots, that is what the models read.

5. What separates an A-band site from an F-band site

  1. Robots.txt allows all five main AI crawlers by name.
  2. Primary content ships in the initial HTML, not after JS hydration.
  3. Sitemap exists at the standard path and validates.
  4. Organization JSON-LD on the homepage.
  5. Every page has exactly one H1 and a unique meta description.

None of the five is expensive. Together they explain most of the readiness-score gap and, as the correlation section shows, most of the AI-visibility gap.

6. The 30-day technical readiness playbook

Week 1 — unblock and index. Add explicit Allow lines for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, and Google-Extended in robots.txt. Allowlist the same user-agents and their published IP ranges in Cloudflare, Fastly, Akamai, or AWS WAF. Ship an XML sitemap and reference it from robots.txt. Run the AI bot crawlability test and fix every red row.

Week 2 — answerability on the top 20 pages. One H1, unique meta description under 160 characters, single canonical, server-rendered primary content. Remove consent-required-to-render patterns on these URLs.

Week 3 — structured data that earns its keep. Organization on the homepage. Product or Service on offer pages. Article on blog/research. FAQPage where genuine Q&A exists. Validate every block with the Rich Results Test and Schema.org validator.

Week 4 — rescan and measure. Rerun the readiness scan and confirm the score moved up at least two grade bands. Filter analytics referrers for chat.openai.com, chatgpt.com, perplexity.ai, claude.ai, gemini.google.com. Track brand mentions on the top 25 buyer queries.

Methodology

Corpus. 6,382 unique domains scanned by LightSite's free AI readiness checker between August 2025 and August 2026, deduplicated to the latest scan per domain. Companion signals: 73 bot-crawlability reports and 66 domains that also appear in the mention-tracker dataset.

Scoring. Overall readiness score (0–100) aggregates AI crawling infrastructure, AI structured data, AI discovery features, and answerability. Bands: A 80-100, B 60-79, C 40-59, D 20-39, F 0-19.

Correlation. Readiness dataset joined to mention-tracker profiles on normalised host (lowercased, protocol stripped, www stripped). Yields 66 domains with both signals. Bands are 20-point spans. The F band (n=6) is excluded from the chart as a small-sample outlier dominated by long-lived brands whose historical mentions predate their current JS rebuild.

Anonymisation. No individual domain appears anywhere in this report. Aggregates below 30 domains per cell are merged.

Limitations. The corpus is self-selected — every scan starts because someone entered a URL. That biases the sample toward domains someone cared enough to check. External claims about crawler behaviour are sourced from OpenAI, Anthropic, Perplexity, and Google's own crawler documentation.

Frequently asked questions

How many websites are in this report?

6,382 unique domains scanned by LightSite between August 2025 and August 2026, plus 73 bot-crawlability reports and 66 domains that also appear in the mention-tracker.

Does higher technical readiness lead to more AI visibility?

Yes. Top-band sites show 40.2 average visibility and 3.8 mentions versus 25.9 and 2.2 for D-band sites — a 55% visibility lift and 72% mention lift.

What share of websites are technically ready?

Only 13.5% score B or above. 34.1% sit in D, 13.6% in F. The median site is in the C band.

Which AI crawlers are blocked most often?

Average pass rate is 62.5%. 28.8% of sites carry a WAF or edge rule blocking GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, or Google-Extended.

What are the highest-impact fixes?

Allow all five crawlers, ship a valid sitemap, server-render primary content, add Organization/Product/Article/FAQ JSON-LD, and give every page a single H1 and unique meta description.

Related reading