Can AI bots actually reach your website?

The AI Bot Crawlability Test is a free 20-second probe that answers a question most marketing teams cannot answer on their own: can the AI bots behind ChatGPT, Claude, Perplexity, Google's AI Overviews, and Bing/Copilot actually reach your website — or are they being silently blocked by your CDN, WAF, or a stale robots.txt line no one has touched in three years?

Which bots we test

  • GPTBot (OpenAI — training crawler)
  • OAI-SearchBot (OpenAI — powers ChatGPT search results)
  • ChatGPT-User (OpenAI — fetches on behalf of a live ChatGPT user)
  • ClaudeBot (Anthropic)
  • PerplexityBot (Perplexity)
  • Google-Extended (Google's AI training crawler, separate from Googlebot)
  • Bingbot (Microsoft — also powers Copilot answers)
  • CCBot (Common Crawl — the corpus that trains most open models)

How the test works

You submit a URL. We fetch it fresh with each bot's canonical User-Agent string, pull your /robots.txt and parse it per bot using longest-match rule precedence, then classify every response — reachable, blocked by robots.txt, blocked by edge or WAF (Cloudflare, Akamai, Imperva, Sucuri, PerimeterX), forbidden, server error, or not found. A deterministic weighted score turns the raw results into a 0–100 number, and a language model writes a short plain-English summary and a prioritized fix list.

Why marketers rarely see these blocks

Most AI bot blocks come from network-layer security that treats non-browser user-agents as suspicious. That decision lives with your infrastructure, DevOps, or CDN team — not your CMS, not your marketing stack. This test surfaces the block in a form you can hand to that team: bot name, HTTP code, WAF signal, and the specific fix to request.

Frequently Asked Questions

Is the AI Bot Crawlability Test free?

Yes. Free, no signup, results in about 20 seconds. Each report is shareable via a public URL.

Is this the same as a real AI crawl?

No. We probe from a generic egress IP with the correct User-Agent. If your edge blocks by vendor ASN or IP range, real bots may still fail even when we succeed. Treat this as a fast diagnostic and verify with your CDN team.

What if robots.txt disallows a bot?

We flag it — most marketing teams didn't know a stale default robots.txt was blocking Google-Extended, GPTBot, or CCBot. Fix is a one-line change.

Related Resources