Every AI company asks the open web for machine access. This page records, per crawler identity, what each major AI vendor's own website says in robots.txt and what its server does when that crawler requests the homepage, including the vendors' own crawlers and their competitors'. The full dataset, regenerated by a script with no manual edits, lives at /crawler-matrix.json.
Reading the table
First token: robots.txt verdict for that bot (— means no rule names it). Second token: live HTTP
status returned to that bot's exact user-agent. "challenge" is a Cloudflare managed-challenge
response (CF-Mitigated: challenge). The two can disagree: a site can permit a bot in
policy and refuse it at the network layer, or stay silent in robots.txt while blocking in practice.
| Vendor site | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Google-Extended | PerplexityBot | CCBot |
|---|---|---|---|---|---|---|---|---|
| openai.com | — / 403 (challenge) | — / 403 (challenge) | — / 403 (challenge) | — / 403 (challenge) | — / 403 (challenge) | — / 403 (challenge) | — / 403 (challenge) | — / 403 (challenge) |
| anthropic.com | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ |
| google.com | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ |
| perplexity.ai | — / 403 (cf-403) | — / 403 (challenge) | — / 200 ✓ | — / 403 (cf-403) | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 403 (challenge) |
| mistral.ai | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ |
| cohere.com | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ | — / 200 ✓ |
| meta.ai | — / 403 | — / 403 | — / 403 | — / 403 | — / 403 | — / 403 | — / 403 | — / 403 |
| x.ai | — / 200 ✓ | — / 200 ✓ | — / 403 (cf-403) | — / 403 (cf-403) | — / 403 (cf-403) | — / 403 (cf-403) | — / 403 (cf-403) | — / 200 ✓ |
Observations from this scan
- openai.com returned 200 to GPTBot, and 403 with a Cloudflare challenge to ChatGPT-User, the fetcher ChatGPT uses when a user asks it to read a page.
- x.ai returned 200 to GPTBot and 403 to ClaudeBot.
- perplexity.ai returned 403 with a challenge to every checked crawler, including PerplexityBot, its own crawler.
- meta.ai returned 403 to all eight crawlers with no robots.txt rules naming any of them: refusal at the network layer only.
- anthropic.com, google.com, mistral.ai and cohere.com returned 200 to all eight, with no robots.txt rules naming any of them.
Method
- robots: fetch /robots.txt as ourselves, parse the User-agent group naming the bot; block-all = Disallow: / in that group, restricted = other rules, no-rule = no group names the bot
- http: fetch / with the bot's exact user-agent; record status plus Cloudflare challenge markers (CF-Mitigated: challenge)
Vantage: TestedActually build machine, datacenter IP, 2026-10-30. Crawler behaviour can vary by IP reputation and path; we publish the vantage so results can be independently reproduced. Limitations: homepage only, eight vendors, eight crawler identities, one vantage point, no authentication-bearing fetchers. Week-over-week diffs are kept in the public repository history.
Reproduce
The scanner is a single script:
node scripts/vendor-crawler-matrix.mjs in the site's repository.
It rewrites /crawler-matrix.json from live requests; no manual edits.