AI coding assistants · Original data
AI crawler access test: can answer engines read these vendors?
Updated 2026-09-27 · markdown mirror
Verdict
8 of 8 tool sites reachable from our build machine on 2026-09-27. 2 of 8 restrict any checked AI crawler in robots.txt. 5 of 8 ship an /llms.txt (claude-code, cursor, github-copilot, gemini-cli, kiro). Median homepage TTFB: kiro fastest at 21ms, opencode slowest at 542ms.
| Tool | TTFB ms (median) | HTML KB | /llms.txt | AI crawler policy | Blocked bots |
|---|---|---|---|---|---|
| claude-code | 98 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 320.8 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | yes Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | open Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | none Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| codex | 68 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | — Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | no Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | open Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | none Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| cursor | 73 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 617.8 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | yes Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | open Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | none Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| github-copilot | 31 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 562.7 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | yes Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | partial Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | Bytespider Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| opencode | 542 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 65.3 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | no Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | open Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | none Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| gemini-cli | 29 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 562.7 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | yes Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | partial Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | Bytespider Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| kilo | 371 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 482 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | no Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | open Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | none Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
| kiro | 21 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | 281.6 Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | yes Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | open Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page | none Tested by us Checked 2026-09-27 /ai-coding-assistants/benchmarks/ai-coding-assistants-ai-crawler-access/ Our benchmark (n=8 tools × 3 runs) — methodology on the benchmark page |
● tested by us · ● documented at the source · ● unconfirmed, queued in /confirm.txt — hover or tap a dot for the source and date
Methodology
- Fetch https://<domain>/robots.txt with a plain HTTP GET and a desktop user agent.
- Parse User-agent groups line by line; a bot is blocked only if a group naming it (or the * wildcard) contains a 'Disallow: /' rule.
- Bots checked: GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider.
- Policy: blocked = GPTBot, ClaudeBot, and CCBot all disallowed from /; partial = at least one checked bot disallowed; open = no restrictions; unknown = robots.txt unreachable.
- Fetch https://<domain>/llms.txt; llmsTxt = HTTP 200.
- Fetch the homepage 3 times; TTFB is time to response headers; report the median. HTML size is the decoded body length of the last successful fetch.
Limitations
- Single machine, single network location — TTFB is directional, not global CDN truth.
- n=3 runs per tool; we report the median, not a distribution.
- Simplified robots.txt parsing: bot names matched case-insensitively, only Disallow rules read, path wildcards beyond 'Disallow: /' not modeled.
- Homepage only; vendor pricing pages may have different crawler policies.
- Snapshot of 2026-09-27; vendors change crawler policies without notice.
Script: scripts/crawler-test.mjs · Run 2026-09-27 · Reproduce with npm run bench