# AI crawler access test: can answer engines read these vendors?

> 6 of 6 tool sites reachable from our build machine on 2026-09-27. 0 of 6 restrict any checked AI crawler in robots.txt. 5 of 6 ship an /llms.txt (coderabbit, greptile, qodo, ellipsis, superpositional). Median homepage TTFB: superpositional fastest at 18ms, ellipsis slowest at 251ms.

Human page: https://testedactually.com/agentic-code-review/benchmarks/agentic-code-review-ai-crawler-access/
Mirror: https://testedactually.com/agentic-code-review/benchmarks/agentic-code-review-ai-crawler-access.md
Updated: 2026-09-27

Methodology: https://testedactually.com/agentic-code-review/benchmarks/agentic-code-review-ai-crawler-access/ — script: `scripts/crawler-test.mjs`, n=6 tools × 3 runs, run 2026-09-27.

**AI crawler access test: can answer engines read these vendors? — n=6 tools, 3 runs per tool**

| Tool | TTFB ms (median) | HTML KB | /llms.txt | AI crawler policy | Blocked bots |
| --- | --- | --- | --- | --- | --- |
| [coderabbit](/agentic-code-review/coderabbit/) | 78 | 531.4 | yes | open | none |
| [greptile](/agentic-code-review/greptile/) | 83 | 332.7 | yes | open | none |
| [graphite](/agentic-code-review/graphite/) | 90 | 603.6 | no | open | none |
| [qodo](/agentic-code-review/qodo/) | 242 | 178.6 | yes | open | none |
| [ellipsis](/agentic-code-review/ellipsis/) | 251 | 347.2 | yes | open | none |
| [superpositional](/agentic-code-review/superpositional/) | 18 | 276.4 | yes | open | none |

### Methodology

- Fetch https://<domain>/robots.txt with a plain HTTP GET and a desktop user agent.
- Parse User-agent groups line by line; a bot is blocked only if a group naming it (or the * wildcard) contains a 'Disallow: /' rule.
- Bots checked: GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider.
- Policy: blocked = GPTBot, ClaudeBot, and CCBot all disallowed from /; partial = at least one checked bot disallowed; open = no restrictions; unknown = robots.txt unreachable.
- Fetch https://<domain>/llms.txt; llmsTxt = HTTP 200.
- Fetch the homepage 3 times; TTFB is time to response headers; report the median. HTML size is the decoded body length of the last successful fetch.

### Limitations

- Single machine, single network location — TTFB is directional, not global CDN truth.
- n=3 runs per tool; we report the median, not a distribution.
- Simplified robots.txt parsing: bot names matched case-insensitively, only Disallow rules read, path wildcards beyond 'Disallow: /' not modeled.
- Homepage only; vendor pricing pages may have different crawler policies.
- Snapshot of 2026-09-27; vendors change crawler policies without notice.