Which AI tool vendors can machines actually read?

49 of 69 tracked AI-tool vendors publish an llms.txt; 64 of 69 name at least one AI crawler in robots.txt; 0 declare Content-Signals; 3 send RFC 9727 Link headers; 14 offer any markdown path. Machine-readability is mostly robots.txt compliance plus a minority of purpose-built surfaces: the llms.txt standard has real adoption (71% here), the newer signals barely exist yet. We measure and label; we do not grade.

Reading the labels

"AI-crawler robots verdicts" summarises robots.txt groups for GPTBot, ClaudeBot, Claude-SearchBot, OAI-SearchBot, PerplexityBot, CCBot, and Google-Extended. An exact-name group wins over the wildcard group; a root Disallow: / in the winning group is "blocks". "no AI-crawler blocks" means no checked crawler is disallowed, whether by silence, wildcard permission, or explicit allow groups.

An "error" is our fetch failing (timeout, DNS, TLS), never a vendor choice: single-vantage probes fail for reasons that say nothing about the target, so errors are recorded as errors and never upgraded to labels about intent. The same discipline applies in reverse: a 200 to our probe is a measurement, not a policy statement.

Method

1. Probe each tracked vendor domain over HTTPS: /llms.txt, /llms-full.txt, robots.txt, Link headers on /, homepage with Accept: text/markdown, and /index.md.
2. Parse robots.txt per crawler group; exact-name groups win over the wildcard group; a root Disallow in the winning group means disallowed.
3. Every probe carries our research user-agent; statuses are recorded as observed. 'error' (timeout, DNS, TLS) is never read as a vendor choice.
4. One vantage point, no JavaScript rendering. Re-scan cadence: on publication of this dataset, ad hoc.

Vantage: TestedActually build machine, datacenter IP, 2026-10-07. The vendor list is every domain in our tool index (69 after deduplication) — the same set our comparisons draw on, so this table answers "can machines read the tools we cover?" with no cherry-picking.

Limitations

• One vantage point; geo-routing or IP reputation can change what a server returns.
• No JavaScript rendering: pages whose llms.txt equivalent is generated client-side read as absent.
• Homepage-level probes only; a vendor could expose markdown on documentation paths we did not probe.
• llms.txt adoption is counted by presence of a 200 response, not by conformance to the spec.

Reproduce

The scanner is a single script (scripts/machine-readability-scan.py) in the site's repository. The machine-readable version of this page is at /research/machine-readability.json. Our own site publishes every surface this table measures: /llms.txt, per-crawler robots groups, Content-Signals, and RFC 9727 Link headers.