SEO & LLM Diagnostic & Reference Tools

The check sends 24 or 25 small requests to the page’s site from this server: one as an ordinary browser, one under a made-up crawler user agent, one for robots.txt, one for each of the 18 crawlers whose user-agent string is documented, a second apart, and then three or four for what the site declares about AI use: llms.txt, /.well-known/tdmrep.json, the page asked for as Markdown and, when the page links to a Markdown copy or its address could have one, that copy. A user agent that gets a 429 (too many requests) is asked once more about 10 seconds later, followed by one more browser request. Use it on sites you own or are allowed to test. Results are kept for 15 minutes; a few checks per hour per visitor. One extra request goes through Anthropic’s API, so the text of the checked page passes through Anthropic. The page address and the results are recorded to build a dataset of how sites treat crawlers; nothing about who ran the check is kept.