SiftwrightWhich AI crawlers can read your site?
Check your robots.txt against 24 AI and search crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more), see Content-Signal lines, and validate llms.txt, including links your own robots.txt blocks.
Try: nytimes.com · github.com · anthropic.com
What the checker looks at
Two public files, read the way a crawler reads them. Results show what your published rules say, which is not always what your firewall enforces.
Per-bot verdicts
For each of 24 crawlers: allowed or blocked, which rule decided it, and whether a named group or your * group applied.
Training vs search vs user
Blocking model training is a different choice from blocking AI search citations. The report groups bots by purpose so you can see which you chose.
Content-Signal
Lists any Content-Signal lines (search, ai-input, ai-train) found in robots.txt, including those Cloudflare adds with managed robots.txt.
llms.txt
Title, summary, sections and links, plus a warning if the file is really an HTML error page.
Conflicts
Flags llms.txt links that robots.txt blocks for GPTBot, ClaudeBot or PerplexityBot, so you are not inviting a model to a page it may not fetch.
Fail-safe rules
A 404 robots.txt means no restrictions; a server error or unreachable file is treated as "disallow all", as RFC 9309 says crawlers should.
Robots.txt snippets
Block model training but stay visible in AI search and assistant answers:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /Allow everything and say so explicitly, with a Content-Signal preference (a stated preference, not enforcement):
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /Crawler names change; confirm tokens against each operator's documentation (OpenAI, Anthropic, Google, Perplexity) before relying on a list.
Audit many sites
Need to check a portfolio, a client list or your competitors' policies on a schedule? Pair this with our Tech Stack Detector to find sites by platform, then audit their robots.txt in bulk. For recurring reports, ask for a data feed.
Frequently asked questions
What does this checker do?
It downloads your public robots.txt and llms.txt (the same files any crawler can read), then works out for 24 AI and search crawlers whether the rules let them fetch your home page, using the matching rules from RFC 9309: the most specific user-agent group wins, and the longest matching path rule wins. It also lists any Content-Signal lines and checks that llms.txt is well formed.
Does it test whether my firewall or CDN blocks AI bots?
No. robots.txt is a request that well-behaved crawlers choose to follow. A CDN, WAF or bot-management setting can block a crawler even when robots.txt allows it, and the reverse. This tool reports what your published rules say, not what your network enforces, so check your CDN's bot or AI-crawler controls and your server logs as well.
What is the difference between training, search and user bots?
Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended and similar) collect pages that may feed model training. Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build an index used to cite your pages in AI answers. User agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page when a person asks an assistant about it. Many sites block training but allow search and user fetches, and you can set each separately.
Why are Google-Extended and Applebot-Extended in the list if they never crawl?
They are robots.txt tokens, not separate crawlers. Google and Apple fetch with their normal crawlers and use the token only to decide whether the content may be used for certain AI purposes. Blocking Google-Extended does not affect Google Search ranking, according to Google's documentation.
What is a Content-Signal line?
Cloudflare introduced a Content-Signal directive for robots.txt with three signals: search, ai-input and ai-train, each set to yes or no. They state preferences about how content may be used after it is fetched. They are not enforcement, and as of this check no major crawler operator has publicly committed to honouring them, so treat them as a declaration of intent.
What does the llms.txt check look for?
A Markdown file at /llms.txt that starts with an H1 title, has a short blockquote summary, and groups links under H2 headings, as in the llms.txt proposal. We also flag the common mistake of returning an HTML page with a 200 status, and links inside llms.txt that your own robots.txt blocks for AI crawlers. Whether any model provider reads llms.txt is not established, so do not expect traffic from it by itself.
Is it free? Do you keep my data?
Free, no signup. We fetch two public files from the domain you enter, cache the result for 30 minutes, and keep no list of domains or visitors. About 60 checks per day per network are allowed.
Web data for your own pipeline
Transcripts, screenshots, news, contacts and tech stacks by API or MCP. Plans from $29/month; failures are free.