Siftwrightrobots.txt generator for AI crawlers
Choose which AI bots may read your site (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more), add an optional Content-Signal preference, then copy or download a ready robots.txt.
Starting points are suggestions, not advice for your business. Googlebot and Bingbot are never blocked here. After you publish the file, confirm it with the AI crawler checker.
Choosing what to block
Blocking training bots keeps new pages out of future model datasets but does not remove what was already collected. Blocking search bots can remove you from AI answers that cite sources. Blocking user agents stops assistants fetching your page when someone asks about it. Many publishers block training and allow the other two; that is the default preset here.
Frequently asked questions
Does blocking a bot in robots.txt actually stop it?
Only if the operator chooses to obey it. Major AI companies such as OpenAI, Anthropic, Google and Perplexity document that their crawlers respect robots.txt, but it is a request, not access control. For enforcement use your CDN or firewall bot controls, and check your server logs.
What is the difference between training, search and user bots?
Training crawlers collect pages that may feed model training. Search crawlers build an index used to cite your pages in AI answers. User agents fetch a page when a person asks an assistant about it. You can set each separately, for example block training but stay visible in AI search.
What does the Content-Signal line do?
Content-Signal is a draft robots.txt extension introduced by Cloudflare. It states preferences for search, ai-input and ai-train as yes or no. It is not enforcement, and support from crawler operators is limited and still changing, so the line is optional here and off in the Allow-all preset.
Will blocking these bots hurt my Google ranking?
Blocking Google-Extended or Applebot-Extended does not remove you from normal search, per Google's and Apple's documentation. This tool never blocks Googlebot or Bingbot. If you block search-type AI bots such as OAI-SearchBot or PerplexityBot you may stop appearing as a cited source in those products.
Is my input sent anywhere?
No. The file is generated in your browser; nothing you type is uploaded. To verify a live site afterwards, use the AI crawler checker, which fetches that site's public robots.txt.
Where do I put the file?
At the root of your domain, served as plain text at /robots.txt, one file per host. Crawler names change over time, so recheck each operator's documentation before relying on a list.
Web data for your own pipeline
Transcripts, screenshots, news, contacts and tech stacks by API or MCP. Plans from $29/month; failures are free.