robots.txt Generator

Build custom crawler rules, add your sitemap, download the file.

A free online robots.txt generator — add rule groups per crawler, block AI bots with one click, and get a ready-to-upload file. No signup, runs entirely in your browser.

Sitemap URL (optional)

robots.txt

What does robots.txt do?

A robots.txt file tells search engine and other crawlers which parts of your site they're allowed to request — one plain-text file placed at your domain's root, checked by well-behaved bots before they crawl anything else. It doesn't hide pages from being indexed on its own (that's what a noindex meta tag does) — it just controls crawling.

User-agent groups

Each group starts with a User-agent line naming which crawler the rules below it apply to — * matches every crawler that doesn't have its own more specific group. Adding a separate group for a specific bot (like GPTBot) lets you give it different rules than everyone else, without affecting how search engines crawl your site.

Disallow vs Allow

Disallow blocks a path from being crawled — /admin/ blocks that whole directory. Allow carves out an exception inside a blocked directory — useful when you want to block a whole folder except one specific file or subfolder inside it. An empty Disallow: line (no path) means nothing is blocked for that group.

Blocking AI crawlers

Several AI companies run separate, named crawlers for training data collection (GPTBot, CCBot, ClaudeBot, Google-Extended, and others) that respect robots.txt independently of a company's main search crawler. The "Block AI crawlers" preset adds a disallow-all group for the major ones, so you can opt your content out of AI training scraping while leaving regular search indexing untouched.

Frequently asked questions

Add one or more rule groups — each with a user-agent and the paths to allow or disallow — then add your sitemap URL. The file updates live; copy it or download robots.txt and upload it to your site's root.

At the root of your domain — https://example.com/robots.txt. It won't be recognized anywhere else, including subdirectories.

Yes — use the "Block AI crawlers" preset to add Disallow rules for the major AI training/scraping bots (GPTBot, CCBot, ClaudeBot, Google-Extended, and others) as separate user-agent groups, without affecting how regular search engines crawl your site.

No. The file is generated locally in your browser with JavaScript — nothing you enter is ever sent anywhere.