🤖 Robots.txt Generator

About the Robots.txt Generator

This robots.txt generator runs entirely in your browser. Choose a default rule, add per-user-agent rules, optionally include a sitemap directive, and the tool produces a ready-to-paste robots.txt file. Nothing is uploaded — all generation happens locally with JavaScript on your device.

What is robots.txt?

robots.txt is a plain-text file at the root of your domain (e.g. https://example.com/robots.txt) that tells crawlers which parts of your site they may access. It uses the Robots Exclusion Protocol, supported by all major search engines including Google, Bing, and DuckDuckGo.

Key directives

• User-agent — The crawler the following rules apply to. * matches every crawler.
• Allow — A path that the named crawler may access.
• Disallow — A path that the named crawler must not access.
• Sitemap — The absolute URL of your XML sitemap. Each Sitemap: line lists one sitemap.

How to use the generator

1. Choose a Default rule: Allow all opens the site to every crawler, Block all closes it.
2. Add specific rules with the Add Rule button — pick a user-agent, an action, and a path.
3. Optionally enter a Sitemap URL so search engines can find your sitemap.
4. Click Generate then Copy, and paste the result into a file named robots.txt at the root of your site.

Frequently asked questions

Does this tool upload my data? No. Everything happens locally in your browser. Your inputs never leave your device.

Is robots.txt a security control? No. It is a request, not an enforcement. Malicious bots can ignore it. Never use it to hide sensitive URLs — protect those with authentication instead.

What happens if I block a URL by mistake? The blocked URL will not be crawled, but it may still appear in search results if other pages link to it. Use the noindex meta tag or response header to fully remove a URL.

Where do I put robots.txt? At the root of your domain: https://yourdomain.com/robots.txt. Only one file per host is honored.