Robots.txt Generator
100% PrivateSEO ToolsBuild a valid robots.txt file from visual rules for crawlers, paths and sitemaps.
Runs entirely in your browser. Your input is not uploaded to Ovatly or anywhere else.
User-agent: *
- 1In the first group, list the user agents the rules apply to (use * for all crawlers).
- 2Add Disallow paths to block and Allow paths to reopen parts of a blocked section, one per line.
- 3Add another group if one crawler needs different rules.
- 4Add your sitemap URL, then copy or download the generated robots.txt.
One group with user agent "*", Disallow paths "/admin/" and "/tmp/", and the sitemap "https://example.com/sitemap.xml" produces exactly: User-agent: * Disallow: /admin/ Disallow: /tmp/ Sitemap: https://example.com/sitemap.xml
- robots.txt controls crawling, not indexing or security. A disallowed page can still appear in search results if other sites link to it, and anyone can read the file, so never list secret paths in it.
- Crawl-delay is honoured by some crawlers (for example Bing) but ignored by Googlebot.
- The file must live at the root of the host, for example https://example.com/robots.txt, and each subdomain needs its own file.
Disallow tells crawlers not to fetch paths under a prefix; Allow reopens a more specific path inside a disallowed area. When both match, the longest pattern wins.
About this tool
The Robots.txt Generator builds a valid robots.txt file from visual rule groups: which crawlers each group applies to, which paths to disallow or allow, an optional crawl-delay, and your sitemap addresses. It produces the plain text file you place at the root of your domain.
Related tools
Part of Ovatly's free tools in SEO Tools.