Robots.txt Builder
Create a clean, valid robots.txt file for your website. Set crawl rules, add sitemaps, and copy the result to your site root.
Load Existing
Imported with warnings
Sitemaps
Add one or more sitemap URLs. These are written at the end of your robots.txt file.
Used by Yandex to identify your preferred domain. Leave blank if you don't need it.
User-agent Group
Rules in this block apply to the user-agents listed below.
Seconds between requests. Google ignores this; some other crawlers may use it.
robots.txt Output
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Save the output as robots.txt at the root of your website (for example,https://example.com/robots.txt).
Upload it to your server or add it through your hosting or CMS file manager. Changes can take time to be picked up by crawlers.
Save to Browser
Store this tool state in local storage on this device. You can load it again later from Load Existing.
Saving will update your current saved item. Use Save as New to keep both.
Load from Saved
Choose a previously saved version of this tool from this browser.
· Current
No saved items match that search.
Load from URL
Fetch robots.txt from a public website. Enter a site URL or a direct robots.txt link.
We'll fetch /robots.txt from the site root when you enter a homepage URL.
Paste robots.txt
Paste an existing robots.txt file to import its rules into the builder.
Delete Saved Item
This removes the saved copy from this browser only.
Delete ?
Build a robots.txt file in minutes
A robots.txt file tells search engine crawlers which areas of your website they can and can't crawl. It's one of the simplest ways to guide how search engines access your site and is commonly used to prevent unnecessary pages, folders, or files from being crawled.
The Robots.txt Builder helps you generate a valid robots.txt file without needing to remember the syntax. Add crawl rules, user-agent groups, and sitemap URLs, then copy the finished file directly to your website.
Understanding Allow and Disallow
The two most common robots.txt directives are Allow and Disallow. A Disallow rule tells crawlers not to request matching URLs, while Allow creates exceptions within broader blocked sections.
For most websites, a single User-agent: * group is all that's needed. Create additional groups only when specific crawlers, such as Googlebot or Bingbot, require different rules.
Help search engines discover your sitemaps
Adding one or more Sitemap directives helps search engines find your XML sitemaps more efficiently. This is particularly useful for larger websites or sites that publish separate sitemaps for pages, images, videos, news, or other content types.
What this builder supports
- Build robots.txt files with multiple user-agent groups and crawl rules.
- See the generated file update instantly as you edit.
- Copy the finished robots.txt with one click.
- Validate paths, sitemap URLs, and crawl-delay values.
- Import an existing robots.txt from a URL, saved draft, or pasted text.
- Save drafts locally in your browser.
- Add one or more sitemap URLs and an optional Host directive.
When should I use robots.txt?
Use robots.txt whenever you want to guide how search engine crawlers access your website. It's commonly used to block admin areas, internal search pages, temporary content, development environments, or other sections that don't need to be crawled.
Robots.txt controls crawling, not access permissions. It shouldn't be used to protect private content or confidential files, and it doesn't guarantee that blocked pages won't appear in search results. For pages that should remain accessible but not be indexed, use thenoindex directive instead.
Publishing your robots.txt file
Once you've generated your robots.txt file, save it asrobots.txt and upload it to the root of your website so it's available at:
https://yourdomain.com/robots.txt
After publishing, test your robots.txt file using Google Search Console to confirm that your rules behave as expected.
Best practices
- Test your robots.txt in Google Search Console after publishing.
- Don't use robots.txt to protect sensitive content - use authentication instead.
- Avoid blocking CSS, JavaScript, or image files that search engines need to render your pages.
- Only create separate user-agent groups when different crawlers genuinely need different rules.
- Keep your robots.txt simple and only block paths you don't want crawled.
- Remember that robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other pages link to it.
Privacy
Everything you create stays on your device.
- Everything you create and edit stays in your browser.
- No sign-up or account is required.
- Saved drafts are stored locally on your device, not on Contismo's servers.
Frequently asked questions
robots.txt is a plain text file placed at the root of your website that gives instructions to search engine crawlers about which parts of your site they should or shouldn't crawl. It only applies to crawlers that choose to follow the standard.
Upload the file to the root of your website so it's available at https://yourdomain.com/robots.txt. Search engines look for it in this location, so placing it elsewhere won't have the intended effect.
Not necessarily. If you don't need to restrict crawler access or advertise your XML sitemap, your website can work perfectly without one. Many small websites don't require a robots.txt file.
No. Each website should have a single robots.txt file at the root of its domain. That file can contain multiple user-agent groups, crawl rules, and sitemap directives.
robots.txt controls whether search engine crawlers are asked to crawl a URL, while meta robots tags and the X-Robots-Tag HTTP header control whether content can be indexed after it's been crawled.
Use robots.txt for site-wide crawl management and meta robots when you need page-level indexing control.
Yes. Blocking a page in robots.txt doesn't guarantee it won't appear in search results. If other websites link to the page, search engines may still index the URL without crawling its content. Use a noindex directive if you want to prevent indexing.
Yes. You can add multiple Sitemap entries to a robots.txt file. This is commonly used for websites that publish separate sitemaps for pages, images, videos, news, or other content types.
Yes. You can import an existing robots.txt file from a public URL, load a saved draft stored in your browser, or paste the file contents directly into the builder.
No. Drafts are stored locally in your browser and aren't uploaded to Contismo's servers. When importing from a URL, only the requested URL is sent so the robots.txt file can be retrieved.
