Robots.txt Generator

How Robots.txt Generator Works

A robots.txt file provides crawling instructions for compliant search engine crawlers. It identifies which website paths crawlers are requested to access or avoid. This generator creates a basic robots.txt file using a crawler identifier, one rule and an optional sitemap URL.

1. Understanding User-Agent

User-Agent identifies the crawler that a rule applies to. Use an asterisk (*) for all crawlers that follow the Robots Exclusion Protocol, or enter a crawler token such as Googlebot.

2. Understanding Allow and Disallow

Allow specifies a path that a compliant crawler may crawl. Disallow requests that the crawler should not crawl a matching path.

For example, Disallow: /private/ requests crawlers to avoid URLs under the /private/ directory.

3. Understanding Website Paths

Paths should begin with a forward slash and are interpreted relative to the website origin. Examples include /private/, /admin/ and /temporary/.

4. Adding a Sitemap

A Sitemap directive provides crawlers with the absolute URL of your public sitemap. Use a complete HTTP or HTTPS address. The reference helps crawlers discover the sitemap, but does not guarantee crawling or indexing.

5. Complete Example

User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Replace example.com with your actual website domain and change the restricted path to match your needs.

6. Where to Publish robots.txt

The standard robots.txt file belongs at the root of the website origin. For example: https://example.com/robots.txt

For a GitHub Pages project website hosted inside a subfolder, the hostname root and project folder are different locations. A robots.txt file inside the project folder does not replace the hostname root file.

7. Important Limitations

• Robots.txt does not protect confidential files.
• Disallow does not prevent visitors from opening a URL.
• Crawl restrictions do not guarantee removal from search results.
• Different crawlers may support different directives.
• A sitemap reference does not guarantee indexing.
• This generator creates one crawler group and one rule.
• Advanced multi-group configurations require manual editing.

8. How to Use the Generator

1. Enter a crawler identifier.
2. Choose Allow or Disallow.
3. Enter a path beginning with /.
4. Optionally enter your public sitemap URL.
5. Click Generate Robots.txt.
6. Review and copy the generated content.
7. Save it as robots.txt and publish it at the correct hostname root.

Where Your Data Goes

The generator processes the entered values using JavaScript in your browser. It does not send the rules to a separate generation API or fetch the sitemap URL for verification. The website itself loads its HTML, CSS and JavaScript from its hosting provider.