Robots Txt Generator

WHAT IT DOES

Robots Txt Generator creates robots.txt files that control how search engine crawlers interact with your website. Set user-agent directives to target specific crawlers, define allow and disallow paths to control access, add sitemap URLs, and configure crawl-delay settings. The tool includes five preset configurations: Allow All, Block All, WordPress, SEO Optimized, and Staging. A live preview shows your generated robots.txt file as you build it. Copy the content to your clipboard or download it as a robots.txt file.

WHERE IT RUNS

The extension runs entirely within your browser's local environment using the BotGentz framework. All robots.txt generation, rule formatting, and previewing occur on your machine without any network requests. No website data, generated rules, or user activity is ever transmitted, uploaded, or stored externally. The tool works on any website and respects the browser's security boundaries, operating within the isolated extension sandbox. Robots.txt generation uses plain text formatting with standard directive syntax.

HOW TO USE

Click the extension toolbar icon to open the Robots Txt Generator panel. Enter the User-agent (default is "*" for all crawlers). Add Allow paths (one per line) to specify directories crawlers can access. Add Disallow paths (one per line) to block access to specific directories. Set a Sitemap URL to help crawlers find your sitemap. Set a Crawl-delay value (in seconds) to control how often crawlers should request pages. Add an optional comment to document your robots.txt file. Click a preset (Allow All, Block All, WordPress, SEO Optimized, or Staging) to load a pre-configured set of rules. The output area shows the generated robots.txt file in real time. Click "Copy" to copy the content to your clipboard. Click "Download" to save the file as robots.txt. Toggle "Include comment" to include the comment in the output. Select the output format between Plain text and Markdown.

THE ROBOTS.TXT GENERATION ENGINE — WHY IT WORKS

Most robots.txt generators require you to know the exact syntax and formatting. This tool provides a user-friendly interface that generates correctly formatted robots.txt files according to the Robots Exclusion Protocol. The engine collects each directive from the input fields—User-agent, Allow, Disallow, Sitemap, and Crawl-delay—and formats them with the proper syntax. Allow and Disallow paths are processed line by line, with each path on its own line. The Sitemap directive is added at the end of the file with a blank line before it. The Crawl-delay value is formatted as a decimal number. The preset system loads pre-configured rule sets for common use cases, saving time when setting up new websites. The output is displayed as plain text in a monospace font, ready to be copied or downloaded. This approach ensures you get a clean, valid robots.txt file every time.

THE PANEL

The Robots Txt Generator panel is fully draggable by its title bar and remembers its position on screen per website. It snaps to the edges of your browser window for clean alignment. Collapse the panel to a compact icon strip using the minimize button, restoring it with a single click. Resize the panel by dragging any edge or corner; the input fields, presets, and output area adjust proportionally. Press the Escape key to close the panel instantly. The panel maintains independent state for each browser tab, allowing you to generate robots.txt files for multiple projects simultaneously.

PLEASE NOTE

This extension requires the free BotGentz extension framework to be installed and enabled. The tool generates robots.txt files but does not upload them to your server—you must copy or download the file and place it in your website's root directory. The presets are templates and should be customized for your specific site structure. The Crawl-delay directive is respected by some crawlers but not all. All generated content is processed locally and is never transmitted, logged, or shared. The extension requests no network permissions and functions entirely offline. The source code is fully auditable and available for review.