Robots.txt Generator

The Robots.txt Generator is a free online tool that builds a standards-compliant robots.txt file in seconds — no hand-editing, no syntax mistakes. A robots.txt file is the first thing Googlebot, Bingbot, and every other crawler reads when they visit your domain: it tells them which URLs they may fetch, which folders to leave alone, and where to find your XML sitemap. With this generator you pick a default rule (allow everything, or block everything), then add per-bot rules for specific paths — for example disallow /admin/ and /api/ from all crawlers, stop GPTBot and ClaudeBot from using your content for AI training, or lock down an entire staging environment until launch. Every rule you add is written to the output with correct user-agent syntax, so the only step left is copying the file to the root of your domain as robots.txt. Everything runs in your browser — the settings you enter never leave your device — and the generated file follows the Robots Exclusion Protocol exactly as Google and Bing implement it.

1. 1

   Choose the default rule: "Allow all" for a normal live site, or "Disallow all" for a site under construction or a staging environment.

2. 2

   Add custom rules for specific crawlers — pick a user-agent (Googlebot, Bingbot, GPTBot, or any custom name) and the paths to Allow or Disallow for it.

3. 3

   Enter your XML sitemap URL so every crawler can discover your sitemap from the robots.txt file itself.

4. 4

   Review the live preview, then click Copy to grab the complete robots.txt content.

5. 5

   Upload the file to the root of your domain (public_html/robots.txt or the equivalent) and verify it at https://yourdomain.com/robots.txt.

• Block /admin/, /api/, and other internal folders from search engine crawls to protect crawl budget.
• Stop AI training crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot) from harvesting your content.
• Lock down a staging or development site so it can never appear in Google results.
• Prevent parameterized URLs (search, filters, pagination) from being crawled as duplicate content.
• Declare your sitemap location centrally so every crawler finds it.
• Temporarily pause crawlers during a migration or rebuild, then re-allow when you launch.

• Robots.txt controls crawling, not indexing. To remove a page that is already indexed, use a meta robots noindex tag — never robots.txt alone.
• Disallow private folders, but don't rely on robots.txt for security: it is a public request, not an access control, and malicious actors ignore it.
• Block AI crawlers explicitly by user-agent. GPTBot, ClaudeBot (Claude-Web), CCBot, and PerplexityBot each respect robots.txt — if you don't block them, they can and will train on your content.
• Keep the file small and simple. Google and Bing may ignore robots.txt files larger than 500 KB entirely.
• One file per site, at the domain root, named exactly robots.txt — lowercase. Anything else will be ignored.
• After publishing changes, test the file in Google Search Console's robots.txt tester, and remember that Google caches robots.txt for up to 24 hours (Bing and others may cache longer).

• Robots.txt supports wildcards: Disallow: /*?* blocks all URLs with query strings, and a trailing $ anchors the match to the end of a URL.
• Google caches robots.txt for up to 24 hours — if you make a change, wait a day before concluding it didn't work.
• The most specific matching rule wins, so a per-page Allow can override a folder-wide Disallow.
• Blocking AI crawlers does not affect your search rankings — Googlebot and GPTBot are completely separate systems.