Skip to content
SEO

Robots.txt Generator

Build a robots.txt with rules for bots and sitemaps.

robotstxtcrawlerspiderseo
Loading tool…

About this robots.txt generator

The Robots.txt Generator is a free online tool that builds a standards-compliant robots.txt file in seconds — no hand-editing, no syntax mistakes. A robots.txt file is the first thing Googlebot, Bingbot, and every other crawler reads when they visit your domain: it tells them which URLs they may fetch, which folders to leave alone, and where to find your XML sitemap. With this generator you pick a default rule (allow everything, or block everything), then add per-bot rules for specific paths — for example disallow /admin/ and /api/ from all crawlers, stop GPTBot and ClaudeBot from using your content for AI training, or lock down an entire staging environment until launch. Every rule you add is written to the output with correct user-agent syntax, so the only step left is copying the file to the root of your domain as robots.txt. Everything runs in your browser — the settings you enter never leave your device — and the generated file follows the Robots Exclusion Protocol exactly as Google and Bing implement it.

Robots.txt examples

Input
Default rule: Allow all
Sitemap: https://example.com/sitemap.xml
Output
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Note: The simplest healthy file: everyone may crawl everything, and the sitemap location is declared for all bots.
Input
Default: Allow all
Rule: GPTBot → Disallow: /
Output
User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /
Note: Search engines keep crawling while GPTBot (used by OpenAI for AI training) is blocked site-wide. Add ClaudeBot, CCBot and PerplexityBot the same way.
Input
Default: Disallow all (staging site)
Output
User-agent: *
Disallow: /
Note: A staging or development site that must never appear in search results — nothing may be crawled until you change the default rule.

How to make a robots.txt file online

  1. 1

    Choose the default rule: "Allow all" for a normal live site, or "Disallow all" for a site under construction or a staging environment.

  2. 2

    Add custom rules for specific crawlers — pick a user-agent (Googlebot, Bingbot, GPTBot, or any custom name) and the paths to Allow or Disallow for it.

  3. 3

    Enter your XML sitemap URL so every crawler can discover your sitemap from the robots.txt file itself.

  4. 4

    Review the live preview, then click Copy to grab the complete robots.txt content.

  5. 5

    Upload the file to the root of your domain (public_html/robots.txt or the equivalent) and verify it at https://yourdomain.com/robots.txt.

Why generate a robots.txt

  • Block /admin/, /api/, and other internal folders from search engine crawls to protect crawl budget.
  • Stop AI training crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot) from harvesting your content.
  • Lock down a staging or development site so it can never appear in Google results.
  • Prevent parameterized URLs (search, filters, pagination) from being crawled as duplicate content.
  • Declare your sitemap location centrally so every crawler finds it.
  • Temporarily pause crawlers during a migration or rebuild, then re-allow when you launch.

Robots.txt best practices

  • Robots.txt controls crawling, not indexing. To remove a page that is already indexed, use a meta robots noindex tag — never robots.txt alone.
  • Disallow private folders, but don't rely on robots.txt for security: it is a public request, not an access control, and malicious actors ignore it.
  • Block AI crawlers explicitly by user-agent. GPTBot, ClaudeBot (Claude-Web), CCBot, and PerplexityBot each respect robots.txt — if you don't block them, they can and will train on your content.
  • Keep the file small and simple. Google and Bing may ignore robots.txt files larger than 500 KB entirely.
  • One file per site, at the domain root, named exactly robots.txt — lowercase. Anything else will be ignored.
  • After publishing changes, test the file in Google Search Console's robots.txt tester, and remember that Google caches robots.txt for up to 24 hours (Bing and others may cache longer).

Robots.txt generator tips

  • Robots.txt supports wildcards: Disallow: /*?* blocks all URLs with query strings, and a trailing $ anchors the match to the end of a URL.
  • Google caches robots.txt for up to 24 hours — if you make a change, wait a day before concluding it didn't work.
  • The most specific matching rule wins, so a per-page Allow can override a folder-wide Disallow.
  • Blocking AI crawlers does not affect your search rankings — Googlebot and GPTBot are completely separate systems.

Robots.txt generator FAQ

A robots.txt file is a plain-text file at the root of your domain that tells crawlers which parts of your site they are allowed to fetch. It is defined by the Robots Exclusion Protocol and read by Google, Bing, and most other crawlers before anything else. Even if you want everything crawled, a robots.txt file is still worth having because it is the standard place to declare your XML sitemap location — which helps new pages get discovered faster. If you want to make robots txt online without hand-editing syntax, this robots txt generator writes the robots txt file for you in seconds.

Disallow tells a crawler not to fetch a path; Allow explicitly permits a path — most useful for re-enabling one URL inside a disallowed folder. Both match by URL prefix, so Disallow: /blog blocks /blog, /blog/2026, and everything underneath. When a path matches both an Allow and a Disallow rule, the most specific match wins, and longer patterns win over shorter ones. In practice, most sites only need Disallow rules plus one Allow: / if they allow everything by default.

Add a rule group for each AI crawler in the generator — set the user-agent to GPTBot, ClaudeBot, CCBot, or PerplexityBot and add Disallow: / to block them site-wide. The generated file will look like: User-agent: GPTBot, then Disallow: / on the next line, repeated per bot. Search engines are unaffected because Googlebot and Bingbot have their own user-agents. This is the officially documented way to opt out of AI training for OpenAI, Anthropic, and Common Crawl — the crawlers check robots.txt before fetching.

Two layers are safest. First, generate a robots.txt with the default rule set to "Disallow all" (User-agent: * / Disallow: /) and upload it to the staging domain — this stops all crawling. Second, block indexing at the page level with a meta robots noindex tag or an X-Robots-Tag: noindex HTTP header, because Google can still index a page it cannot crawl if it discovers the URL through links. Never use password protection alone as your SEO safety net — add the noindex header too.

No. Disallowing a page in robots.txt stops crawling, but pages already in the index can stay there for a long time, and Google may show a snippet-less result ("no information is available for this page"). To remove an existing page, add a meta robots noindex tag or X-Robots-Tag: noindex header and keep the page crawlable — Google must be able to fetch the page to see the noindex. Use Search Console's Removals tool for urgent cases.

The file must be served at the root of your domain: https://yourdomain.com/robots.txt. For www and non-www versions both are separate hosts, so both should serve the file (or one should redirect to the other). After uploading, open the URL in a browser to confirm it loads, then paste it into Google Search Console's robots.txt tester to see exactly which URLs are blocked and allowed — and re-submit the sitemap if you changed its location.

Explore more seo tools

Browse the full collection of seo tools on the hub, or jump back to all categories.

All SEO toolsAll categories

Related tools

Sitemap GeneratorCanonical URL CheckerRedirect CheckerMeta Tag GeneratorTitle GeneratorCTA GeneratorJSON-LD GeneratorOpen Graph Preview

Did this tool help?

Anonymous — one tap, no sign-up.