Robots.txt: How to Create One That Googlebot Loves

What is robots.txt?

robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may crawl. It follows a tiny, strict syntax:

User-agent: *
Disallow: /admin/
Allow: /public/

Sitemap: https://example.com/sitemap.xml

Every line is either a rule (user-agent, allow, disallow) or a declaration (sitemap, host). Rules apply to the crawl — not to indexing — which is the most important thing to understand about the file.

robots.txt controls crawling, not indexing

This is where most people get burned:

• robots.txt blocked → Google can't crawl the page, but it may still index it (from links, sitemaps, or external references) and display it with a thin, contentless snippet. Worse, you can't see the page's errors in Search Console because crawling is blocked.
• noindex (meta tag or header) → the page is crawled, then explicitly excluded from the index.

So: to hide a page from Google, use noindex. Use robots.txt to stop crawlers from wasting resources on areas they shouldn't enter.

What should you block in robots.txt?

Block areas that are low-value, high-volume, or unsafe to crawl:

• Admin and staging paths — /admin/, /wp-admin/ (WordPress blocks its own), /staging/
• Parameter-heavy URLs — Disallow: /*?* for sites where query strings create infinite duplicates
• Search result pages — /search, /filter
• File duplication — PDF duplicates of web content, cache directories
• Private data — anything behind authentication shouldn't be crawled (but sensitive data must also be protected by login — robots.txt is not a security mechanism)

What you should not block: CSS, JS, and image files. Google needs them to render your pages correctly (it uses them for mobile-first rendering). The old advice to block static assets is obsolete.

How to reference your sitemap

Always declare your sitemap in robots.txt — it's the standard place search engines check:

Sitemap: https://example.com/sitemap.xml

You can list multiple sitemaps. This is separate from submitting the sitemap in Google Search Console — doing both is fine and recommended.

Common robots.txt mistakes

1. Blocking everything with Disallow: / — the whole site disappears from search. The only valid use is temporary maintenance windows.
2. Using robots.txt instead of noindex — as explained above, it doesn't stop indexing.
3. Typos in paths — rules are case-sensitive and prefix-based. Disallow: /Admin won't block /admin.
4. Conflicting rules — Google uses the most specific match, and Allow can win over Disallow even when it appears later in the file. Keep rules simple and obvious.
5. Huge files — Google may stop honoring rules in files larger than 500 KB. Keep yours under a few KB.

How to generate a valid robots.txt

The Robots.txt Generator builds a correct file from your answers — allow everything, block specific paths, add your sitemap URL — and gives you copy-ready output:

1. Choose whether to allow all crawlers or block specific directories.
2. Add the paths to block (e.g. /admin/, /private/).
3. Enter your sitemap URL.
4. Copy the generated file and upload it to the root of your site (yourdomain.com/robots.txt).

Verify it works

• Visit https://yourdomain.com/robots.txt in a browser — you should see your file.
• In Google Search Console, use the robots.txt Tester (under Crawling → robots.txt) to simulate a specific URL.
• After changes, remember Google caches the file for up to 60 days, so don't expect instant effects.

A sensible starter file

For a typical content site, this is enough:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

And when you're ready to protect crawl budget, add targeted disallows — never a blanket block. Generate yours with the Robots.txt Generator and pair it with a proper sitemap.