robots.txt Guide: Correct Setup and Common Mistakes

A robots txt file is a small but critical text file that tells search engine bots which parts of your site they may crawl. Written correctly, it protects your crawl budget and stops bots from wasting time on useless pages; written incorrectly, it can close your entire site off to Google and wipe out your organic traffic overnight. That is why this seemingly simple file is one of the most error-prone spots in technical SEO.

What does robots.txt actually do?

robots.txt controls crawling behaviour only, not indexing. That distinction matters. The file sits in the root of your site, at https://yoursite.com/robots.txt. Before requesting a URL, a bot reads this file and applies the rules that target it. The standard was formalised as RFC 9309 (the Robots Exclusion Protocol).

The most common misconception is: "if I block a page in robots.txt, it will drop out of Google." It will not. robots.txt does not prevent indexing; it only prevents the content from being read. A page that is blocked from crawling but linked from other sites can still appear in search results with a "No information is available for this page" note. To remove a page from results entirely, use a noindex meta tag or HTTP header — and do not block that page in robots.txt, because the bot must be able to crawl the page in order to see the noindex directive.

Basic structure and directives

The file is plain text (UTF-8) and is read in groups. Each group starts with a User-agent line:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/global.css

Sitemap: https://yoursite.com/sitemap.xml

• User-agent: specifies which bot the rule applies to. * means all bots. For a specific bot you can write e.g. Googlebot.
• Disallow: forbids crawling of URLs starting with the given path prefix. Disallow: / closes the whole site; an empty Disallow: blocks nothing.
• Allow: creates an exception inside a Disallow rule. In Google, the longest (most specific) matching rule wins.
• Sitemap: declares the full URL of your sitemap. It is independent of the groups and can appear anywhere in the file.

Google supports the wildcard * (any sequence of characters) and $ to anchor the end of a URL. For example, to block all PDF files:

User-agent: *
Disallow: /*.pdf$

Common mistakes and their consequences

Mistakes in this file are silent: the site keeps working, but over months your traffic erodes. The most frequent ones are:

• Accidentally blocking the whole site. Forgetting to remove the Disallow: / line when moving from staging to production is the classic disaster. If that line is present on the live site, Google stops crawling.
• Blocking CSS and JavaScript. A rule like Disallow: /assets/ stops Google from rendering the page correctly. Google draws the page like a browser; without access to styles and scripts it misjudges your mobile-friendliness and layout.
• Treating robots.txt as an indexing tool. Listing sensitive resources (e.g. /secret-report.pdf) here actually exposes them — the file is public and hands curious visitors a directory of paths. To truly restrict access, use authentication or server-level protection.
• Ignoring case sensitivity. Paths are case sensitive: /Page and /page are treated as different.
• Returning a 5xx error. If robots.txt returns a server error, Google may temporarily interpret it as "everything disallowed" and slow crawling. A 404, by contrast, is treated as "everything allowed".

Crawl-delay and other misunderstood directives

The Crawl-delay directive is not supported by Google and is ignored, although engines such as Bing and Yandex do honour it. If you want to manage Googlebot's crawl rate, do it through Search Console or by improving your server response times. Likewise, the Noindex: directive worked unofficially for a while, but Google removed support in 2019 — it no longer works. The only correct way to block indexing is a page-level <meta name="robots" content="noindex"> or an X-Robots-Tag HTTP header.

Testing your file

Always validate before going live. A practical approach:

• The robots.txt report in Google Search Console shows which version of the file Google sees and whether it contains errors.
• Test whether a specific URL is blocked with the live URL Inspection tool.
• After a change, open the file directly in the browser (/robots.txt) to confirm it really returns what you expect; caching or redirects sometimes serve an old version.

Keep it small and simple. For most sites, a clear ten-line file is far safer than a sprawling hundred-line structure. The fewer exceptions, the lower the risk of error.

Frequently Asked Questions

Will my site be crawled without a robots.txt?

Yes. robots.txt is not mandatory; if the file is missing, bots treat everything as crawlable. Even so, keeping an empty file is a good habit so you can declare your sitemap and add rules when needed.

Can I block a page with both robots.txt and noindex?

No, do not use the two together. If you block the page in robots.txt, the bot cannot enter and see the noindex tag, so the page may stay indexed. To block indexing, leave the page crawlable and use noindex only.

How long does a change take to take effect?

Google usually caches robots.txt for about 24 hours. If you have made an urgent fix you can request a recrawl from Search Console; otherwise it updates automatically within a few hours.

Have your site's crawl settings gone out of control? If you want your robots.txt, sitemap and indexing setup audited end to end and put on a safe footing, get in touch with me.