robots.txt Explained: Syntax, Examples & Testing
Table of Contents
Every few months, a site owner discovers their staging server — `staging.example.com`, full of unfinished pages and placeholder text — ranking in Google. Or worse, they discover their admin login page in the search results. And every time, the story includes the sentence "but we blocked it in robots.txt," delivered with the confidence of someone who believes robots.txt is a lock. It isn't. It's a polite note taped to the door, and malicious crawlers can't read.
Once you internalize that robots.txt is a suggestion that well-behaved crawlers voluntarily follow, everything about it makes sense: the simple syntax, the reason it can't protect anything sensitive, and the specific Disallow mistakes that accidentally nuke sites from search results. Let's get the syntax down and the misconceptions out.
What robots.txt Actually Is: A Suggestion
When a respectable crawler (Googlebot, Bingbot) wants to visit your site, it first fetches `https://yoursite.com/robots.txt` and reads the rules. The file says things like "don't crawl /admin/" or "crawl everything but stay out of /tmp/." The crawler then chooses to comply. There's no enforcement mechanism — a scraper, a vulnerability scanner, or anyone with `curl` ignores the file completely, because nothing forces them to fetch it in the first place.
This has two consequences people get backwards. First, robots.txt cannot protect sensitive content. If a URL must not be seen, it needs authentication — a login, a token, IP restriction. Putting `Disallow: /secret-admin-panel/` in robots.txt is arguably worse than nothing, because the file is public and you've just published a treasure map of your sensitive paths. Attackers read robots.txt routinely for exactly this reason.
Second, blocking a page in robots.txt doesn't remove it from search results. Google can still index a blocked URL if other sites link to it — it just can't crawl the page content, so the listing appears without a description. Site owners who "blocked" embarrassing pages and then found them still listed (title only, no snippet) learn this the unpleasant way. To actually remove a page from the index, use `noindex` (a page-level directive crawlers honor when they can crawl the page) or authentication — not robots.txt.
So what is robots.txt for? Managing crawl budget and crawler behavior on content that's fine to exist but wasteful to crawl: faceted search URLs generating infinite parameter combinations, internal search result pages, printer-friendly duplicates, staging environments (combined with noindex and ideally auth). It's traffic management for polite robots, nothing more.
The Syntax in Five Minutes
The whole format is four directives you'll use and a few you'll rarely touch. Rules are grouped by user-agent:
`User-agent: *` starts a group applying to all crawlers; `User-agent: Googlebot` targets one specifically. More specific groups override the general one for that crawler.
`Disallow: /admin/` blocks everything under /admin/. `Disallow: /` blocks the entire site (the nuclear option — used on staging, catastrophic on production). An empty `Disallow:` with nothing after it means "disallow nothing," i.e., allow everything.
`Allow: /blog/` carves exceptions inside disallowed areas — e.g., disallow `/downloads/` but allow `/downloads/whitepaper.pdf`. Note that Allow isn't in the original 1994 spec; it's a widely-supported extension that all major crawlers honor, but obscure bots might not.
`Sitemap: https://example.com/sitemap.xml` points crawlers at your XML sitemap. It can appear anywhere in the file and you can list multiple sitemaps.
A minimal production file is often just:
User-agent star, allow everything, plus the sitemap line. That's genuinely all most small sites need. Complexity in robots.txt correlates strongly with mistakes — every additional rule is another chance to block something you meant to allow.
Two syntax details that cause disproportionate trouble: matching is prefix-based (`Disallow: /fish` blocks /fish, /fishing, and /fishsticks — use `/fish/` with the trailing slash to block only the directory), and the file must live at the domain root (`/robots.txt`) — a robots.txt in a subdirectory is ignored by every crawler.
Disallow Pitfalls That Hurt Real Sites
The hall of fame of robots.txt disasters, so you can avoid joining it:
Blocking CSS and JS. `Disallow: /assets/` seems tidy until you realize Googlebot needs your CSS and JavaScript to render pages and judge their quality. Blocking render resources can directly hurt rankings — Google has explicitly warned about this. Let crawlers fetch your static assets.
The staging file deployed to production. Someone copies the staging robots.txt (`Disallow: /`) to the live server during a deploy, and the site vanishes from search over the following weeks. This happens constantly. Make robots.txt environment-aware in your deploy pipeline — different file per environment, verified after every deploy.
Blocking parameterized URLs too aggressively. `Disallow: /*?` blocks every URL containing a query string — which on many sites includes pagination, sorting, and tracking variants of perfectly good content. Use Search Console's parameter handling or targeted rules instead of a blanket ban.
Case sensitivity. Paths are case-sensitive: `Disallow: /Admin/` does not block `/admin/`. Servers on Linux treat these as different directories, and so does the crawler.
Assuming order matters. Within a group, crawlers apply the most specific matching rule, not the first one — so you don't need to obsess over rule ordering the way you do with .htaccess. Put the rules in whatever order reads clearly.
If you're generating the file rather than hand-writing it, the Robots.txt & XML Sitemap Directive Generator builds valid syntax from checkboxes instead of memory, which sidesteps the prefix and formatting mistakes that hand-written files accumulate.
Testing in Search Console
Writing the file is half the job; verifying it does what you think is the other half. Google Search Console's robots.txt tester (under Settings) is the authority here: it fetches your live file, lets you test any URL against it, and tells you exactly which rule allowed or blocked it. Test the URLs you care about — the ones you blocked on purpose and a sample of ones you didn't.
Beyond the tester, watch the Page indexing report in Search Console for surprises: URLs "blocked by robots.txt" that you didn't intend to block, or a sudden spike in blocked pages after a deploy. And the Crawl stats report shows whether your rules are actually reducing wasteful crawling — if you blocked faceted URLs but crawl stats show Googlebot still hammering them, something's wrong with the rule.
For non-Google verification, `curl https://yoursite.com/robots.txt` confirms the file is served correctly (watch for a 200 status — a 404 means no rules apply, which is fine, but a 500 means crawlers may assume everything is blocked, which is not fine).
When to Use Noindex Instead
Since blocking ≠ removing, here's the decision rule: if the page should not appear in search results at all, don't block it in robots.txt — let crawlers reach it and serve a `noindex` directive (meta tag or `X-Robots-Tag` HTTP header). The crawler fetches the page, reads "don't index this," and drops it from the index. This is how you handle staging pages, thank-you pages, internal search results, and anything else that exists but shouldn't rank.
Use robots.txt blocking for: crawl-budget management (infinite parameter spaces, calendar archives generating endless URLs), politely keeping well-behaved crawlers out of utility paths, and staging environments in combination with noindex and authentication. Use noindex for: anything that must not appear in results. Use authentication for: anything that must not be seen, period. Three tools, three jobs — mixing them up is where the horror stories come from.
Try it free — no sign-up
Run the numbers yourself in seconds. Everything happens in your browser.
Open the Robots.txt & XML Sitemap Directive GeneratorFrequently Asked Questions
Does robots.txt block hackers or bad bots?
No. Robots.txt is a voluntary convention — malicious crawlers, scrapers, and vulnerability scanners simply ignore it, since nothing forces them to read it. Anything sensitive needs real authentication. Worse, listing sensitive paths in Disallow advertises them, because the file is publicly readable.
Why is my blocked page still showing in Google?
Because blocking crawling doesn't block indexing. If other sites link to the URL, Google may list it (without a description snippet) even though it can't fetch the content. To remove a page from results, allow crawling and serve a `noindex` directive instead — or require authentication.
Where does the robots.txt file go?
At the domain root: `https://example.com/robots.txt`. Crawlers only check that exact location; a robots.txt in a subdirectory is ignored. Each subdomain needs its own file (`staging.example.com/robots.txt` is separate from `example.com/robots.txt`).
What does "Disallow: /" do?
It blocks all compliant crawlers from the entire site. It's correct for staging and development environments, and catastrophic on production — deploying a staging robots.txt to production is one of the most common SEO disasters. Make the file environment-specific in your deploy process.
Should I block CSS and JavaScript files in robots.txt?
No. Googlebot needs your CSS and JS to render pages properly and assess their quality; blocking render resources can hurt your rankings. Keep static assets crawlable and block only genuinely wasteful URL patterns.
How do I test my robots.txt file?
Use Google Search Console's robots.txt tester (under Settings) to check specific URLs against your live file and see which rule applied. Also monitor the Page indexing report for unintended blocks after deploys, and `curl` the file directly to confirm it's served with a 200 status — a 500 error can make crawlers assume everything is disallowed.