robots.txt is a plain text file at the root of your domain that tells crawlers which paths to leave alone. It is simple enough to write by hand and easy to get catastrophically wrong, mostly because of one misunderstanding about what Disallow actually does.
Disallow does not mean "keep out of Google"
This is the single most consequential misconception in the subject. Disallow tells a crawler not to fetch a URL. It does not tell a search engine not to list it. Those are different things, and conflating them produces exactly the opposite of what people intend.
If another site links to your disallowed page, Google can index the URL without ever fetching it. You have seen the result: a search listing with the address and the line "No information is available for this page." The page is in the index. You just prevented Google from reading the very thing that would have described it.
Which leads to a genuine paradox worth internalising. To keep a page out of the index you need a noindex directive — a meta tag or an HTTP header on the page itself. But Google can only see that directive by fetching the page. If robots.txt blocks the fetch, the noindex is never read. Disallow and noindex on the same URL cancel each other out, and the page stays indexed. Pick one: Disallow to save crawl budget, or noindex to remove from search — never both.
It is a public file listing your secrets
robots.txt is fetchable by anyone at yourdomain.com/robots.txt. It is one of the first things a penetration tester reads, for the obvious reason: a file whose whole purpose is enumerating the paths you would rather people did not visit is a gift.
Writing Disallow: /admin-backup-2019/ does not hide that directory. It advertises it, to everyone, forever, in a file designed to be read. Compliance is voluntary — well-behaved crawlers honour it and anything malicious ignores it entirely, while now knowing exactly where to look.
The rule that follows: robots.txt is for things you do not want crawled, never for things you do not want found. Anything genuinely sensitive needs authentication. If a path must not be reached without a login, put a login on it — that is the only mechanism that is not a polite request.
Blocking AI crawlers is an honour system
The AI bot section here covers the crawlers that currently declare themselves: GPTBot, Google-Extended, CCBot, Claude-Web, anthropic-ai and Omgilibot. Blocking them is a reasonable choice and it works exactly as far as their operators choose to make it work.
Two things worth knowing. Google-Extended is not a crawler at all — it is a token that controls whether content already fetched by Googlebot may be used for AI training. Blocking it does not affect your search ranking, which is a distinction Google made deliberately and that a lot of site owners have missed in both directions.
And the list dates. New crawlers appear constantly, they choose their own user-agent strings, and nothing obliges them to announce themselves. A robots.txt written today blocks the bots that existed today. If this matters to you, it is a file to revisit rather than write once.
The rules that catch people out
The file must be at the root, exactly at /robots.txt. A file at /subfolder/robots.txt is ignored entirely, and subdomains need their own — a robots.txt on example.com says nothing about shop.example.com.
Paths are case-sensitive: Disallow: /Admin does not block /admin. The match is a prefix, so Disallow: /page blocks /page, /pages and /page-two alike — a trailing slash matters more than it looks.
When a URL matches several rules, the most specific one wins rather than the first, which is why Allow can carve an exception out of a broader Disallow. And an empty Disallow: means allow everything, while Disallow: / means block the entire site. One character between those two, and it is the character most likely to be left in after a staging deploy.
Common questions
Is anything sent to a server?
No. This tool is marked "client": the file is assembled in your browser tab from what you type. Nothing is transmitted.
I blocked a page and it is still in Google. Why?
Because blocking the crawl does not remove the listing — see above. If the page is already indexed, the fix is to remove the Disallow so Google can fetch the page again, serve a noindex on it, wait for a recrawl, and only then block it if you still want to. Counterintuitive, and it is the only sequence that works.
Do I even need a robots.txt?
Not for a small site with nothing to hide — no file means crawl everything, which is usually what you want. It earns its place when you have paths worth keeping out of the crawl: search result pages that generate infinite URLs, faceted filters, staging areas. A missing robots.txt returns a 404 and crawlers handle that fine, so an empty file is not something to add for the sake of it.
Should I list my sitemap in it?
Yes, it costs one line and every major engine reads it. It is not a substitute for submitting the sitemap in Search Console — that is where you see whether the URLs actually got indexed — but it is how crawlers that are not Google find it.
Does Crawl-delay work?
Not with Google, which ignores it entirely; crawl rate is controlled in Search Console instead. Bing and Yandex do honour it. If Googlebot is genuinely overloading your server that is a signal something is wrong with the server rather than the crawler — but the setting exists in Search Console if you need it.