Robots.txt

Robots.txt is a plain text file placed at the root of a website that gives search engine crawlers instructions about which parts of the site they are allowed to access. When a well-behaved crawler such as Googlebot arrives at a domain, one of the first things it does is request the file at the site's root address, for example www.example.com/robots.txt. The rules it finds there shape how the crawler moves through the rest of the site. Because the file lives in a single, predictable location and uses a simple syntax, it has become one of the most widely honored conventions on the web.

The mechanics are straightforward. The file is organized into groups, each beginning with a user-agent line that names the crawler the rules apply to. An asterisk means the rules apply to every crawler. Below the user-agent line, disallow directives list URL paths the crawler should not fetch, and allow directives carve out exceptions within those paths. A common pattern blocks an internal search results folder or an admin directory while leaving the rest of the site open. The file can also point to the location of an XML sitemap, which helps crawlers find the pages you do want indexed. It is important to understand that these directives are requests, not enforcement. Reputable search engines obey them, but malicious bots and scrapers can and do ignore the file entirely.

The convention dates to 1994, when the Robots Exclusion Protocol was proposed as an informal standard to give site owners a way to keep crawlers out of sensitive or resource-heavy areas. For decades it remained a de facto agreement rather than a formal specification. In 2019 Google helped push the protocol toward standardization, publishing its parser and working to codify the rules that had been loosely followed for years. The name itself is literal: it addresses the robots, or automated crawlers, and carries the .txt extension of a plain text file.

For a business, robots.txt matters because it influences how efficiently search engines spend their limited attention on your site. Large sites with thousands of URLs benefit from steering crawlers away from low-value pages, such as faceted filter combinations or session-based duplicates, so that crawl activity concentrates on pages that actually earn traffic. A clean, correct file supports healthy crawlability and protects crawl budget. A careless one can quietly cause serious damage.

The most common and costly mistake is confusing blocking with removal from search. Disallowing a page in robots.txt only stops crawling; it does not remove the page from the index. A URL that is blocked but linked from elsewhere can still appear in results, often with no description because the crawler was never allowed to read it. To keep a page out of the index, you use a noindex directive through a meta robots tag or an HTTP header, and crucially you must leave the page crawlable so the engine can see that instruction. Blocking CSS or JavaScript files is another frequent error, because it prevents search engines from rendering pages the way users see them. Treated with care, robots.txt is a precise tool. Treated carelessly, a single stray line can hide an entire site.