Robots.txt Disallow: What It Does and How to Use It

What does Disallow mean in a robots.txt file?

The robots.txt file is a small text document at the root of a host. Web robots, including most search engine crawlers, read it to learn which parts of the site they may request. Martijn Koster proposed the idea in 1994, and the Robots Exclusion Protocol became an official IETF standard in 2022 as the RFC 9309 standard. Disallow is the directive most people mean when they say they "blocked" something, and it has a counterpart, the Allow directive, which carves exceptions out of a blocked path.

A Disallow line lives inside a group that opens with one or more User-agent lines. Its value is a path that starts with a slash, and the crawler compares that value with the beginning of every URL path on the host. Field names are not case sensitive, so "disallow" and "Disallow" behave the same. The path is case sensitive, which means /Photos/ and /photos/ are two different places.

Two values deserve special mention. An empty value, written as "Disallow:" with nothing after the colon, blocks nothing at all. A single slash, "Disallow: /", blocks the entire host for that crawler. One character separates an open site from a closed one, and that is why a forgotten slash causes so many painful launches.

The two parts of a disallow rule

Every rule has two parts. The User-agent directive names the crawler the rule is for, and a star means every crawler that has no group of its own. The Disallow line gives the path prefix those crawlers should stay out of.

# Keep every crawler out of the cart area
User-agent: *
Disallow: /cart/

The first line is a comment, because anything after a hash sign is ignored. Every rule below a User-agent line belongs to that group. A new group starts when a User-agent line appears after at least one rule.

Where the robots.txt file must live

The file has to answer at /robots.txt on the root of the host, never inside a subfolder. A copy at /blog/robots.txt is simply never requested. Scope is also set per protocol, host and port, so https://example.com and https://www.example.com are separate hosts, and a blog subdomain needs a file of its own. Rules written for www do not cover it.

Use UTF-8 plain text, and keep the file small, because Google applies a 500 KiB limit and ignores everything after the first 500 KiB. Our guide to robots.txt basics explains what the file is and how to create it. This page stays with the Disallow line itself.

How SEO Site Checker tests the disallow directive

Our free SEO score checker looks at one narrow thing: does the host publish a robots.txt file that contains a Disallow line? It takes the host name from the address you scanned, subdomain and port included, and requests https://HOST/robots.txt. The request follows redirects, carries no browser headers and gives up if the server fails to connect or send data within 3 seconds.

The exact rule: The check passes when the server answers with HTTP 200 and at least one line starts with the word Disallow, in any capital letters. The value after the colon does not matter, so a bare "Disallow:" passes.

A warning means the answer was 200 but no line starts with Disallow. That covers an empty file, a file with only Sitemap or Allow lines, and a normal web page that your server returns with status 200 at the robots address. A fail means anything else: a 404, 403 or 500 status, a timeout, a certificate error or no answer at all. A warning and a fail show the same alert box and the same red How to Fix button.

All three states are part of the score total, but only a pass adds to the passed count. A warning and a fail therefore cost the same share of your score.

The report has limits worth knowing. It does not test whether the page you scanned is blocked, it does not read User-agent groups, and it does not parse wildcards. Each line is matched from its first character, so an indented rule or a rule hidden behind a hash sign is not counted. RFC 9309 allows leading whitespace before a rule, so parsers that follow the standard usually accept an indented rule, but our check will not count it. Keep every rule at the start of the line.

Firewall warning: Some firewalls and bot filters return 403 to unknown script clients. If your browser shows the file but our report shows a fail, allow the request through, or open /robots.txt yourself to confirm what visitors see.

Pass, warning and fail at a glance

The table below pairs each state with a typical server answer, a sample file and the effect on your score.

What a pass does not prove

A pass is not permission to crawl your key pages, because the check cannot tell a careful rule set from a destructive one. Picture a file whose only rule is "Disallow: /blog/" under "User-agent: *", added by a plugin during a redesign. The report shows green, your guides quietly drop out of crawling, and nothing in the score warns you. The same goes for a file that blocks the whole host, or one that names an obscure bot and leaves everyone else alone.

When the report is green but traffic looks wrong, read the file yourself and test the affected URL with the steps further down. Then run the report again after any change, to see the file the way our scanner sees it.

Check my site again

Why the disallow directive matters for SEO

The main benefit is crawl effort. Filters, sort orders, internal search results, session IDs, tracking parameters and calendar pages, the typical output of faceted navigation, can generate nearly endless URLs that add nothing to your rankings. A few Disallow lines keep crawlers on the pages you want found. Google says crawl budget matters mainly for very large sites, so a small business site rarely needs more than a handful of rules.

The areas people usually block are utility areas: cart and checkout pages, account pages, internal search, staging copies and admin screens. Blocking them saves fetches. It is not protection, because anyone can read the file and request those URLs directly.

The risk is just as real. One wrong rule can hide revenue pages from every crawler, and blocking CSS or JavaScript stops Google from rendering the page the way a visitor sees it. A blocked URL cannot pass on a noindex or canonical hint either, because the crawler never reads the page, a trap the noindex section below walks through.

Some publishers add named groups for AI crawlers to opt out of certain training or search products, which makes robots.txt part of their wider bot management. Common tokens are GPTBot from OpenAI, CCBot from Common Crawl and Google-Extended, which controls whether Google may use your content for its generative AI products. Treat that as a policy choice about your content. It is not a ranking factor, and it does not change how Googlebot or Bingbot treat the rest of the site.

Robots.txt disallow examples you can copy

These patterns answer the questions people type most often: how to disallow all, how to block a folder, a file or a parameter, and how to keep one part open. Copy what fits, and test every pattern against real URLs before it goes live.

Block or allow the whole site

This first file shuts every crawler out of the host, which is correct for a private staging copy and nowhere else.

# Block everything (staging only)
User-agent: *
Disallow: /

This second file allows everything, because an empty Disallow value blocks nothing.

# Allow everything
User-agent: *
Disallow:

Use one or the other, never both in the same robots.txt, because groups with the same name are combined and the slash wins.

Block folders, files and parameters

Three patterns cover most needs. Folder rules end in a slash. To block a file type, combine the star with a dollar sign. To block a query parameter, match the start of the query string.

User-agent: *
Disallow: /private/
Disallow: /thank-you.html
Disallow: /*.pdf$
Disallow: /*?sort=
Disallow: /search

Each line does a different job. The first blocks a folder and everything below it. The second blocks one file. The third uses a wildcard and a dollar sign that anchors the end of the URL, so it blocks every address ending in .pdf. The last two target a sort parameter and the internal search path, and you should test those before you trust them, because a pattern that is too broad can hide real pages. Note that "Disallow: /*?sort=" catches only URLs where sort is the first query parameter. A URL such as /page?color=red&sort=price stays open, and you need "Disallow: /*sort=" or "Disallow: /*?*sort=" to catch the parameter in any position.

Matching works by prefix, which is easy to forget. Here is what the single rule "Disallow: /private" would catch if you left off the trailing slash:

/private/             blocked
/private/report.html blocked
/private-offers.html blocked
/privateer           blocked
/public/             not blocked

With the slash added, the rule covers only the folder. /private-offers.html would then stay open, which is usually what you meant.

Exceptions and named crawlers

To block a folder but keep one part of it open, pair Disallow with a longer Allow rule. Here the shop is blocked and the sale pages stay crawlable, because the longer path wins.

User-agent: *
Disallow: /shop/
Allow: /shop/sale/

User-agent: GPTBot
Disallow: /

# Our XML sitemap
Sitemap: https://example.com/sitemap.xml

The GPTBot group shows how a named crawler works. That bot follows only its own group, so it ignores the star rules above it, and nothing about /shop/ applies to it. Vendors also publish separate tokens for training and for search. In the OpenAI crawler documentation, GPTBot and OAI-SearchBot are listed as independent settings, so blocking one does not block the other. Comment lines start with a hash, and the Sitemap directive takes a full URL and works outside any group.

How robots.txt disallow rules are matched

Most real mistakes come from wrong guesses about matching, so here is the logic in one place. A rule is compared with the start of the URL path and includes the query string. That is why "Disallow: /search" also blocks /search?q=shoes.

Only two special characters work in a wildcard pattern: the star, which matches any run of characters, and the dollar sign, which anchors the end of the URL, and full regular expressions are not supported. Paths are case sensitive, while field names and crawler names are not.

Google documents how it reads these rules in how Google reads robots.txt, including precedence, wildcards, file size and caching. Two fields it does not honour are crawl-delay and any noindex or nofollow line placed in robots.txt. Google stopped supporting noindex in the file in 2019, so do not offer it as a fix.

Wildcards and end anchors

The star is flexible and the dollar sign is exact. A safe example is "Disallow: /*.pdf$", which blocks only URLs that end in .pdf. A risky one is "Disallow: /*?", which blocks every URL with any query string and can wipe out paginated or filtered pages you wanted indexed. Write the narrowest pattern that does the job.

Which rule wins

When several rules match a URL, the longest matching path wins, whatever the order of the lines. If an Allow and a Disallow tie in length, Google uses the less restrictive one, which is the Allow. A crawler follows only the most specific group that names it, and groups for different names are never merged. A GPTBot group therefore does not inherit the star rules, while two groups with the same name are combined.

The table applies three rules to five URLs: Disallow: /shop/, Allow: /shop/sale/ and Disallow: /*.pdf$.

Robots.txt disallow vs noindex: which one removes a page?

Disallow stops the fetch, while noindex removes a page from results. Mixing them up is the most common misunderstanding on this topic. If other pages link to a blocked URL, it can still be indexed, usually as a URL only listing with no description, because Google knows it exists even though it never read the content. Search Console reports this state as indexed though blocked by robots.txt, and it is the clearest example of why crawling and indexing are two separate questions.

A noindex instruction, delivered through a robots meta tag or an X-Robots-Tag header, does remove a page, but only when the crawler is allowed to fetch the page and read the instruction. Our noindex tag guide covers the syntax. To see which meta robots tag a page carries, run it through the Meta Tag Analyzer.

Pick the right tool for the job

Start from the goal, and the right tool follows. The table below pairs each common goal with the tool that fits and the reason.

Privacy: Neither Disallow nor noindex protects sensitive data, because both are public requests that a visitor can ignore. Put anything private behind authentication.

The disallow plus noindex trap

Putting Disallow and noindex on the same URL is a trap. The crawler is told not to fetch the page, so it never sees the tag, and the URL can stay in the index for as long as links point at it.

If a page is already blocked and you want it gone, work in this order. Remove the Disallow line, confirm the page carries a noindex tag, let it be recrawled and wait until it drops out of results. Only then decide whether to block it again.

How to check whether a URL is blocked by disallow

Our report only confirms that a Disallow line exists, so it cannot tell you whether one particular URL is blocked. These five steps can: read the live file by hand, then cross-check it with Google's robots.txt report and the URL Inspection tool.

1. Open the file. Load https://yourdomain.com/robots.txt in a browser. It should appear as plain text with status 200 and show the host you expect.
2. Read the right group. Find the group for the crawler, such as Googlebot or the star group, read every Disallow and Allow line in it and apply the longest match rule.
3. Use the Search Console report. The robots.txt report in Google Search Console shows the fetched file, its size and any parsing problems. It replaced Google's old standalone robots.txt tester, so a third-party validator is the alternative if you want a second opinion.
4. Inspect the page. The URL Inspection tool tells you whether crawling is allowed or blocked by robots.txt for that exact address.
5. Check from a terminal. The command below prints all response headers, including the status line and the Content-Type. Add -L to follow redirects the way our report does.

curl -I https://yourdomain.com/robots.txt

Look for a 200 status and a content type of text/plain. An HTML content type usually means a catch all page is answering instead of a real file. The -I flag sends a HEAD request, and a few servers answer HEAD differently from a normal GET, so a 405 or an odd status here does not prove the file is broken. In that case, run this variant, which makes a GET request and prints only the status code and content type:

curl -sL -o /dev/null -w "%{http_code} %{content_type}\n" https://yourdomain.com/robots.txt

How to fix a failed or warning disallow directive check

Read the state first. A fail means the file is missing or unreachable, and a warning means it exists but has no Disallow line. The two need different fixes.

Context helps you decide how urgent this is. Google treats a 404 or another 4xx status, except 429, as if no robots.txt existed, so crawling stays open. Server errors (5xx) are different, because they make Google pause crawling while it retries, so fix those first.

Failed: create or restore the file

1. Create the file. Make a plain text document called robots.txt, spelled in lowercase, and save it as UTF-8.
2. Place it at the root. Upload it to the web root of the exact host you scanned, not to a subfolder.
3. Confirm status 200. Open /robots.txt in a browser and check that it loads as text.
4. Rule out the usual blockers. A firewall or bot filter that answers 403 to script clients, or a server error, shows as a fail. A redirect to a login page, or a catch all route that returns an HTML page with status 200, shows as a warning, so open the file and check that it is plain text and not a web page.

If you have no file yet, the free Robots.txt Generator builds one for you, and our longer guide on how to create a robots.txt file walks through the details.

Warning: add rules on purpose

If everything on the site should be crawlable, write a star group and an empty Disallow line. If there are cart, search or admin URLs, add one precise Disallow line for each. Either way, finish with a Sitemap line.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help/

Sitemap: https://yourdomain.com/sitemap.xml

Remember that these rules are prefixes. "Disallow: /search" also catches /search?q=shoes, which is the point, but it would block a page such as /search-tips too. If you have pages like that, write "Disallow: /search/" for the folder and add a separate pattern for the query form.

Before you save, check that no Disallow: / remains from staging, and that no rule matches your CSS, JavaScript or image folders.

Then re-test. Our report fetches the live file on every scan, so run it again right after saving. Google refreshes its cached copy about every 24 hours, and the robots.txt report in Search Console has a Request a recrawl option if you do not want to wait.

Edit robots.txt disallow rules on WordPress, Shopify, Wix and your own server

Menus change over time, so treat the labels below as a guide and check the current wording in your dashboard. This section covers editing Disallow lines only. Creating the file from scratch is covered in the guides linked above.

WordPress, Yoast SEO and Rank Math

When no physical file exists, WordPress builds a virtual robots.txt on the fly. By default it holds Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, so a stock install already passes our check. To change it, use the Yoast SEO file editor (SEO, Tools, File editor) or the Rank Math edit robots.txt screen (Rank Math SEO, General Settings, Edit robots.txt). A real robots.txt file in the web root overrides the generated one, and Yoast writes such a file when you save. If the File editor is missing from the Yoast Tools screen, the host or a security plugin has probably switched off file editing inside WordPress, so edit robots.txt over SFTP instead.

The WordPress setting called Discourage search engines from indexing this site adds a noindex signal. It does not write a Disallow line, so it has no effect on this check.

Shopify and Wix

On Shopify, add a robots.txt.liquid template in the theme code editor. Loop over robots.default_groups so the default rules stay in place, then add your own lines inside the group they belong to. Starting from the defaults is wise, since they already block cart and checkout paths. The sample below prints every default group and adds one extra Disallow line to the group for all crawlers.

{% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.user_agent.value == '*' -%}
Disallow: /internal-search/
{% endif -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor %}

The condition on the user agent value is what limits the new line to the star group, so named crawlers keep only their defaults. The dashes inside the Liquid tags trim the whitespace around them, which keeps every rendered line flush left and leaves a blank line between groups. After saving, open /robots.txt and confirm that no line is indented, because our check matches from the first character. Treat the sample as a starting point, because Shopify can change the objects available in the template. Open /robots.txt after every save and confirm the output is what you intended.

On Wix, open the SEO section of the dashboard, then Tools and settings, then Robots.txt Editor. Edit the lines there and save.

Apache, Nginx and static hosting

Upload the file to the document root (on Apache, the folder named in DocumentRoot) and make sure no rewrite or catch all route swallows /robots.txt. On Nginx, an exact location block for /robots.txt stops a catch all try_files rule or a framework front controller from answering /robots.txt with an HTML page.

location = /robots.txt {
    try_files $uri =404;
}

A 404 from this block means the physical file is missing from the root. Skip the block if you rely on WordPress to generate the file, since it would return 404 instead. Behind a CDN such as Cloudflare, purge the CDN cache after every edit so crawlers and our report see the new version.

Common robots.txt disallow mistakes that hurt crawling

• Leaving the staging block in place. A "Disallow: /" copied from a test site goes live with the launch. Add a robots.txt check to the release list and run the report after every deployment.
• Blocking resources the page needs. If a rule covers CSS, JavaScript or image folders, Google cannot render the page properly, and your layout may be misjudged.
• Treating Disallow as security. The file is public, so listing /secret-admin/ advertises it. Use authentication, and for exposed folders read the directory browsing check.
• Expecting removal from results. Disallow does not delete an indexed page, as the noindex section above explains.
• Slips with case, slashes and hosts. /Blog/ is not /blog/, and a path without a leading slash does not match. Rules on www do not protect the bare domain or a subdomain.
• Bad file formatting. Two paths on one line fail, and so does a file saved as rich text or in another encoding. Save UTF-8 plain text and stay under 500 KiB.
• Never reloading the file. Reload /robots.txt after every edit, because a cache or plugin can serve an older copy.