Robots.txt: What It Is and How to Fix a Missing File

What is a robots.txt file and what does it do?

A robots.txt file is a UTF-8 text document named exactly robots.txt, in lowercase, that sits in the root directory of one host. Martijn Koster proposed the idea in 1994 as the Robots Exclusion Protocol, also called the robots exclusion standard, and the IETF published it as the RFC 9309 standard in September 2022. Rules cover only the protocol, host and port that served the file, so www.example.com and blog.example.com each need their own copy in their own top level directory.

Inside, a web crawler finds each rule group starting with a User-agent line, followed by Allow or Disallow rules, plus an optional Sitemap directive and comment lines. Paths may use a wildcard asterisk, and a dollar sign marks the end of a URL. The robots.txt syntax and examples guide covers that, so we won't repeat it here.

The purpose is crawl management. Owners use the file to keep a search engine bot away from internal search results, carts, filtered listings and admin screens.

It is not access control, and it cannot remove a page from Google. Crawling and indexing are separate steps, so a blocked URL can still be indexed from links elsewhere, and a malicious bot will ignore the file. The standard says crawlers must parse at least 500 KiB, and Google ignores anything past that. Google also generally caches the file for up to 24 hours, so an edit can take a day to be noticed.

How SEO Site Checker tests your robots.txt file

The exact rule: our server takes the host from the address you entered and sends a GET request to https://HOST/robots.txt, following redirects, with a 3 second timeout on connecting and on waiting for data, and a user agent that identifies as SEO Checker Bot. HTTP 200 is a pass, 401 or 403 is a warning, and everything else is a fail. Only the status code is judged.

Because the request always uses HTTPS, a site without a valid certificate fails even if the file exists on plain HTTP. A warning and a fail both count as not passed in the score.

A web application firewall or bot protection rule often produces a 403 forbidden warning even though the file opens for people. Request the file the way our bot does, with a GET that follows redirects and prints only the status:

curl -sL -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (SEO Checker Bot)" https://example.com/robots.txt

If curl prints 403 while your browser shows the file, a security rule is rejecting the user agent or our address range. Warning and fail share one report message, which says the site lacks a robots.txt file, so use the table to see which case applies.

What pass, warning and fail mean in our report

What the robots.txt check does not judge

The body of the response is never read. We don't check the Content-Type, validate syntax or count rules, so any 200 passes, including a homepage or soft 404 page that the server hands back for an unknown path. Open the file yourself and confirm it is plain text with readable rules.

For syntax problems, use the Search Console robots.txt report, which lists parse errors and warnings. To test whether one URL is blocked, try the URL Inspection tool.

Where else our report reads your robots.txt

The Disallow directive check requests the same file again with its own request, reads the rules and grades them as a separate row, so a missing file fails there too. That request sends no custom user agent, and the row warns when the file has no Disallow lines.

The AI Search Readiness card also reads the file for what you say about six AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. That card is calculated after the score, so it never changes it.

Check my website free

Why a robots.txt file matters for SEO

Crawlers ask for robots.txt before they crawl pages. With no file, a crawler assumes everything is allowed, so absence is no penalty. What you lose is the one place that states your crawl preferences and sitemap address.

A broken response is worse than a missing one. According to the Google robots.txt specification, 4xx codes other than 429 are read as no file, but a 5xx server error makes Google pause crawling for about 12 hours while it retries, then work from the last good cached copy for up to 30 days. In the Search Console robots.txt report, a file that failed shows as Not fetched with the reason beside it, and that deserves a quick fix.

On big sites, keeping bots away from endless low value URLs such as filters, internal search and carts protects crawl budget. A site with a few hundred pages rarely sees a change, and we'd rather say so than promise a boost. The article on how Google crawls sites gives the background.

The Sitemap line also lets every search engine find your XML sitemap, and your logs get cleaner because bots stop producing a 404 on every visit.

How to find and check the robots.txt of any website

1. Add /robots.txt to the address. Type the host followed by /robots.txt in your browser. You should see plain text, not a styled page with menus.
2. Request the headers. Use the follow redirects flag, because our report follows redirects too. A healthy answer contains 200 and content-type: text/plain. A HEAD request can be answered differently from the GET our report sends, so if they disagree, trust the report.
3. Use Search Console for your own site. The robots.txt report in Google Search Console settings lists the files Google found for your top hosts, when each was last fetched and any errors or warnings.
4. Run the free report. Our free SEO report grades the row as pass, warning or fail.

curl -sIL https://example.com/robots.txt

Repeat the check for every subdomain and for both the www and bare domain, since each host has its own file.

How to fix a failed or warning robots.txt check

Find which status your file returns, then jump to the matching block. If you'd rather not write the file by hand, the free Robots.txt Generator builds one for you.

If the file is missing (404)

1. Create the file. Use a code editor, not a word processor, save it as UTF-8 and name it robots.txt in lowercase.
2. Add the basics. Include one User-agent group and a Sitemap line with your absolute sitemap URL.
3. Upload it. Place it in the web root next to your homepage file, usually public_html, www or htdocs, not in a subfolder.

If the result is a warning (401 or 403)

1. Hunt for the blocker. Look for HTTP password protection on the whole site, a firewall or bot protection rule, a security plugin, or a hosting rule that refuses automated requests.
2. Open the path. Allow public GET requests for /robots.txt, then re-test from outside your own network.

If you see a server error, a timeout or an SSL error

1. Read the log. Check the server error log for the request, then look for a rewrite rule that hands /robots.txt to a script and for an overloaded host.
2. Mind the clock. Our request gives up when the server stalls, so a file that eventually loads can still fail.
3. Check HTTPS. The check uses only https, so confirm a valid certificate is installed and the host answers on port 443.

If the report is green but the file is really a web page

A catch all rewrite is usually sending unknown paths to the homepage, and the report can't tell because the status is 200. Add a rule that serves the real file or returns a true 404, then confirm the content type is text/plain.

After any fix, reload the URL, run the header request again and re-run the report.

A starter robots.txt you can copy

Paste one of these into your editor, change the domain and save it as UTF-8. The first allows everything and announces your sitemap:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

The empty Disallow line blocks nothing. To keep bots out of one low value area, such as a cart, name it:

User-agent: *
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

WordPress sites commonly block the admin folder but keep the AJAX endpoint open for public pages:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

In every example the Sitemap line takes an absolute URL. No sitemap yet? The XML Sitemap Generator builds one.

Warning: a lone Disallow: / blocks the entire site. A staging copy that reaches production is a classic cause of vanished traffic, so check the live file after each deploy.

For per bot groups, crawl delay and wildcards, read the full robots.txt syntax guide.

Add or edit robots.txt on WordPress, Shopify, Wix and your own server

Platform support teams usually won't help with custom robots.txt edits, so test changes yourself.

WordPress

With no physical file, WordPress serves a virtual one built by the WordPress do_robots function, which includes the default admin rules. A real robots.txt in the site root takes over from the WordPress virtual robots.txt. You can also edit it with the Yoast SEO file editor or the Rank Math robots.txt editor. The Search Engine Visibility checkbox in current versions adds a noindex robots meta tag, not a Disallow line.

Shopify

Shopify generates the file by default. To customise it, add a Shopify robots.txt.liquid template from Online Store, Edit code, Add a new template. Shopify support does not cover edits to that template.

Wix

Open the SEO tools in the dashboard and choose the Wix Robots.txt Editor. Editing is flagged as an advanced feature, with a reset to default option.

Squarespace

The Squarespace robots.txt is generated automatically and has no editor. Use its crawler and per page search visibility settings instead.

Your own server, Apache or Nginx

Use an FTP upload or the cPanel File Manager to place the file in the Apache document root, often the public_html folder. On Nginx, an exact match location block can serve the file or return 404. If a CDN sits in front, confirm it adds nothing unexpected, since a Cloudflare managed robots.txt can prepend its own rules.

Robots.txt mistakes that give a failed or misleading result

• Wrong location or name. A file in a subfolder, or named ROBOTS.TXT, is never found at /robots.txt.
• Word processor formatting. It adds curly quotes or hidden characters that make rules unreadable.
• Wrong host. Rules on the www host do not cover a subdomain, and the report tests only the host you entered.
• An HTML page with a 200. The row is green, but crawlers read markup instead of rules. Open the file and look.
• Firewall answering 403. A security plugin or CDN rule can refuse our bot while visitors load the file fine.
• Copied staging rules. A staging Disallow: / on production, or blocked CSS and JavaScript folders, stops pages from rendering properly.
• Treating it as security. The file is public, so a listed path advertises your private areas. Use a login instead.
• Disallow for removal. A blocked page can still be indexed without its content, which Search Console reports as "Indexed, though blocked by robots.txt". To drop a page, use a noindex meta tag (or an X-Robots-Tag header) on a crawlable page, as the noindex tag guide explains, and confirm it with the Meta Tag Analyzer.