“We noticed you may be using an ad blocker. SEO Site Checker relies on ads to keep our SEO tools free. Please consider disabling your ad blocker to support us - thank you!”“We noticed you may be using an ad blocker. SEO Site Checker relies on ads to keep our SEO tools free. Please consider disabling your ad blocker to support us - thank you!”
Test any robots.txt file for Googlebot, Bingbot or AI crawlers. Fetch the live file of a website or paste a draft, and see which rule allows or blocks each address and the line it is on.
What this test does and does not tell you
It follows the robots.txt rules Google publishes (they match RFC 9309). Other crawlers can read the same file differently, so this is not a promise about any search engine or AI crawler.
A robots.txt rule controls crawling, not indexing. An address that is blocked here can still appear in search results without its content if other pages link to it. To keep a page out of results, use a noindex rule on a page that crawlers are allowed to fetch.
The addresses you list are only compared with the rules, as text. We never request them.
A live test reads the file our server receives. A site can answer a different crawler, country or user agent differently, so paste the text to test the version you have in hand.
A robots.txt file is a request that well-behaved crawlers follow. It does not stop anyone from visiting an address.
What Is a Robots.txt Tester?
Quick answer: A robots.txt tester checks whether a crawler may fetch an address and names the rule that decided it. Ours reads a site's live file or pasted text, tests up to 25 addresses and shows each rule's line number. Test your own robots.txt above.
A robots.txt file is plain text UTF-8 at a site's root that asks crawlers to skip some paths. A free robots.txt tester shows what those rules do before a crawler finds out, which beats learning it from a traffic graph. It never requests the addresses you list, and it is not Google's own tool. No file yet? Our post on how to create one covers it.
How to Test a Robots.txt File
Is my page blocked by robots.txt? This robots.txt checker gives you the answer in five steps.
Choose the source. "Test a live website" reads a public site's file: type example.com or a page address. "Paste a robots.txt" tests a draft before upload, and covers a staging site behind a password.
Pick a crawler. "Crawler to test" starts on Googlebot. For others choose "Another crawler (type its name)" and enter one name: letters, digits, hyphen and underscore, up to 50 characters.
List the addresses. One path or full http or https address per line, up to 25 at once. Keep the query string, because it can decide the result.
Press Test robots.txt. Every address gets a row with its verdict and the rule behind it.
Read, edit, test again. "Copy this file into the paste box" turns a live file into a draft. "Fetch again" re-reads the live file once your copy is 30 seconds old.
Start with the costly ones: the home page, a stylesheet, a page you meant to hide and an address with a query string. Our own file, checked November 2, 2026, allows /website-seo-score-checker but blocks it once ?url=example.com is added (Disallow: /*?url=, line 13). Try your own file the same way.
How the Tester Decides: Allow, Disallow and the Longest Match
We follow Google's robots.txt specification, which builds on the IETF standard, RFC 9309. The matcher is ours. This sample file went through a Googlebot robots.txt test with eight addresses, six shown below.
Tested as GPTBot, the same file blocks the home page with Disallow: / on line 8, because that group replaces the * group.
A crawler follows the group that names it most closely, else the * group. Groups naming the same crawler are combined.
The longest matching rule wins, and an Allow rule wins a tie. Our guide covers what a Disallow rule does.
The asterisk matches any run of characters; the dollar sign end anchor stops the match there.
Paths are case sensitive, names are not. A non-ASCII path matches its percent-encoding, so /über/ and /%C3%BCber/ match.
Test Googlebot, Bingbot and AI Crawlers Such as GPTBot and ClaudeBot
"Crawler to test" holds eleven crawlers: Googlebot, Googlebot-Image, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot and SEOSiteCheckerBot. Any other user agent token, such as Claude-SearchBot, goes through "Another crawler (type its name)".
Take a file where GPTBot, ClaudeBot, CCBot and Google-Extended each get a Disallow: / group, and a * group carries Allow: /. Those four are blocked from the home page; the other seven follow the * group and are allowed. The "All listed crawlers at a glance" table shows that split.
OpenAI's crawler documentation says its OAI-SearchBot and GPTBot settings are independent, and that robots.txt rules may not apply to ChatGPT-User because users start those requests. A file naming only GPTBot leaves OAI-SearchBot and ChatGPT-User on the * group, if any.
Google says Google-Extended is a control token with no user agent string of its own, so the tester reads the group written for it. It reports what your file says, not whether an AI product honors it. For the bigger picture, read how AI search finds your pages.
Read the Result: Deciding Rule, Line Number and Group
Each row has a badge (Allowed, Blocked or Not decided), the Deciding rule, a line link into the file and a Why sentence. The Why is re-checked against the parser and dropped if they disagree. "Other rules that also match" lists the losers. Not decided means the file was too large or too complicated to test in time. A file that cannot be read at all gives no badges.
Read "The group Googlebot follows" before editing a rule: it flags a * group that goes unused. A Disallow: / with no Allow exception raises a red site-wide banner.
Warnings work like a robots.txt validator for syntax: no colon, a misspelled field such as Disalow, a rule before any User-agent, a relative Sitemap address. None of those lines is applied, and an empty Disallow blocks nothing. Sitemap lines are listed, not fetched.
What the Tester Shows When the robots.txt File Cannot Be Read
The HTTP status code, or a failure, decides what you see.
What the site answers
What the tester shows
What it means
404 not found, or another 4xx except 429
No robots.txt file
Nothing is restricted. A 401 or 403 warns that a real file may exist.
429 too many requests, or a 5xx server error
The server did not deliver the file
No verdict. After a 5xx, Google stops crawling for a while.
A long redirect chain or a loop
Too many redirects
Everything shows as allowed. Google follows at least five hops; we read five and reject a sixth, our own cut-off.
HTML instead of text
A web page, not a robots.txt file
First lines shown, everything allowed.
A block or protection page
Our request met a protection page
No verdict.
No answer at all
The website could not be reached
No verdict. DNS, a timeout or TLS failed.
A site can answer our server differently from Googlebot, so after a 403, paste the file from your browser. Google and this tester both read only the first 500 KiB. Seeing "No robots.txt file" on your own site? Our guide shows how to fix a missing robots.txt file.
What This Test Cannot Show You
The box under the tool covers crawling versus indexing. Five more limits:
No noindex. It does not read a meta robots tag or an X-Robots-Tag header. The noindex tag guide covers those.
Cached copy. Google generally caches robots.txt for up to 24 hours, so a live edit can take a day to arrive. We reuse a fetched file for up to 10 minutes.
Typos. Google calls a line like Disalow invalid, yet its own reference code tolerates a few frequent typos and a missing colon. We do not apply such lines, so fix the line and retest.
Special cases. AdsBot-Google, which Google says ignores the * group, is judged by that group here. Crawl-delay is shown, not applied, and whether an address exists is not tested. For what happens after a fetch, read how Google crawls, renders and indexes.
What Our Server Does With Your Request
This free online robots.txt tester needs no account. A live test requests only the typed site's /robots.txt, following redirects, as SEOSiteCheckerBot, and never the addresses you list. Pasted text is sent to our server to be tested, and no website is contacted.
Frequently Asked Questions (FAQs) is a list of common questions and answers provided to quickly address common concerns or inquiries.
How do I test a robots.txt file for free?
Use the tool above: choose Test a live website or Paste a robots.txt, pick a crawler, list up to 25 addresses and press Test robots.txt. Each address comes back Allowed or Blocked with its rule and line. It is free, with no account.
How do I check if a URL is blocked by robots.txt?
Enter the site, put each path or full address on its own line and keep any query string. A Blocked row names the Disallow rule and its line; a longer matching Allow turns it into Allowed. For Google's own verdict, use URL inspection in Search Console.
Which rule wins when an Allow and a Disallow rule both match?
The longest matching rule wins, because a longer path is more specific. When an Allow and a Disallow match with the same length, the Allow rule wins the tie. Wildcards count by pattern length as written, so for /public/data.json the longer Allow: /public/*.json$ beats Disallow: /*.json$.
Why does the tester say a URL is allowed when my robots.txt seems to block it?
Usually one of six causes: the crawler has its own group, a longer Allow won, paths are case sensitive, a $ or query string changed the match, a typo or missing colon was warned about but not applied here, or the fetched file is up to 10 minutes old.
Does a Disallow rule keep a page out of Google?
Not by itself. Disallow blocks crawling, not indexing, so an address can be indexed though blocked and appear in search results if other pages link to it. To keep a page out, let crawlers fetch it and add a noindex rule, because a blocked crawler never sees it.
Does blocking GPTBot also block ChatGPT-User and OAI-SearchBot?
No. A crawler uses the group that names it, else the * group if any, so a file naming only GPTBot leaves OAI-SearchBot and ChatGPT-User on the * group. OpenAI says the GPTBot and OAI-SearchBot settings are independent, and that robots.txt rules may not apply to ChatGPT-User because users start those requests.
What happens if a site has no robots.txt, or the file returns an error or a redirect?
A 404 or other 4xx except 429 counts as no file, so nothing is restricted. A 5xx makes Google stop crawling for a while; we treat a 429 alike. Google follows at least five redirect hops, then treats the file as missing; this tester stops after five.
Can a crawler ignore robots.txt?
Yes. The file is a request that well-behaved crawlers follow, and it stops nobody from visiting an address. It is not an access control, and RFC 9309 notes that listing a path exposes it publicly. Anything private needs authentication, not a Disallow line.
Does this tester request the pages I list, and what happens to a pasted file?
No. Listed addresses are only compared with the rules and never requested. A live test requests only that site's robots.txt and reuses it for up to 10 minutes. Pasted text goes to our server to be tested, and no website is contacted.
How accurate is this robots.txt tester compared with Google's own tools?
It applies the rules Google publishes, which build on RFC 9309, with our own code, not Google's. Other crawlers can read the same file differently, and Google generally caches the file for up to a day. For Google's own view, use the Search Console robots.txt report.
We use cookies to personalize content and ads, to provide social media features and to analyze our traffic. You consent to our cookies if you continue to use our website.
Install SEO Site CheckerFree SEO tools - always in your pocket.
