SEOSiteCheckerBot
SEOSiteCheckerBot is the user agent our address-checking tools send when a visitor asks us to check an address. See what it requests, how often it visits and how to block it.
Free to use 6 min read Updated
What SEOSiteCheckerBot is
Quick answer: SEOSiteCheckerBot is the user agent our address-checking tools send when a visitor types your address into one of them. It requests that address, the redirects it leads to and, for the broken link checker, a limited number of links found on that page. The XML sitemap generator follows internal links across your website, up to a set number of pages. It never starts on its own.
seositechecker.pro offers free website tools, for example tools that follow redirects, show the robots directives of a page or look for broken links. When someone types a web address into one of them, our server fetches that address so it can show the result. SEOSiteCheckerBot is the name it uses while doing that, so the visit appears in your logs under a name you can look up and, if you wish, block.
It is not a search engine crawler. It does not build an index, it does not follow links on its own, and it does not go looking for addresses on its own. A few other tools on the site send a different user agent, and the section on when it visits says which.
The user agent string
Every request from SEOSiteCheckerBot, including each redirect it follows, sends this user agent:
SEOSiteCheckerBot/1.0 (+https://seositechecker.pro/page/seositechecker-bot)The token to look for in your logs is SEOSiteCheckerBot. The address in brackets leads to this page. When you see the token, a person asked one of our tools to look at your address. The one exception is a line whose user agent also ends with the word internal-research, which the section below describes.
When it visits
SEOSiteCheckerBot runs when a person submits an address to one of our tools, with one exception described below the list. There is no schedule, no list of websites to revisit and no discovery of new sites. What it requests depends on the tool.
- Tools that look at one address. They request the address the visitor typed and the redirects it leads to, up to ten hops. A tool that tests your robots.txt file, or checks whether a page is blocked from search engines, also requests the robots.txt file of the website the visitor typed.
- The broken link checker. It requests the page the visitor typed, reads the links on that page and then requests a limited number of those links to see whether they still work. It sends a HEAD request for each link and, when a website answers that badly, a GET request, one request at a time and at least a quarter of a second apart on the same website. It does not follow the links it finds any deeper than that.
- The XML sitemap generator. It starts at the address the visitor typed and the home page of that website, follows the links to other pages of the same website and requests each page once, up to 600 pages for a guest and 2,000 for a signed in visitor, or about 15 minutes, whichever comes first. It reads robots.txt and the sitemap files of the website, the ones robots.txt names and the ones at the usual addresses, and it starts two requests at least a quarter of a second apart.
One run that no visitor started. On November 18, 2026 we ran it ourselves to choose the sites for a study of popular home pages. We requested the https home page of each of the 172 highest ranked domains on the Tranco list, trying the www name as well when the bare name gave no usable page, with half a second between websites, redirects followed and robots.txt not read. Those requests added the word internal-research to the end of the user agent string. We then audited the 100 websites we kept once each with our SEO audit, which sends an ordinary browser user agent instead of this one.
Other tools do not send this token. The SEO audit and the website screenshot tool open the page in a real browser engine so they can show it the way a visitor sees it, and they send an ordinary browser user agent because the page has to behave the way it does for people. Some of our older tools also identify themselves differently.
How often it can visit
The tools that send this user agent share limits that keep one visitor, or one website, from being hit hard through us.
| What | Limit |
|---|---|
| Runs of tools that look at one address | Up to 20 a minute against one website, counted across every visitor and every tool that looks at one address |
| The broken link checker | One request at a time and at least 250 milliseconds between two requests to one website, which keeps it at or below about 60 requests a minute to one website |
| The XML sitemap generator | One request at a time, two requests started at least 250 milliseconds apart, which keeps it at or below 4 requests a second to one website, for about 15 minutes a run |
| Redirects followed in one run | Up to 10 |
| Request methods | GET and HEAD only |
| Connections | http and https, on ports 80, 443, 8080 and 8443 only |
What it does with the pages it fetches
The tools that send this user agent show the result to the visitor who asked for it. They do not add the pages they fetch to an archive or a search index. To avoid asking your server twice for the same thing, a result can sit in a short-lived cache and is then discarded. SEOSiteCheckerBot does not log in, submit forms, run scripts or change anything on your site.
The SEO audit works differently. It runs in a real browser and keeps its result, with screenshots of the page, as a report page on this website.
robots.txt and SEOSiteCheckerBot
A tool that fetches one address a visitor typed does not use robots.txt to decide whether to fetch it. A person asked for that exact address, which is the same as opening it in a browser. Some of these tools do read the file, but only to show the visitor what it says. The broken link checker and the XML sitemap generator are different, because they contact addresses they find on pages rather than only the one that was typed. They obey the robots.txt file of the website they read and skip a page that robots.txt disallows for the SEOSiteCheckerBot token. The sitemap generator also answers to the older name SitemapBot, waits as long as a Crawl-delay line asks, up to 10 seconds, and stops if the file cannot be read because the server answers with an error. The broken link checker reads that file for the website a link points at, not for a different website a redirect may lead to. The rules in that file follow the Robots Exclusion Protocol.
How to block SEOSiteCheckerBot
Pick the method that fits your setup. The first affects the broken link checker and the XML sitemap generator, the tools that obey robots.txt. The second stops every tool that sends this user agent.
Option 1: add a robots.txt group
Put these two lines in the robots.txt file at the root of your website.
User-agent: SEOSiteCheckerBot
Disallow: /Option 2: block the user agent at your server
Match the token SEOSiteCheckerBot and answer with a 403 status. On Apache, add this to your .htaccess file.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} SEOSiteCheckerBot [NC]
RewriteRule .* - [F,L]On nginx, add this inside the server block that serves your website.
if ($http_user_agent ~* "SEOSiteCheckerBot") {
return 403;
}Blocking affects only checks that visitors run on your website through the tools that send this user agent. To ask us to stop every check of your website, including the ones that use a browser, write to us through the contact page below. Blocking has no effect on how search engines crawl or rank your website, because they use their own user agents.
How to reach us
If you think SEOSiteCheckerBot is misbehaving, or you want to ask about a visit, write to us through the contact page. Include your website address and the lines you saw in your logs.
Frequently Asked Questions
Is SEOSiteCheckerBot a search engine crawler?
Why did SEOSiteCheckerBot visit my website?
How do I stop SEOSiteCheckerBot from visiting my website?
Does SEOSiteCheckerBot ignore robots.txt?
See how your site scores right now
Paste your address into the free report, then use the How to Fix button on anything that needs work.
Free to use. Results in under a minute for most sites.