How Google Crawls Websites: Discovery, Rendering and Indexing

Kendall Chris Kendall Chris Updated
dot shape
How Google Crawls Websites: Discovery, Rendering and Indexing

How Google Crawls Websites: The Four Stages

Quick answer: How Google crawls websites comes down to four stages. It finds URLs through links and sitemaps, Googlebot fetches each URL it is allowed to, the page is rendered in a recent version of Chrome, and indexing decides whether to store it. Crawling is not indexing, and Google promises neither.

Google crawling and indexing are separate jobs. Google's guide to how Search works names three stages, crawling, indexing and serving search results, and says not every page makes it through each one. We split out discovery and rendering because each can fail on its own. Serving, meaning ranking, is outside this post. For the wider picture of what SEO covers, see what SEO is and how it works.

StageWhat Google doesWhat can stop itWhere to look
URL discoveryFinds URLs in links on known pages, in sitemaps and from earlier visitsNothing links to the page and it is not in a sitemapURL Inspection, Discovery section
CrawlingGooglebot requests the URL after checking robots.txtA robots.txt rule, a login wall, server errors, slow responsesCrawl stats report; URL Inspection, Page fetch
RenderingRuns the page's JavaScript in headless Chromium and reads the resultBlocked scripts, scripts that fail, a non-200 statusURL Inspection, View crawled page
IndexingAnalyzes the page, clusters duplicates, picks a canonical, stores it in the search indexA noindex rule, a duplicate, low quality contentPage indexing report

The rest of this post follows how Google crawls websites row by row, then shows how to check each stage and which report rows relate.

How Google Finds New Pages: Links and Sitemaps

Google has no central registry of web pages, so it keeps looking for new and updated ones. It learns a URL from pages it already visited, from a link on a known page, or from a sitemap you submit. An orphan, a page nothing links to, is hard to find, and the internal links guide shows how to spot one.

Google can reliably follow only a link written as an <a> element with an href attribute. A span with an href, or an element that only runs an onclick script, may never be parsed.

A sitemap helps discovery and does not promise crawling or indexing. Google's test: you probably need one if the site is large, new with few external links, or heavy on video and images, and probably not at about 500 pages or fewer when everything is linked. Our XML sitemap guide covers the format, and the free XML sitemap generator builds the file.

One tip from practice: link each new page from the home page or a busy category page on publishing day. Google takes no payment to crawl more often.

What Googlebot Does When It Visits a URL

What is Googlebot? It is the Google web crawler, in two versions, Googlebot Smartphone and Googlebot Desktop. Both obey one robots.txt user agent token, so you cannot write a rule for just one. Google uses mobile-first indexing for most sites, so most requests come from the smartphone crawler.

How Googlebot works is easiest to see on one URL. It checks the robots.txt rules, makes the request and reads the HTTP status code:

  • 200: the content goes on to indexing, which is still not certain.
  • 301 or 308: a strong signal that the target should be processed; 302 or 307 is weak. Googlebot generally follows up to 10 redirect hops, so keep your redirects short.
  • 4xx other than 429: the content is ignored, and an indexed URL drops out over time.
  • 429 and 5xx: Google reads these as a server in trouble and slows down.

For most sites, Google says, Googlebot should come no more often than once every few seconds on average. It reads the first 2 MB of a supported file such as an HTML page and the first 64 MB of a PDF, fetching each CSS or JavaScript file separately under the same limit, measured on uncompressed data. In our reading, an ordinary article never nears 2 MB of HTML.

The Googlebot user agent is often faked, so confirm a visit with a reverse DNS lookup or Google's published IP ranges.

How Google Renders JavaScript Before Indexing

JavaScript rendering comes after the fetch. Googlebot queues every page that returns a 200 status for rendering, unless the page carries a robots meta tag or header saying not to index it. A page may sit in that render queue for a few seconds or longer, and then headless Chromium runs the scripts. Google reads the rendered HTML for links again and uses it for indexing, but it does not run JavaScript on blocked pages or from blocked files.

If your text and links already sit in the HTML the server sends, rendering adds little to worry about. If they appear only after scripts run, use View crawled page in URL Inspection to see what Google received. Google still recommends server-side or pre-rendering.

Our report also reads rendered HTML from headless Chromium, but it is not Google's renderer. URL Inspection has the final word.

robots.txt vs noindex: Crawling Control Is Not Indexing Control

robots.txt manages crawling, not indexing. A URL it blocks can still be indexed from links on other pages and then shows without a description, which Search Console labels Indexed, though blocked by robots.txt.

What you wantUseThe catch
Stop Googlebot fetching a URLA robots.txt Disallow ruleThe URL can still be indexed from outside links
Keep a page out of resultsA noindex meta tag or X-Robots-Tag headerGoogle must crawl the page to see it
Remove a page for goodA 404 status or a 410 statusGoogle may recheck the URL, less often over time
Merge duplicate URLsA 301 redirect or a canonical tagStrong signals, but Google makes the final pick
Keep a page privatePassword protectionThe other four secure nothing

Never combine them: a page disallowed in robots.txt is never fetched, so Google never sees its noindex rule. Use one or the other per URL, as Google's robots meta tag documentation says.

A missing robots.txt blocks nothing: on a 4xx status other than 429, Google assumes no crawl restrictions, as RFC 9309 from the Internet Engineering Task Force allows. On a 5xx, Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days.

See the robots.txt guide for syntax, the robots.txt generator to build one, and the noindex tag guide and canonical tag guide for the other controls.

Crawl Budget: Who Needs to Care and What Wastes It

Crawl budget is the set of URLs Google can and wants to crawl. The crawl capacity limit (hostload) is how much your server can take, and Google lowers the crawl rate when responses slow down or return 5xx errors or a 429 status. Crawl demand rests on site size, update frequency, page quality and duplicate URLs; each hostname has its own budget.

Most sites can ignore all this. Google aims its guide at sites with 1 million or more unique pages that change about weekly, 10,000 or more that change daily, or many URLs that Search Console labels Discovered, currently not indexed. Google calls these numbers a rough estimate, not exact thresholds. Everyone else needs a current sitemap and a look at the Page indexing report.

Waste comes from duplicate URLs, soft 404 pages that Google keeps crawling and long redirect chains, and faceted navigation is a common source of duplicates. Block unimportant URLs with robots.txt, not noindex, which still costs a request. Return 404 or 410 for removed pages, support HTTP caching with 304 Not Modified and keep server response time low.

Google ignores crawl-delay in robots.txt, and blocking pages there moves no budget to others unless the site is at its capacity limit. More in Google's crawl budget guide.

How to Check How Google Crawls Your Site

Google Search Console shows how Google crawls websites, and four steps cover most questions.

  1. Inspect one URL. Paste the full URL into the search bar at the top of Search Console to open the URL Inspection tool. Read Discovery, Last crawl, Crawled as, Crawl allowed?, Page fetch, Indexing allowed?, User-declared canonical and Google-selected canonical.
  2. Test the live page. Test live URL fetches the page now; View tested page, then More info, lists loaded resources and page code. The live test cannot predict canonical selection.
  3. Read the Page indexing report. Crawled, currently not indexed means Google fetched the page but has not indexed it, and it may or may not later; you need not resubmit. Discovered, currently not indexed means Google knows the URL but has not crawled it, typically to avoid overloading the site. Search Console prints both with a hyphen; we write them with a comma.
  4. Open Crawl stats. It sits under Property settings, for a Domain or root URL-prefix property only. It lists crawl requests, average response time, host status, file types and Googlebot type, so watch for rising 5xx responses. Google says sites with fewer than a thousand pages should not need it.

Request indexing in URL Inspection suits a handful of URLs and has a quota, and asking again does not speed it up.

What Our SEO Report Checks About Crawling and Indexing

The free SEO Site Checker report runs more than 60 checks, and several relate to the stages above. To try them on your own page, run the free SEO Site Checker report. A pass never proves that Google can crawl or index a page, so read the rows first.

Report rowWhat it testsWhat it does not test
robots.txt and Disallow DirectiveThe file answers 200 (401 or 403 warns, else fails); lists Disallow lines, warns if noneWhether a rule blocks Googlebot. A missing file fails, though Google assumes no restrictions
SitemapsWhether /sitemap.xml at the site root answers 200Other sitemap names, the Sitemap line in robots.txt, the URLs inside
NoIndex TagFails when the page's robots meta tag contains noindexAn X-Robots-Tag header, or a meta tag addressed to googlebot
Meta Canonical and URL CanonicalizationOne canonical link tag passes, several warn, none failsWhether the canonical points to the right page
URL RedirectsWarns when the address redirects before the final pageThe type or length of the chain
Custom 404 Error PagesRequests a made-up URL: a 404 page saying 404 or not found passes, a bare 404 warns, anything else, a 200 reply included, failsReal broken pages on your site
In-page LinksRequests the first 30 links and lists those that do not answer with a working pageOrphan pages, and links beyond 30

The report loads the page in headless Chromium, which identifies itself as HeadlessChrome, not Googlebot, and fetches robots.txt, the sitemap and the 404 probe with plain requests. A firewall that treats unknown bots differently can change the results, and a banner appears when the report only captured a bot-protection challenge page.

The AI Search Readiness section also checks whether robots.txt blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot from the whole site.

A Short Checklist Before You Ask Google to Recrawl a Page

Run through these six before you press Request indexing.

  • It returns 200. No redirect chain and no soft 404 in the way.
  • robots.txt allows it. The URL is not disallowed.
  • It has no noindex rule. Check the meta tag and the header.
  • Its canonical URL points where you want. Usually that is its own address.
  • A crawled page links to it. Use descriptive anchor text.
  • It is in the sitemap. Add a lastmod date if the content changed.

For the rest of the audit, work through our technical SEO checklist.

SEO

Frequently Asked Questions

Frequently Asked Questions (FAQs) is a list of common questions and answers provided to quickly address common concerns or inquiries.

How does Google crawl a website?

Google finds URLs through links on pages it knows and through sitemaps. Googlebot then fetches each URL it is allowed to, renders the page in a recent version of Chrome and hands the result to indexing. Google does not promise to crawl or index every page it finds.

What is the difference between crawling and indexing?

Crawling is fetching a page. Indexing is analyzing what was fetched, choosing a canonical and storing the page. A page can be crawled and still not indexed, which Search Console reports as Crawled, currently not indexed. Blocking crawling with robots.txt is not the same as blocking indexing.

How often does Google crawl a website?

There is no fixed schedule. Google says it may recrawl a news home page every few minutes and wait about a month on a page unchanged for years, and a sitemap can influence it. Your own numbers are in the Crawl stats report in Search Console.

How can I check whether Google has crawled my page?

Open the URL Inspection tool in Search Console and read Last crawl and Crawled as. For the whole site, use the Page indexing report and, on a root-level property, the Crawl stats report. A server log also shows Googlebot requests, but verify the visitor with a reverse DNS lookup.

How long does it take Google to index a new page or site?

Google says crawling may need a few days to a few weeks, and its Page indexing help says a new page or site can take about a week or so. Requesting indexing does not promise inclusion, and repeated requests do not speed it up. Links and a sitemap help discovery.

Can Google index a page that robots.txt blocks?

Yes. If other pages link to it, the URL can be indexed without a fetch and appears without a description. Search Console reports this as Indexed, though blocked by robots.txt. To keep a page out of results, use noindex on a crawlable page or password protection, and never combine noindex with a robots.txt block.

Does Googlebot run JavaScript?

Yes. Pages that return a 200 status are queued for rendering, and headless Chromium then runs their JavaScript, which can take seconds or longer. Google does not run JavaScript on blocked pages or from blocked files. Server-side rendering is still a good idea, and View crawled page in URL Inspection shows what Google received.

Do I need to worry about crawl budget?

Usually not. Google aims its crawl budget guide at sites with 1 million or more pages that change weekly, 10,000 or more that change daily, or many URLs that Search Console labels Discovered, currently not indexed. It calls those numbers rough estimates. Smaller sites should keep a sitemap current and watch the Page indexing report.
Kendall Chris
Written by Kendall Chris Kendall Chris

Kendal is an SEO specialist with 5+ years of experience helping small businesses and freelancers grow their organic traffic. She writes about on-page SEO, content strategy and website optimization at SEO Site Checker.

Share on Social Media: