What Is Crawl Budget? Do Small and Mid-Sized Sites Need to Care?

Crawl budget is the number of URLs Googlebot can and wants to crawl on a site in a given period. It has two parts: how much it can crawl without straining your server (the crawl rate limit) and how interested it is in your pages (crawl demand). According to Google, the topic matters mainly for very large sites with many pages; for most small and mid-sized sites it is not a problem.

Crawl budget is often blown out of proportion in the SEO world. Thinking that a corporate site of a few hundred pages is losing rankings because of crawl budget is usually a wrong diagnosis. On the other hand, for e-commerce sites that generate many unnecessary URLs, news sites or listing portals, the topic genuinely matters. This article explains the concept plainly, who should care, how to measure it and how to reduce waste.

What exactly is crawl budget?

Crawl budget is the balance of capacity and interest that determines how often and how much Googlebot crawls the pages on your site. Google does not have unlimited resources to crawl the whole internet, so it keeps prioritising how much time to give each site. The share your site receives is the result of that prioritisation.

Here two separate concepts need to be told apart: crawling and indexing. Not every crawled page is indexed, and not every indexed page ranks highly. Crawl budget relates only to the first step, Googlebot visiting the page. A page never being crawled is one reason it does not appear in the index, but it is not the only one.

• —Crawling: Googlebot visiting a URL and fetching its content.
• —Indexing: Google deciding to add the page to its search index.
• —Ranking: ordering the indexed pages for queries.
• —Crawl budget affects only the first step; the others are determined by content quality and relevance.

What are the two components of crawl budget?

According to Google, crawl budget consists of the crawl rate limit and crawl demand. The crawl rate limit is the amount of simultaneous connections and requests Googlebot can make without overloading your server. Crawl demand is how much Google wants to crawl your pages, determined by factors such as popularity and how stale the content is.

• —Crawl rate limit: can rise if your server responds quickly and healthily; if it slows down or errors increase, Google cuts crawling.
• —Crawl demand: high for popular, frequently updated and valuable pages; low for old, unchanging and low-value pages.
• —Site events: a major migration or URL structure change can temporarily raise demand.
• —Result: crawl budget is not a fixed number but a balance that shifts with circumstances.

The crawl rate limit is directly tied to server health. If the server responds slowly or returns errors often, Googlebot restrains itself. Server performance is therefore an indirect but important component of crawl budget. We cover page speed in a separate article.

Who does Google warn about crawl budget?

Google's crawl budget documentation states that the topic is mainly important for very large sites. By the approximate yardsticks in the documentation, that covers roughly sites with a million or more unique pages whose content changes moderately often, and sites with more than ten thousand unique pages whose content changes very frequently, for example daily.

A third signal is seeing many URLs in Search Console as “Discovered – currently not indexed”. This can show that Google knows the URLs but has not scheduled them for crawling. Remember, though, that this status has other common causes: weak content, duplicate pages or a slow server.

• —Very large site: hundreds of thousands or millions of unique URLs.
• —Very fast-changing site: news, listings, or sites where price or stock information is refreshed daily.
• —A sign of trouble: important pages not being crawled for months, or many URLs discovered but never crawled.
• —Size alone is not enough: even if the site is large, there is no need to worry about budget when there is no problem.

These yardsticks are approximate guidelines from Google's own documentation and may change over time; check the official documentation for the latest. If you are unsure whether your site falls into that scope, following the measurement steps in the sections below is the right path.

Is crawl budget a problem for small and mid-sized sites?

For most small and mid-sized sites, crawl budget is not a problem. A corporate site of a few hundred or a few thousand pages, with service pages and blog posts, is generally crawled by Google comfortably. On such sites, the reason pages do not enter the index is almost always not crawling but content value, duplication, a technical barrier or weak internal linking.

This honesty matters, because some sources sell crawl budget optimisation to small businesses unnecessarily. If your website belongs to a service firm, clinic, workshop or local business with a few hundred pages, spending your time on content quality, local signals and internal linking is more productive than on crawl budget.

• —A corporate site of a few hundred pages: generally no crawl budget issue.
• —A blog or catalogue of a few thousand pages: usually fine if the structure is orderly.
• —An e-commerce site generating thousands of filter URLs: a risk worth watching.
• —If a page is not entering the index: first check content, canonical, noindex and internal links.

What causes the most crawl waste?

Crawl waste is Googlebot spending time on URLs of little or no value. The biggest causes are usually filters that generate endless URL combinations, session and sorting parameters, duplicate content, long redirect chains and large numbers of 404 and soft 404 pages.

• —Faceted navigation: colour, size, price and sorting combinations produce many unique URLs.
• —Parameter URLs: session IDs, tracking parameters and sorting parameters serve the same content at different addresses.
• —Duplicate content: http/https, www/non-www, and versions with and without a trailing slash.
• —Redirect chains: chains from A to B and B to C use crawl at each step.
• —Error pages: many 404 and soft 404 URLs can keep being crawled persistently.
• —Infinite spaces: calendars, infinite scroll or unlimited pagination links.
• —Low-quality auto-generated pages: valueless tag, archive and search result pages.
• —Slow server: as response time grows, Googlebot crawls fewer pages.

A site may not have all of these problems; often one or two main sources consume most of the crawling. That is why the first step is measuring where crawling goes.

How do e-commerce filter URLs affect crawl budget?

On e-commerce sites, when filters for colour, size, brand, price range and sorting combine, a few hundred products can turn into thousands or even millions of unique URLs. If Googlebot tries to discover and crawl these combinations, the time it can devote to real product and category pages shrinks.

The solution is deciding which filter combinations are pages that genuinely have search demand and rich content. For instance, a separate page with unique content may make sense for a searched combination like “black leather jacket”, while a combination like “black, size 38, price ascending” should not be indexed. This distinction matters for both crawling and duplicate content.

• —Deliberately open to indexing the filter pages that have search demand and customised content.
• —Consider blocking crawling of other filter combinations with robots.txt or limiting internal links to them.
• —Tie sorting and session parameters to a single address with canonical.
• —Avoid offering filter links as crawlable links on every page when it is unnecessary.
• —Put only the canonical pages you want indexed in the sitemap.

These topics are central to e-commerce SEO. You can find product and category page optimisation in detail in the guide where we cover it separately. Every solution has risks; for example, blocking a pattern with robots.txt may also block important pages through a wrong rule. Always test before making changes.

How do you read the Search Console Crawl Stats report?

The “Crawl stats” report in the Settings section of Search Console shows Googlebot's recent requests to your site: total crawl requests, total download size, average response time, breakdown by response code, breakdown by file type and host status. It is the most accessible and free way to measure crawl budget.

• —Total crawl requests: sudden drops or rises in daily crawl volume are an important sign.
• —Average response time: a rising figure can indicate a server issue or overload.
• —Breakdown by response code: a large share of non-200 responses (redirects, 404, 5xx) is a sign of waste.
• —Breakdown by file type: be alert if most crawling goes to unnecessary resources rather than HTML.
• —Breakdown by purpose: is it mostly discovery of new pages or refresh of existing ones?
• —Host status: the history of robots.txt, DNS and server connectivity problems.

When reading the report, look at the general trend rather than a single day. A sudden rise in server errors, a temporary increase after a migration or a technical change you made can be seen clearly in the charts. The aim of this report is not to find a “budget number” but to understand whether crawling is healthy and purposeful.

When is log file analysis needed?

Server log files show line by line which URLs Googlebot requested, when and with what response code. While the Crawl Stats report offers a summary, log analysis shows which individual URLs were or were not crawled. On large sites and for complex problems log analysis is very valuable; on small sites it is often unnecessary.

When doing log analysis, use the verification methods Google documents to filter out fake bots impersonating the real Googlebot. Then list the most crawled URL patterns, important pages never crawled and addresses returning errors. A picture in which important pages are never crawled while unnecessary patterns are crawled constantly shows the optimisation opportunity clearly.

• —May be needed: sites with hundreds of thousands of URLs, complex filter structures, after large migrations.
• —Usually not needed: corporate sites of a few hundred pages.
• —What to look at: most crawled URL patterns, important pages not crawled, response code distribution.
• —Caution: log data may contain personal data; take care over privacy when sharing and storing it.

How do you reduce crawl waste? A step-by-step plan

To reduce crawl waste, first measure where crawling goes, then close the biggest source: stop generating unnecessary URLs, shorten redirect chains, give deleted pages the right status code, clean up the sitemap and speed up the server. The biggest gain usually comes from fixing the first one or two problems.

• —Step 1: Look at the response code and file type breakdown in the Crawl Stats report.
• —Step 2: Crawl your site with a crawler and list redirect chains, 404s and duplicate URLs.
• —Step 3: Cut redirect chains to a single hop; point internal links directly at the final address.
• —Step 4: Return 404 or 410 for pages that no longer exist and 301 for moved pages.
• —Step 5: Stop generating or linking to endless or low-value URL patterns (filters, sessions, search).
• —Step 6: Consider closing patterns you truly do not want crawled with robots.txt, and test it.
• —Step 7: Limit the sitemap to canonical, 200-response URLs you want indexed.
• —Step 8: Improve server response time; consider caching, a CDN and database optimisation.
• —Step 9: Monitor the Crawl Stats report for a few weeks after changes.

Note one point: a noindex tag does not save crawl budget, because Google must still crawl the page to see the noindex. Robots.txt blocks crawling but is not an indexing signal; a blocked URL that still receives external links can appear in the index with limited information. Each tool has a different purpose.

How do server speed and site structure affect crawl budget?

If your server responds quickly and stably, Googlebot can make more requests; if it is slow or erroring, Googlebot restrains itself. Server speed therefore gives a two-way benefit for both user experience and crawl capacity. In site structure, too, a flat, understandable hierarchy and strong internal linking make it easier to discover important pages.

• —Response time (TTFB): reducing time to first byte eases crawling too.
• —Caching and CDN: speeds up repeated requests and reduces server load.
• —Flat site structure: important pages should be reachable from the homepage in few clicks.
• —Internal linking: orphan pages make discovery harder; every important page should be linked from at least one strong page.
• —Error rate: 5xx errors and timeouts directly restrict crawling.

If you want a lasting fix for server problems, tackle speed on both the page side and the server side. Our articles on Core Web Vitals and page speed will support you here.

What are the common misunderstandings about crawl budget?

The most common misunderstandings are that crawl budget is critical for every site, that more crawling brings higher rankings and that a specific “budget number” can be found. In reality more crawling does not automatically mean better rankings; Google runs crawling and ranking as separate processes.

• —“Every site must optimise crawl budget”: according to Google's documentation this is mainly for large sites.
• —“More crawling means better ranking”: crawling is not a ranking factor.
• —“If a page is not indexed, the budget must be insufficient”: the cause is usually content value, duplication or a technical barrier.
• —“noindex saves crawl budget”: Google must still crawl the page to see the noindex.
• —“Crawl-delay works for Google”: Googlebot does not support the crawl-delay directive.
• —“Every URL in the sitemap gets crawled”: a sitemap is a hint, not a guarantee.

Because of these misunderstandings, some sites spend time on unnecessary technical work while ignoring the real issues, namely weak content and missing internal links. The honest approach is to diagnose first, then set priorities accordingly.

Does your site have a crawl budget problem? A quick checklist

To tell whether your site has a real crawl budget problem, look for answers to a few simple questions: how many unique URLs does your site have, how do important pages appear in crawl stats and the index report, and is there unnecessary URL generation? For most small sites, the answers show no problem.

• —Is your number of unique, valuable URLs in the hundreds to few thousands, or does it exceed hundreds of thousands?
• —In Search Console, are many important pages in the “Discovered – currently not indexed” status?
• —In crawl stats, is a large share of response codes non-200 (redirects, 404, 5xx)?
• —Do filter, session or search parameters generate thousands of URLs?
• —Are there redirect chains, duplicate versions (http/https, www) and soft 404s?
• —Is server response time consistent and fast?
• —Are newly added pages discovered and crawled within a reasonable time?

If your answers are mostly “no”, crawl budget is not a priority for you; spend your energy on content, internal linking and local visibility. If some items come out “yes”, start with the steps above. At 404 Ajans we follow this order in technical audits: measurement first, then intervention only where there is a genuine problem.