Crawl Budget Explained: When It Matters and How to Fix It
By Kavi, SEO Consultant · Last updated: 3 September 2026
Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given period. It is the product of two things: crawl capacity, which is how much your server can handle, and crawl demand, which is how much Google wants your content. For most sites under a few thousand pages, crawl budget is not a real constraint and should not be optimised for.
What is crawl budget?
Crawl budget is Google’s shorthand for how much crawling your site gets. Google defines it as the combination of two independent factors:
- Crawl capacity limit. How many simultaneous connections Googlebot can use without degrading your server. If your server slows or returns errors, Google backs off automatically.
- Crawl demand. How much Google actually wants to crawl, based on your site’s popularity, how often content genuinely changes, and its perceived quality.
Both must be present. A fast server with no demand gets little crawling, and high demand on a slow server gets throttled.
I want to be blunt about this, because crawl budget attracts more wasted effort than almost any other technical SEO topic. Google has repeatedly said that most sites do not need to worry about it. If your site is under a few thousand URLs and your pages get indexed within a reasonable time, your crawling is fine and the problem you are chasing is elsewhere.
Quick facts
| Item | Detail |
|---|---|
| Two components | Crawl capacity limit and crawl demand |
| Matters for | Large sites, roughly 10,000+ URLs, and fast-changing sites |
| Does not matter for | Most blogs, small business and brochure sites |
| Is it a ranking factor? | No. Crawling is a prerequisite for ranking, not a ranking signal. |
| Where to measure | Search Console Crawl Stats report; server log files |
| Biggest single waste | Faceted navigation and URL parameters |
| Fastest lever | Server response time |
When does crawl budget actually matter?
Crawl budget becomes a genuine constraint in a small number of situations:
- Large ecommerce sites where filter combinations generate hundreds of thousands of URLs.
- Sites over roughly 10,000 pages where new content is slow to be discovered.
- News and job boards where content must be crawled within hours to have value.
- Sites with heavy URL parameter usage, such as session IDs or sort orders.
- Sites on slow or unreliable hosting, where Google throttles itself defensively.
Diagnostic rule I use: before optimising crawl budget, publish a new page, link it from your homepage, and see how long it takes to be crawled. If it is crawled within a day or two, you do not have a crawl budget problem. Pages that are crawled promptly but not indexed have a content quality or duplication problem, and no amount of crawl optimisation will fix that.
How does crawl budget work?
- Google maintains a queue of known URLs for your site, gathered from links, sitemaps, and past crawls.
- It assigns a crawl rate based on how your server responds. Fast, stable responses earn a higher rate; timeouts and 5xx errors reduce it quickly.
- It prioritises the queue by demand: popular pages, frequently updated pages, and pages Google believes are valuable get crawled more often.
- Waste consumes the allocation. Every crawl spent on a duplicate, a redirect chain, or a soft 404 is a crawl not spent on real content.
What wastes crawl budget
| Waste source | Why it consumes crawling | Fix |
|---|---|---|
| Faceted navigation | Filter combinations multiply into millions of URLs | Disallow parameter patterns in robots.txt |
| Session IDs and tracking parameters | Every visit can create a new crawlable URL | Self-referencing canonicals; avoid parameters in links |
| Internal site search results | Effectively infinite thin pages | Disallow the search path |
| Redirect chains | Each hop is a separate fetch | Collapse to one hop |
| Soft 404s | Google crawls and re-crawls empty pages | Return real 404 or 410 |
| Duplicate content | Multiple fetches of the same content | Canonical tags and server-level consolidation |
| Thin tag and date archives | Large numbers of near-empty pages | Noindex, or remove entirely |
| Slow server responses | Google reduces its own crawl rate | Caching, better hosting, database tuning |
| Broken internal links | Crawls spent on 404s | Fix links at the source |
| Infinite calendar or pagination | Endlessly generated URLs | Cap pagination; block calendar navigation |
How to diagnose crawl budget problems
Step 1: Read the Crawl Stats report
In Search Console, go to Settings, then Crawl Stats. Look at three things: total crawl requests over time, average response time, and the breakdown by response code and by file type. A rising crawl count with flat indexing means waste. Rising average response time means you are approaching a capacity limit.
Step 2: Check the response code breakdown
A healthy site is dominated by 200s. If a large share of crawls return 301, 404, or 5xx, that is your budget leaking. Any meaningful volume of 5xx responses is urgent, because Google will reduce your crawl rate in response.
Step 3: Compare crawled against indexed
In the Pages report, look at “Crawled, currently not indexed” and “Discovered, currently not indexed”. Large numbers under Discovered suggest Google knows about pages but is choosing not to fetch them, which points at crawl demand or capacity. Large numbers under Crawled suggest a quality problem, not a crawl problem.
Step 4: Read your server logs
Logs are the only complete record of what Googlebot actually requested. They reveal which URL patterns absorb crawling, which sections are neglected, and whether crawl activity matches your commercial priorities. Sample a week and group requests by directory.
How to improve crawl efficiency
Step 1: Speed up the server
This is the highest-leverage change because crawl capacity is directly tied to response time. Caching, a CDN, and reducing time to first byte all raise the ceiling. It also benefits users and your Core Web Vitals, so the work pays twice.
Step 2: Block the genuine waste
Disallow filter parameters, internal search, and cart paths in robots.txt. Be precise: an over-broad pattern will block real pages. Test each rule against live URLs before deploying.
Step 3: Fix redirect chains and errors
Crawl your own site, export every non-200 response, and fix them in order of internal link count. Chains and 404s reached from your own navigation are the cheapest wins available.
Step 4: Tighten internal linking
Crawl priority follows internal links. Pages three clicks from the homepage are crawled far more often than pages eight clicks deep. Flattening your architecture, as described in the internal linking 3-link formula, redirects crawl attention towards the pages you care about.
Step 5: Keep your sitemap clean and lastmod honest
An accurate lastmod in your XML sitemap helps Google prioritise recrawls of genuinely changed pages. A sitemap full of redirects and noindexed URLs does the opposite.
Step 6: Remove or consolidate low-value pages
Crawl demand is partly a quality judgement. Sites with a high proportion of thin pages get crawled less. Pruning or merging weak content raises demand for what remains.
What does not help
- Crawl-delay in robots.txt. Google ignores it entirely.
- Nofollow on internal links. The value is discarded, not redirected. See noindex vs nofollow vs disallow.
- Priority and changefreq in sitemaps. Google ignores both.
- Repeatedly requesting indexing. It does not raise your allocation.
- Noindex to save crawl budget. Noindexed pages are still crawled. Only disallow stops the fetch.
Decision guide
- Under 1,000 pages, content indexes within days: do nothing. Spend the time on content.
- Under 10,000 pages, some pages never indexed: this is a quality or internal linking problem, not crawl budget.
- Over 10,000 pages with parameter-driven URLs: audit facets and block the patterns. This is the real case.
- High 5xx rate in Crawl Stats: fix hosting first; nothing else matters until that is stable.
- News or job listings needing same-day crawling: optimise server speed, use accurate lastmod, and consider IndexNow for Bing.
Crawl efficiency checklist
- Crawl Stats shows a stable or falling average response time
- 5xx responses are near zero
- 200 responses dominate the crawl mix
- No redirect chains reachable from internal links
- Faceted and parameter URL patterns are disallowed
- Internal search results are blocked
- Sitemap contains only canonical, indexable, 200-status URLs
- lastmod values are accurate
- Important pages are within three clicks of the homepage
- Thin archives are noindexed or removed
- Server logs reviewed at least quarterly on large sites
Frequently asked questions
Is crawl budget a ranking factor?
No. Crawling is a prerequisite for indexing and ranking, but the amount of crawling you receive is not itself a ranking signal. Improving crawl efficiency helps content get found faster; it does not directly raise positions.
How many pages before crawl budget matters?
Google has indicated most sites need not worry. As a practical threshold, start paying attention above roughly 10,000 URLs, or sooner if faceted navigation is generating URLs faster than you publish content.
Does noindex save crawl budget?
No. Google still has to crawl a page to read the noindex tag. Over a long period it will crawl those pages less often, but only a robots.txt disallow prevents the fetch outright.
Does blocking pages in robots.txt free up budget for other pages?
Yes, in the sense that crawls not spent on blocked patterns become available elsewhere. This is the main legitimate use of disallow on large sites.
Can I increase my crawl rate?
Indirectly, by improving server response time and by raising demand through better content and more external links. You cannot request a higher allocation directly.
Why is my new content not being crawled?
On a small site, almost never crawl budget. Check that the page is linked internally, present in the sitemap, and not blocked. Discovery failures are usually linking failures.
Key takeaways
- Crawl budget is crawl capacity multiplied by crawl demand.
- Most sites under 10,000 URLs do not need to optimise for it.
- It is not a ranking factor.
- Faceted navigation and URL parameters are the dominant source of waste.
- Server response time is the fastest lever on capacity.
- Noindex does not save crawling; only disallow does.
- Test with a new page before assuming you have a crawl problem.
Conclusion
Crawl budget is real but over-diagnosed. Run the simple test first: publish, link, and time how long discovery takes. If it is fast, your effort belongs in content and internal linking. If it is slow and you have tens of thousands of parameter-driven URLs, then the fixes here, faster responses and blocked waste patterns, are the ones that move the number.
Related articles
- Technical SEO guide
- Robots.txt explained
- Log file analysis for SEO
- Pagination and faceted navigation
- Site architecture and URL structure
