Crawl budget refers to the number of URLs Googlebot (or other search engine crawlers) will fetch from your site in a set period. For most small-to-medium sites, crawl budget is rarely a problem — but for large sites with thousands of pages, mismanaging it can mean important pages go unindexed.
Two Components
- Crawl rate limit — how fast Googlebot can crawl without overwhelming your server. Google adjusts this based on your server’s response times and the crawl limit you set in Search Console.
- Crawl demand — how much Google wants to crawl your site, driven by popularity (backlinks, traffic) and staleness (how often content changes).
When Crawl Budget Matters
Crawl budget becomes a concern when you have:
- Thousands of low-quality or duplicate URLs (faceted navigation, session IDs, print versions)
- Pages blocked incorrectly by robots.txt or noindex
- Very large sites where new or updated content is slow to get indexed
How to Optimise Crawl Budget
- Block low-value URLs via robots.txt (admin pages, search result pages, duplicate parameters)
- Use canonical tags to consolidate duplicate content
- Keep your sitemap clean and up to date
- Reduce redirect chains — each hop costs crawl budget
- Improve server response times (faster servers = higher crawl rate limit)