Crawl budget is the number of pages search engines will crawl on your website within a given timeframe. Google calculates this budget based on two factors: crawl capacity limit (how much your server can handle without errors) and crawl demand (how much Google wants to crawl your site). For large websites with thousands of pages, managing this budget ensures search engines discover your important content instead of wasting resources on duplicates or errors.
What is Crawl Budget?
Crawl budget represents the set of URLs that Google can and wants to crawl. It applies not only to HTML pages but also to JavaScript files, CSS, PDFs, hreflang variants, and mobile page variants. Google assigns separate crawl budgets to different hostnames, meaning https://www.example.com/ and https://code.example.com/ are treated independently.
The term is sometimes referred to as crawl space or crawl time. Crawl budget is determined by two main elements: crawl capacity limit and crawl demand (Google Developers).
Why Crawl Budget Matters
If Google does not crawl a page, it cannot index it or rank it in search results. When waste reduces your effective budget, critical pages may go undiscovered.
- Indexation coverage: Large sites risk having important pages remain unindexed if crawlers spend their time on irrelevant URLs.
- Speed of updates: Fresh content and updates get indexed faster when crawlers focus on valuable pages. One site upgrade focused on load speed resulted in Google crawling jumping from 150,000 URLs per day to 600,000 URLs per day (Conductor).
- Server efficiency: Optimizing crawl patterns prevents unnecessary server load from bot traffic hitting broken or duplicate pages.
- SEO performance: Wasted budget on low-quality content directly reduces the visibility of your indexable pages in search results.
How Crawl Budget Works
Google defines your budget by weighing server capacity against crawl interest.
Crawl Capacity Limit
This is the maximum number of simultaneous parallel connections Google uses to crawl your site, plus the time delay between fetches. The limit protects your server from overload. It fluctuates based on:
- Crawl health: Fast server responses increase the limit; slow responses or server errors decrease it.
- Google's resources: Google has finite crawling machines, so limits apply globally.
Crawl Demand
This reflects how much Google wants to crawl your URLs. Key factors include:
- Perceived inventory: The total URLs Google knows about. Duplicates or removed pages that remain accessible waste budget.
- Popularity: URLs with more internal and external links tend to be crawled more frequently.
- Staleness: Frequently updated content gets recrawled more often than static pages.
Even if capacity exists, low demand means Google crawls your site less.
Best Practices
Manage your URL inventory. Consolidate duplicate content using canonical tags. Block unimportant pages (such as filtered faceted navigation or internal search results) using robots.txt. Return 404 or 410 status codes for permanently removed pages rather than letting them return soft 404s.
Improve site speed. Faster loading allows Googlebot to crawl more pages within the same timeframe. Making a site faster improves the user experience while also increasing crawl rate (Backlinko).
Structure internal links strategically. Use a flat architecture where important pages sit few clicks from the homepage. Ensure every valuable page has at least one internal link pointing to it to avoid orphan pages.
Maintain clean XML sitemaps. Include only indexable URLs you want in search results. Update sitemaps regularly and use the <lastmod> tag for changed content. Remove non-indexable pages (3xx, 4xx, 5xx responses) from sitemaps.
Eliminate redirect chains. Long chains slow down crawlers and waste budget. Google follows a maximum of five chained redirects in one crawl attempt.
Monitor crawl stats. Check Google Search Console's Crawl Stats report to see requests per day, response codes, and host status issues. Compare this against your server logs for accuracy.
Common Mistakes
Mistake: Allowing infinite URL parameters (e.g., filter options) to create crawlable pages.
Fix: Block parameter-based URLs in robots.txt or use Google Search Console's URL parameter tool. Add nofollow to filter links.
Mistake: Returning soft 404s (200 OK status for missing pages).
Fix: Ensure removed pages return genuine 404 or 410 status codes so Google stops crawling them.
Mistake: Creating orphan pages with no internal links.
Fix: Link to every important page from your navigation or relevant content pages.
Mistake: Including incorrect URLs in XML sitemaps.
Fix: Audit sitemaps regularly to remove redirects, error pages, and non-indexable URLs.
Mistake: Hosting large sites on shared servers.
Fix: Move to dedicated hosting; shared hosts split crawl limits across all sites on the server, severely restricting your allocation.
Examples
Case Study: Authority Building
Google's Matt Cutts noted that "the number of pages that we crawl is roughly proportional to your PageRank" (Conductor). A site that earned significant high-quality external links saw its crawl budget increase proportionally, allowing deeper pages to be discovered.
Case Study: Technical Optimization
A technical SEO audit of one website revealed that only 7.4% of discovered URLs were actually indexable. By cleaning up non-indexable pages from XML sitemaps, fixing broken links, and consolidating duplicate content, the site recovered significant crawl budget for its valuable pages.
Case Study: Velocity Improvement
One website improved its load speed and saw daily crawled URLs jump from 27 pages to 253 pages over two years (Conductor). This 10X increase allowed new content to appear in search results within hours rather than days.
FAQ
What is crawl budget?
Crawl budget is the number of pages a search engine will crawl on your site within a specific timeframe. It is determined by crawl capacity limit (server handling ability) and crawl demand (Google's interest in your content).
How do I check my crawl budget?
Use Google Search Console's Crawl Stats report (found under Settings > Crawling) to see total requests over the last 90 days. Compare this data against your server logs to verify accuracy.
Do small websites need to worry about crawl budget?
No. Google recommends that only large sites (1 million+ pages) or medium sites (10,000+ pages) with rapidly changing content need to actively manage crawl budget (Google Developers). Smaller sites should focus on general technical SEO health.
What wastes crawl budget?
Duplicate content, soft 404 errors, broken links, redirect chains, URL parameters generating infinite pages, and non-indexable URLs in XML sitemaps all waste crawl budget.
How do I increase my crawl budget?
Increase your site's authority by earning quality backlinks, improve server response times to raise crawl capacity limits, and eliminate low-quality content that reduces crawl demand.
What is the difference between crawl budget and crawl rate?
Crawl budget refers to the total number of pages crawled over time. Crawl rate refers to the speed at which Googlebot makes requests (requests per second). Both are affected by server health.
