Crawl budget is the number of pages a search engine is willing to fetch from your site in a given period. It comes from two things: the load your server can take, and the interest the engine has in your site.
Who does it affect?
On a site with a few hundred pages, crawl budget is not a problem. It becomes one when:
- You run an e-commerce site with tens of thousands of pages
- Filter combinations generate URLs
- Content changes constantly, as on news and listing sites
- Your server is slow
What burns the budget
- Parameterised URLs. Sort, filter and session parameters create infinite variations.
- Redirect chains. Every hop is a separate crawl request.
- Soft 404s. Pages that say "product not found" but return 200.
- Duplicate content. The same content reachable from five URLs.
- Low-value pages. Tag archives, empty categories, print versions.
What log analysis reveals
Server logs are the only source that shows what search bots actually do. Search Console gives you a summary; logs give you raw truth.
Questions worth asking:
- Which pages does the bot crawl most? Are your most valuable pages on that list?
- What percentage of crawl requests return something other than 200?
- How much of total crawling do parameterised URLs consume?
- When were your important pages last crawled?
- Which pages are AI crawlers (GPTBot, PerplexityBot) fetching?
A simple starting point
Filtering Googlebot lines out of your Apache or nginx access log and counting the most-crawled paths surfaces most problems. You do not need a specialised tool; a few lines of shell will do.
Improving the budget
- Exclude worthless URL patterns in
robots.txt - Collapse redirect chains to a single hop
- Restrict your sitemap to indexable URLs that return 200
- Reduce server response time — faster sites get crawled more
- Increase internal links to your important pages
The common misunderstanding
Increasing crawl budget does not raise rankings. The goal is to spend the existing budget on the right pages. A bot crawling a thousand pages a day achieves nothing if those thousand pages are filter combinations.