hi, i’m amr. “optimize your crawl budget” shows up on audit reports for sites with a few hundred pages. google’s own guide says who it is written for, and it gives rough numbers.
what google says, in its own words
- the guide is for “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”
- and for “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”
- it also names sites where many urls show as “Discovered – currently not indexed” in search console
- the page calls its numbers “a rough estimate to help you classify your site,” not exact thresholds
- crawl budget is “the set of URLs that Google can and wants to crawl,” built from a crawl capacity limit and crawl demand
- for permanently removed pages, “Return a 404 or 410 status code.” soft 404 pages “will continue to be crawled, and waste your budget”
- on noindex: “Don’t use noindex, as Google will still request, but then drop the page”
my reading, not google’s wording: if a 300-page b2b site is slow to get indexed, the guide points you away from crawl budget and toward the page indexing report.
what to check this week
- count the unique indexable urls on your site, so you know which side of the 10,000 line you are on
- open the page indexing report and look at “Discovered – currently not indexed” and soft 404 rows
- make sure pages you removed on purpose return 404 or 410, not a 200 with an empty template
- cut long redirect chains and keep the sitemap current. i covered what google reads from sitemaps in an earlier post
- do not add noindex to save crawl time, since the page says google still requests the url
where this stops applying
the page says nothing about crawl rate affecting rankings, so this post makes no ranking claim. its numbers are rough estimates, not limits, and it gives no figure for how many requests per day a site gets. i haven’t audited any site, so treat the list above as a starting point.
start scaling — if your site is closer to 10,000 pages than 300 and new pages take weeks to appear, a short indexing review will show whether crawling is the cause. let’s connect.
Sources: Google Search Central, “Large site owner’s guide to managing your crawl budget” (developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget). Figures are published product documentation, not verified results from Amr’s own client accounts.