Free 20-point conversion & SEO audit for export stores — Free website audit — claim yours

hello@woo.one Mon–Fri · 9:00–18:00 (UTC+8)

Crawl Budget on WooCommerce Stores: Signs, Waste and Fixes

How to tell whether crawl budget is actually your problem, where it goes on a typical store, and the fixes that work in order — including tag policy and historical URLs.

Crawl budget is only a real constraint on large sites, and most stores that worry about it are not large enough for it to matter. But when it does matter — thousands of SKUs, many languages, frequent stock changes — the symptoms are specific and the fixes are structural.

Are you actually affected?

Three signs, in order of reliability:

  • New products take weeks to be indexed while existing pages are crawled daily
  • Crawl stats show mostly low-value URLs — filters, sorting parameters, tag pages, internal search results
  • Indexed page count is far below the number of real pages, while crawled count is high

A store with 200 products does not have a crawl budget problem. It has a different problem, and applying crawl-budget fixes to it wastes effort.

Where the budget actually goes

In rough order of waste on a typical WooCommerce store:

  1. Facet combinations — covered in the faceted navigation notes
  2. Sorting and pagination parameters
  3. Tag archives, which accumulate over years and are rarely useful
  4. Internal search result pages
  5. Author archives, date archives, attachment pages
  6. Legacy URLs from previous platforms, still returning 200 with similar content

Items 3 to 5 share a property: they are features WordPress enables by default and a shop does not need. Disabling them is usually a net gain.

Fixes that work, in order

1. Disable what a shop does not need. Author archives and date archives on a store with one author and no editorial calendar are pure waste. Attachment pages are worse: they create a page for every image with no value to anyone.

2. Decide tag policy. Tags can be useful related-content signals or hundreds of thin archives. Pick one: either maintain a small curated set, or noindex them entirely. The broken middle state — thousands of indexable tags nobody curates — is the worst option.

3. Remove parameters from crawl paths. Canonical sorting variants, keep facet rules from the previous article. If a URL has no inbound crawlable link and no value, it should not be in a sitemap or a menu.

4. Clean up historical URLs. Any URL that returns 200 but duplicates another page is a candidate for a canonical or a redirect. A migration that left the old paths alive as duplicates is a common inheritance.

5. Make the sitemap accurate. It should contain indexable URLs only. A sitemap with 40,000 entries where 3,000 are real pages signals that the site is larger and messier than it is.

The priority question: products or content

On constrained crawlers, new blog content competes with product pages for attention. This is worth stating plainly because it affects editorial planning.

The usual recommendation is that product and category pages come first — they convert — while content supports them. But on a multilingual store the multiplication is severe: eight languages times every URL. If crawl is constrained, translate the commercial pages and leave the blog in one language. Content in six languages that gets crawled monthly is worth less than content in one language that gets crawled daily.

What not to do

Do not touch crawl rate settings. Search engines adjust crawl rate based on server response and site quality. Forcing a higher rate on a slow server produces errors, not faster indexing.

Do not block CSS or JavaScript. Modern crawlers render pages; blocking assets makes them see a broken layout and reduces confidence in the page.

Do not noindex and expect instant removal. Noindex pages stay in the index for a while. If a page must be removed quickly, a 410 is faster.

Do not add thousands of URLs to “request indexing”. That tool is a nudge for a specific page, not a crawl accelerator.

Measuring whether it worked

Track these monthly, and expect movement over six to twelve weeks rather than days:

  • Crawled URLs per day, and the share that return 200 versus 404/301
  • Time from publishing a product to it being indexed
  • Indexed page count versus the count in your sitemap
  • Pages in “crawled – currently not indexed”

The third number is the one to present to a business owner: if the sitemap says 3,000 and the index says 800, either the sitemap is wrong or the pages are not worth indexing. Both are fixable, but only after you know which it is.

Most of this work is configuration and cleanup rather than new development, which is why it tends to pay back quickly on stores that have never audited it. It sits in the same audit as the technical SEO work, and the crawl data is usually the first artefact we produce.

S

Infrastructure and security engineer at woo.one, based in Foshan, China. Works on WordPress, WooCommerce and OpenCart systems, self-hosted mail, and the operational side of running stores for exporters.

More about the team

No pressure · No sales script

Ready for a store that sells while you sleep?

Send us your current site or product list. You will receive a fixed-scope proposal, a 20-point audit and a realistic timeline within two business days.

Reply within 1 business day NDA on request 100% code ownership
Chat with us
WhatsApp Get a quote