Every large website has more URLs than pages. Filters, sort orders, tracking parameters, pagination, tag combinations, and older product variants multiply a modest content inventory into a six-figure URL space. Most of those URLs were never designed; they are byproducts of how the application constructs links.
That is good news. Byproducts follow patterns, and patterns can be fixed in batches. The unit of work in a technical SEO cleanup should never be "the URL" — it should be "the URL pattern".
What counts as a URL pattern
A pattern is a set of URLs that share a structural shape: the same path template, the same parameter set, and usually the same status code, canonical behaviour, and robots signals. For example:
/products?sort=*— every sorted variant of a product listing./category/page/*— pagination across categories./tag/*— auto-generated tag archives./old-product/*— retired product URLs still returning 200.
Grouping by pattern converts a spreadsheet problem into an editorial problem. There are usually fewer than 30 meaningful patterns on a site, even when there are 200,000 URLs.
Eight patterns that create waste
- Sort and filter parameters. The classic. A category with 40 filters can generate thousands of near-identical pages.
- Tracking and session parameters. Analytics tags, affiliate IDs, and session tokens that create URL variants of the same page.
- Deep pagination. Page 138 of a listing that nobody will ever browse or search for.
- Auto-generated tags. Thousands of one-item tag pages with no editorial intent.
- Date archives. Monthly and daily archives from a blog's early years.
- Stale variants. Discontinued products, old colourways, and expired campaign pages still returning 200.
- Case, slash, and index variants.
/Page,/page/, and/page/index.htmlresolving to the same content. - Print and share views. Alternate templates that add no search value.
How to cluster an inventory
You do not need a machine-learning pipeline to do this. Start with a crawl and apply three passes:
- Normalise the path. Replace IDs, slugs, and numbers with placeholders.
/product/1042-blue-shirtbecomes/product/*. - Normalise the query string. Sort parameter names and replace values with
*. Keep the set of parameter names, because that is what identifies the pattern. - Group by signal. Within each shape, group by status code, canonical target, and robots directive. A pattern that returns 200 and self-canonicalises is a different decision from one that redirects.
The output is a table of patterns with counts, example URLs, and signals. That table is the actual deliverable of an SEO cleanup.
Decide the treatment per pattern
There are six treatments. Choose one per pattern and document why.
| Treatment | Use when | Watch out for |
|---|---|---|
| Keep and index | The pattern targets real search demand and has unique value. | Make it self-canonical, linked, and in the sitemap. |
| Canonicalise | It is a legitimate variant that should consolidate into a primary page. | Canonicals are hints; crawlers still fetch the duplicate. |
| Noindex | The URL should be reachable but not appear in search. | Do not also block it in robots.txt, or the noindex may never be seen. |
| Disallow | It is pure crawl waste with no need to pass signals. | Blocked URLs can still be indexed if linked. Do not block CSS or JS. |
| Redirect | There is a genuinely equivalent replacement page. | Use a single 301 hop; avoid chains and loops. |
| Remove (410) | The content is gone and no equivalent exists. | Clean up internal links to the removed URLs. |
An illustrative cleanup
Consider an e-commerce site with 180,000 crawlable URLs and 6,000 products. Grouping the inventory reveals that 142,000 URLs are sort and filter combinations, 18,000 are pagination, and 9,000 are retired products. The real catalogue is much smaller than the crawl makes it look.
A pattern-level plan might keep 800 valuable facet pages (the ones with search demand), canonicalise the rest to their parent category, replace retired product URLs with 410s, and remove deep pagination from the sitemap. Within one crawl cycle, the inventory drops to roughly 40,000 URLs — not because content was deleted, but because unnecessary variants stopped competing for attention.
This example is illustrative, not a client result.
Monitor the cleanup
A cleanup is not finished when the ticket closes. New filters ship, campaigns add parameters, and old templates linger. Re-crawl on a regular cadence and compare pattern counts:
- Did the low-value pattern shrink?
- Did important pages get crawled sooner?
- Did the number of indexed URLs change in the direction you intended?
- Did any legitimate pages disappear by accident?
That is the workflow PruneSEO is built around: crawl, group by pattern, inspect signals, prioritize, and compare the next run. If you want the full audit sequence, start with our technical SEO audit checklist.
Frequently asked questions
What is a URL pattern in SEO?
A URL pattern is a group of URLs that share the same structural shape — the same path template, parameter set, and often the same status, canonical, and robots signals. Grouping by pattern lets you fix thousands of URLs with one decision.
Should I noindex or canonicalize duplicate URLs?
If the duplicate should never appear in search and has no value, noindex can be appropriate. If it is a legitimate alternate version that should consolidate signals to a primary URL, a canonical is usually better. Do not combine conflicting signals, and decide per pattern.
How many URLs is too many?
There is no universal number. What matters is the ratio of valuable URLs to low-value variants and whether important pages are crawled and indexed quickly. A site with 20,000 meaningful pages can be healthy; a site with 20,000 filter combinations around 400 products is usually not.