What Is Duplicate Content? How Do You Fix Parameter URLs and Pagination Issues?

Duplicate content is the same or very similar content appearing at more than one URL. Google has no general “duplicate content penalty”; the real problems are that signals get split between URLs, crawl budget is wasted and Google may show the wrong address in results. The fix is to apply the right method (canonical, 301, noindex or robots) to the right situation.

Most duplicate content problems are not deliberate; they come from the technical setup of the site. The same page can open on www and non-www, http and https, with and without a trailing slash. Ad parameters, filters and sort options each produce a new address. In e-commerce, colour and size variants add more, and on blogs pagination and archive pages do too. In this post we cover the causes, detection methods and which solution fits which situation.

What is duplicate content, and is there really a penalty?

Duplicate content is the same or largely similar content appearing at more than one address, within your site or between your site and others. According to Google's documentation, most duplicate content is not grounds for a penalty unless it has manipulative intent. The system groups similar pages, picks one (the canonical) and shows it in results; the others are filtered.

That does not mean there is no problem. The address Google picks may not be the one you want; links and signals can spread across several URLs; crawl time is wasted. Cases where a penalty is involved are practices with intent, such as mass-copying content from other sites or duplicating content to manipulate results. Those fall under the spam policies.

• —No penalty: no manual action is applied for normal technical duplicates.
• —Signal splitting: links and engagement are spread across different URLs, so none becomes strong.
• —Wrong URL choice: Google shows the address it picked, which may be a parameter URL or one you did not want.
• —Crawl waste: on large sites bots spend time on duplicate addresses and important pages are crawled later.
• —The real risk: mass copying from other sites and thin content.

What is the difference between internal and external duplicates?

Internal duplicates are the same content at different addresses on your own site; external duplicates are the same content on another domain. Internal duplicates usually come from technical setup and can be controlled; external ones come from content distribution, shared product descriptions or unauthorised copying, and are solved differently.

• —Internal: www/non-www, http/https, parameter URLs, pagination, print versions, tag and archive pages. Fix: canonical, 301, noindex.
• —External (legitimate): publishing the same article in another outlet too (syndication). Fix: ask the copy to point a canonical or link to the original.
• —External (manufacturer descriptions): dozens of sellers using the manufacturer's same product text in e-commerce. Fix: make product descriptions original.
• —External (unauthorised): your content being copied. Fix: contact the source site's administrator and, if needed, use Google's copyright (DMCA) notices.
• —Partial similarity: template parts (footer, menu) do not count as duplicates; the problem is repetition of a page's main content.

Why do www/non-www, http/https and trailing slashes create duplicates?

If the server serves the same page on www and non-www, on http and https, and with and without a trailing slash, each one is technically a separate URL. The right fix is to choose one preferred version and 301-redirect the others to it at server level. That way both users and bots always land on the same address.

• —www / non-www: choose one (for example non-www) and 301 the other.
• —http / https: all traffic should be redirected to https; the http version should not stay open.
• —Trailing slash: reduce /services and /services/ to one form; whichever you choose, use the same in canonicals and internal links.
• —Upper/lower case: if /Services and /services return the same content, lower-case it and 301.
• —index.html and homepage: /index.php or /index.html addresses should redirect to the homepage.
• —Multiple domains: old domains and subdomains should 301 to the new domain.
• —Verification: try the four variants (http://www, http://, https://www, https://) in a browser and see that all reach one address.

This fix is the first step on most sites and is usually handled with a one-time server or .htaccess setting. After that, make sure internal links, the sitemap and canonical tags also use the same preferred version.

Why do UTM, filter and sort parameters cause problems?

UTM, filter and sort parameters serve the same content at different URLs: ?utm_source=newsletter, ?colour=blue, ?sort=price. Every combination is a new address and on large sites can create thousands of duplicates. Tracking parameters like UTM do not change the content; filters and sorting change it partly, so the same method does not apply to both.

• —UTM and tracking parameters (utm_*, fbclid, gclid): same content; the page should self-canonicalise to the parameter-free address.
• —Sort parameters (?sort=price): same content in a different order; in most cases canonical to the main listing.
• —Filter parameters (?colour=blue&size=m): content differs partly; a valuable combination (such as “blue running shoes”) can be indexed as its own page, otherwise canonical or noindex.
• —Session IDs (?sid=...): same content; handle with cookies if possible or use canonical.
• —Search result pages (?s=word): usually noindex; they produce endless combinations.
• —Google's old URL Parameters tool has been retired; manage parameters with canonicals, robots and server settings.

Do product variants in e-commerce create duplicate content?

In e-commerce, if separate addresses are opened for colour, size or pack variants of the same product, near-duplicate pages arise because most of the description is identical. The solution is to gather variants on one main product page or to decide whether each variant really has a separate search intent. If intent differs, a separate page fits; if it is the same, one page plus a canonical fits.

• —Small differences (size, colour): one product page with options on the page; variant addresses canonical to the main product.
• —Meaningful differences (separate model, different use): separate page and original description.
• —Manufacturer texts: enrich the description every seller uses with your own product information, dimensions, usage and FAQs.
• —Category pages: if the same products are listed in several categories, the product should have one main address; other paths should canonical to it.
• —Out-of-stock product: if the page is gone for good, 301 or 410; if temporary, keep the page and state the status.
• —Filtered category pages: deliberately turn combinations with high search volume (such as “black leather women's bag”) into separate landing pages.

Designing e-commerce pages with this view both reduces duplicate risk and makes the pages stronger. For a detailed guide, see our e-commerce SEO post.

How should pagination be handled correctly?

Pagination splits a long list (a blog archive, a product category) across several pages. Correct handling means giving each page a standard, crawlable link, having each page self-canonicalise and keeping enough unique content on the pages. Canonicalising pages 2, 3, 4 to page one, and noindexing page 2 onward, are common but generally not recommended mistakes.

Google announced in 2019 that it no longer uses rel="next" and rel="prev" markup for indexing. These tags do not have to be removed entirely and other search engines may use them, but they are not a signal for Google. Google understands pagination sequences through standard links. So what matters is that “Next” and the page numbers are real `<a href>` links; “load more” buttons that work only through JavaScript can stop bots from reaching the content.

• —Each page canonicalises to itself: the /category?page=2 page should show itself as canonical, not page one.
• —Real links: give page numbers as `<a href>`; with infinite scroll, also offer these links separately.
• —Unique titles: use a title/H1 difference such as “Category name - Page 2”; but do not leave page 2 onward as thin content.
• —“View all” page: can be used for small lists when all products load fast on one page; on large lists it causes performance problems.
• —Consistent ordering: use a fixed default order for pagination; tie sort parameters to the main listing with canonical.
• —Internal links: if deep pages are reached only through pagination, link important products from other places too.
• —Sitemap: put the actual content addresses (products, posts), not pagination addresses, in the sitemap.

How do you choose between canonical, 301, noindex and robots.txt?

The choice depends on whether the duplicate address is needed by users and whether you want search engines to see it. An address users do not need is redirected with a 301; an address that must stay open to users but not in the index gets a canonical or noindex; robots.txt blocks crawling and also blocks the canonical or noindex from being read, so it is used carefully.

• —301 redirect: when the duplicate address is not needed at all (www/non-www, http/https, old URL). It carries signals to the target and the user goes to the target too.
• —rel=canonical: when the duplicate must stay open for users but the main address is clear (UTM, sorting, variants). A strong hint for Google, not a strict command.
• —noindex: the page is open to users but not wanted in search results (internal search, thank-you pages, thin filter pages). Google must be able to crawl the page.
• —robots.txt block: to protect crawl budget (endless filter combinations, calendars, action URLs). The canonical and noindex tags of a blocked page are not read.
• —Password or access restriction: for test and staging environments; these should never get into the index.
• —None: if the pages genuinely carry different content and meet a separate intent, no action is needed.

A common mistake is putting noindex, canonical and a robots block on the same page. These signals conflict and leave it unclear which applies. Choose a single method for each type of duplicate. For robots.txt details, see our guide on it.

How do you detect duplicate content?

The most reliable way to detect duplicate content is to look at the Page indexing report in Search Console and crawl your site with a crawler tool. The report lists statuses such as “Duplicate without user-selected canonical”, “Duplicate, Google chose different canonical than user” and “Alternate page with proper canonical tag” separately.

• —Search Console → Page indexing: list the duplicate and canonical statuses and inspect counts and sample addresses.
• —URL Inspection: for a single address, check whether the user-declared canonical differs from the Google-selected canonical.
• —Crawler tool (such as Screaming Frog): group identical titles, identical H1s and similar content hashes.
• —site: query: type “site:yourdomain.com ‘a sentence fragment’” and see on how many addresses the same text appears.
• —Parameter URLs: look for indexed ?utm=, ?sort= and ?page= addresses that are not in the sitemap.
• —Log files: see from server logs which parameter URLs Googlebot spends most time on.
• —Content copy check: search key paragraphs in quotation marks to see whether they appear on other sites.

A duplicate status in the report does not always need fixing. A “Google chose a different canonical” notice shows that the signals you gave were not fully adopted by Google, but if the chosen address is acceptable, no action is required. What matters is that the pages you want to rank are chosen as the main address.

Are duplicate content and keyword cannibalisation the same thing?

No. In duplicate content, the same content is at several addresses. In keyword cannibalisation, several pages with different content target the same search and compete with each other. Duplicate content often comes from technical setup, cannibalisation from content planning; their fixes differ too: one is canonical and redirects, the other merging or differentiation.

The two can overlap. For example, two very similar service pages can create both a duplicate content and a cannibalisation problem. In that case the cleanest fix is usually to merge the pages into one strong page and 301 the old one to the new address. Pages multiplied by swapping a city name are also risky for this reason; every city page must be genuinely unique.

What do you do about external duplicates and content theft?

When another site copies your content, Google mostly identifies the original source correctly; still, taking precautions is sensible. Keep a record of your publication dates, contact the administrator of the copying site and ask for removal, and if that fails, turn to the hosting provider and, if needed, Google's copyright notice process.

• —Proof of publication: dated archive records, a fast indexing request in Search Console, social shares.
• —Self-links: having original internal links in your content can give you a backlink when it is copied.
• —Syndication: if you knowingly give content to another outlet, ask that outlet for a canonical or a link to the original.
• —Removal request: a polite email, then a report to the hosting company.
• —Google notice: a copyright removal (DMCA) notice is the last step for serious violations.

A practical checklist for duplicate content

The list below gives the minimum steps to go through in order when auditing a site for duplicate content. On most sites the first three items solve most of the problem.

• —Do www/non-www, http/https and trailing slash all 301 to one preferred version?
• —Does every indexable page carry a correct canonical tag pointing to itself?
• —Do internal links, the sitemap and canonicals use the same URL format?
• —Do UTM and tracking parameters canonical to the parameter-free address?
• —Are filter and sort combinations under control (canonical, noindex or robots)?
• —Do pagination pages self-canonicalise and carry crawlable `<a href>` links?
• —Are e-commerce variants tied to the main product, and are manufacturer texts made original?
• —Are duplicate and canonical statuses in Search Console checked regularly?
• —Are test and staging environments closed to the index?
• —Is there any conflicting signal (noindex, canonical and robots on the same page)?

On a small corporate site the duplicate problem is mostly limited to version redirects and a few parameters; on a large e-commerce site, filter combinations can become a serious crawling problem. In technical SEO audits at 404 Ajans we go through these items in order, prioritise them and give a clear table of which method applies to which group of addresses; still, the list above is enough for your own team to run a first crawl.