An XML sitemap is a machine-readable file that tells search engines the URLs of pages on your site that you want crawled and indexed. It helps Google discover your pages more easily, but it does not guarantee indexing or rankings. When built properly it is especially useful on large, new or weakly linked sites.
Many people see a sitemap as a magic button: upload the file and Google lists the pages. In reality a sitemap is a hinting mechanism. Google learns which URLs you want to highlight, but crawling and indexing them is still its own decision. This article explains what a sitemap does and does not do, how to prepare one, Google's documented limits and how to submit it in Search Console.
What is an XML sitemap and what is it for?
An XML sitemap is a file that lists your site's important URLs and, optionally, their last modified dates. Search engine bots read it to discover your pages. It speeds up and eases discovery for pages that are hard to find through links, newly published content and large sites.
If your site is small and your pages are already easy to reach through the main menu and internal links, a sitemap is not essential; still, preparing one costs almost nothing and gives you useful reports in Search Console. It helps most in these situations:
- —A new site: with few external links, bots may discover pages late.
- —A large site: reduces the chance of important pages being overlooked when there are thousands of products or posts.
- —Weakly linked pages: helps orphan pages, which no link points to, get found.
- —Frequently updated content: a way to communicate changes to blog, news and campaign pages.
- —Reporting: lets you see in Search Console how many of the submitted URLs are indexed.
What does a sitemap not guarantee?
A sitemap guarantees no crawling, indexing, ranking or traffic. Google may decide not to index a URL you list for reasons such as quality, duplicate content, technical barriers or the overall state of the site. A sitemap is simply an orderly way of saying “these pages exist and they matter”.
Do not think of a sitemap as a quality tool either. Putting weak, duplicate or worthless pages in it does not make them valuable; on the contrary, it pollutes the pool of pages Google sees when assessing your site. A sitemap should be your shop window: only quality, accessible pages you genuinely want in the index.
- —It does not guarantee indexing; Google can ignore a URL if it chooses.
- —It does not directly affect rankings; it only helps discovery and crawling.
- —It does not override a robots.txt block or noindex; listing a blocked page sends a contradictory signal.
- —It does not rescue weak content; page quality and usefulness remain decisive.
Which URLs should be included in a sitemap?
A sitemap should include only URLs that are canonical (preferred), return a 200 status code and that you want indexed. In other words, every line should mean “this is the single valid version of the page, please index it”. Addresses that redirect, error out or point their canonical to another page should not be in the sitemap.
- —Canonical URLs: the single preferred address of each page; keep www/non-www and http/https consistent.
- —Pages returning 200: addresses that return a redirect (301/302), 404 or 5xx should not be listed.
- —Indexable pages: those without a noindex tag and not blocked by robots.txt.
- —Core content pages: valuable pages such as services, products, categories, blog posts and contact.
- —Language alternatives: on a multilingual site, each language version with its own URL and a consistent hreflang structure.
For multilingual sites you can also supply hreflang relationships inside the sitemap; this works well when there are many language versions, without bloating the HTML head. We cover hreflang setup in detail in a separate article.
Canonical and redirect background
Which URLs should be left out of a sitemap?
Leave out addresses you do not want indexed or that cannot be indexed: noindex pages, redirecting URLs, pages returning 404, duplicate parameter URLs, session or cart pages, internal search results and test environments. If they enter the sitemap, they both clutter the Search Console report and waste crawl resources.
- —Pages with a noindex tag and thank-you pages.
- —Old addresses that send visitors elsewhere with a 301/302.
- —Deleted pages returning 404 or 410.
- —Duplicate or filter URLs whose canonical points to another page.
- —Cart, checkout, account and session-parameter addresses.
- —Internal site search result pages.
- —Development, staging and password-protected environments.
This rule matters especially on e-commerce sites. If thousands of URLs generated by filters such as colour, size and sorting are added to the sitemap, both crawl budget and report readability suffer. If you have prepared special, content-enriched pages for important filter combinations, only those should be listed.
What does a sitemap file look like and which tags matter?
A simple XML sitemap has a urlset root and a url element for each page; inside the url element the loc (full URL) is mandatory. The optional lastmod gives the date of the page's last meaningful change. Google states in its documentation that it ignores the priority and changefreq tags, so you do not need to fill them in.
The file must be UTF-8 encoded, URLs must be absolute (full addresses starting with https://) and special characters must be escaped properly. Here is the simplest form in words: the file starts with the <?xml version="1.0" encoding="UTF-8"?> declaration, then <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> opens; for each page you write a line like <url><loc>https://www.example.com/page/</loc><lastmod>2026-10-08</lastmod></url>, and it closes with </urlset>.
- —loc: required; the full, canonical address of the page.
- —lastmod: optional but useful; update it only for real content changes.
- —priority and changefreq: Google says it does not use them; not worth the effort.
- —A sitemap can reliably report only URLs within the host and path scope where it is located.
Technical SEO support
How should the lastmod tag be used correctly?
lastmod should show the date when the page's content last changed in a meaningful way; a minor template change or merely the page load date does not count. When Google sees that lastmod values are consistently reliable, it may use them in crawl scheduling; writing today's date on every page undermines that trust.
Many plugins and CMSs generate lastmod automatically. What matters is that the value genuinely relates to content updates. If a system changes the date of every page on each build or deployment, removing lastmod is better than giving false information.
- —Correct use: update the date when an article is revised or price or product information changes.
- —Incorrect use: writing today's date on all pages with every publish.
- —Format: W3C Datetime, in its simplest form YYYY-MM-DD.
- —When in doubt: leave lastmod out rather than write a wrong date.
What is the sitemap size limit and when do you need a sitemap index?
According to Google's documentation, a single sitemap file may contain at most 50,000 URLs and its uncompressed size must not exceed 50 MB. If you reach either limit, you need to split the URLs into several sitemap files and collect them in a sitemap index file.
A sitemap index is a parent file that lists other sitemap files. It is enough to submit only this index to Search Console. A sensible split also makes reporting easier: if you use separate files for the blog, products, categories and static pages, you can follow each section's indexing status separately.
- —sitemap-pages.xml: static pages such as services, about and contact.
- —sitemap-blog.xml: blog posts.
- —sitemap-products.xml and sitemap-categories.xml: product and category split on e-commerce sites.
- —sitemap_index.xml: the main file listing the files above.
Most small and mid-sized sites never get near these limits; still, a logical split set up from the start saves effort later on sites likely to grow.
How is a sitemap prepared: plugin, by hand or automatic generation?
Depending on your platform, a sitemap is mostly generated automatically: SEO plugins in WordPress, built-in features in many e-commerce platforms, and a build-time script on custom-built sites. Writing one by hand is reasonable only for very small, rarely changing sites.
Whichever method you choose, always check the contents of the generated file. Automatic systems sometimes add noindex pages, redirecting addresses or filter URLs to the list. A sitemap generated directly from content data, as on our own site, stays current by itself when pages are added or removed, which reduces error risk compared with hand-maintained files.
- —CMS or plugin: easy to set up; exclude noindex and unnecessary URL types in the settings.
- —Build-time generation: on static or custom sites, automatic generation from page data is the most consistent method.
- —Crawler-based tools: crawl the existing site and produce a file; suitable for one-off audits.
- —Hand-written: only for sites of a few pages; the risk of forgetting to update is high.
How do you add a sitemap to robots.txt?
You can add your sitemap's address to your robots.txt file with a “Sitemap:” line. The line must contain a full, absolute URL and can sit anywhere in the file, independent of user-agent rules. If you have several sitemaps, you can write a separate line for each.
An example: a single line in robots.txt reading “Sitemap: https://www.example.com/sitemap.xml” is enough. That way not only Google but other search engines that support the line can discover your sitemap through robots.txt. Submitting in Search Console is still recommended as a separate step, because it gives you reports and error visibility.
Remember that listing a path in the sitemap that you block in robots.txt sends a contradictory signal. When deciding which sections to allow, including newer questions like AI bots, our robots.txt guide may be useful.
Think of it together with robots.txt
How do you submit a sitemap to Google Search Console?
After verifying your property in Search Console, go to the “Sitemaps” section in the left menu, enter the sitemap's address and submit. Google reads the file and shows its status. A successful read does not mean the URLs are indexed; it only shows the file was fetched and processed.
- —Step 1: Select the correct property in Search Console (domain or URL prefix).
- —Step 2: Open the “Sitemaps” page.
- —Step 3: Type the sitemap path into the “Add a new sitemap” field, for example sitemap.xml.
- —Step 4: Press “Submit” and wait for the status to read “Success”.
- —Step 5: After a few days, review the status of the submitted URLs in the “Page indexing” report.
You only need to submit the sitemap once; Google then re-reads the file periodically. You do not have to resubmit after each update. Since the old “ping” method is no longer supported, rely instead on keeping the sitemap current, giving accurate lastmod values and listing it in robots.txt.
If the report shows statuses such as “Couldn't fetch”, “Has errors” or “Submitted URL not indexed”, look at the URLs themselves, not the sitemap. We have a separate article on why pages are not indexed.
When do you need image, video or news sitemaps?
A standard sitemap is enough for most sites. Image, video and news sitemaps are used as an additional tool on sites that want to describe these content types to Google in more detail. On sites where images are already served in pages with proper HTML, a separate image sitemap is usually unnecessary.
- —Image sitemap: can help when images are loaded by JavaScript or are hard to discover.
- —Video sitemap: provides title, description and thumbnail information for self-hosted videos.
- —News sitemap: applies only to publishers eligible for Google News and has its own rules.
- —On most service and corporate sites: a single standard sitemap is sufficient.
For each of these special types, check Google's current documentation; rules and supported fields can change over time.
What are the most common sitemap mistakes?
The most common mistakes are leaving redirecting and 404 URLs in the list, adding noindex pages, using an inconsistent host (www/non-www, http/https), writing fake dates in lastmod and never updating the sitemap. Most slip through unnoticed in automatically generated files, so regular checking is needed.
- —Leaving URLs that redirect (301/302) or error out (404/5xx) in the list.
- —Adding pages with noindex or blocked by robots.txt to the sitemap.
- —Listing URLs whose canonical points to another page.
- —Mixing http and https or www and non-www versions.
- —Exceeding the 50,000 URL or 50 MB limit.
- —Changing lastmod in bulk with every publish.
- —Keeping the sitemap outside the root directory, in a path with unclear scope.
- —Not updating the sitemap on the old domain when the site moves.
Making sitemap checks a regular part of technical audits is enough. The “Page indexing” report in Search Console lets you filter which of the URLs given in the sitemap are in the index, helping you quickly spot addresses that should not be in the list or that have problems.
Sitemap checklist: before and after publishing
A short check before publishing the sitemap and after every major site change prevents broken files from going unnoticed for months. The list below provides a sufficient baseline audit for most sites.
- —Does the file open in a browser and is it valid XML?
- —Do the listed URLs return 200? Take a random sample and bulk-test with a crawling tool.
- —Are there noindex, redirecting or non-canonical URLs in the list?
- —Is the host format (https and www preference) consistent across all URLs?
- —Is the file under the limits (50,000 URLs, 50 MB)? Use a sitemap index if needed.
- —Is the sitemap listed in robots.txt and submitted to Search Console?
- —Does the sitemap update itself when a page is added or removed?
- —Have you investigated why “submitted but not indexed” URLs in the Search Console report are not entering the index?
In technical SEO audits at 404 Ajans, we always treat the sitemap together with crawl data: the URLs in the sitemap should match those that can actually be crawled and indexed. A gap between these two sets is usually a sign of the technical problem that needs solving.