My account Get a quote
SEO

Technical SEO Checklist: Crawling, Indexing and Sitemap

Technical SEO checklist: a step-by-step audit of crawling, indexing, sitemap, canonical, hreflang, structured data and status codes, plus a regular routine.

· 6 min read
Technical SEO Checklist: Crawling, Indexing and Sitemap

A technical SEO checklist is a review, in order, of the technical items that ensure search engines can crawl your site without problems, understand it, and index the right pages. The basic sequence is: first make sure pages are crawlable and accessible, then determine which pages should be indexed, and then fine-tune signals such as the sitemap, canonical, hreflang and structured data. This guide explains, for each heading, what to check and which mistakes to avoid.

Why does technical SEO come first?

Even the best content becomes invisible if search engines cannot find it or misinterpret it. Technical SEO organizes the foundation of your content: it depends on bots reaching your pages, search engines knowing which address is the canonical one, and the topic of each page being correctly understood. Checking the technical foundation before starting content and link work prevents effort from being wasted later. For the general framework, you can also read our article what is SEO and how is it done.

1. Crawling: robots.txt and accessibility

Crawling is when search engine bots visit your pages. Check the following:

  • Is there a robots.txt file in the root directory, and does it accidentally block important sections?
  • Was the "block all bots" rule used before publishing removed when the site went live? This is one of the most common and most costly mistakes.
  • Are CSS and JavaScript files open to bots? They are needed so bots can understand how the page looks.
  • Do pages avoid server errors (5xx) and timeouts?
  • Can important pages be reached from the main menu and internal links? Find "orphan" pages that receive no links.

Note: robots.txt prevents a page from being crawled, but it does not definitively prevent the address from appearing in the index. To remove a page from the index, use the noindex directive. Remember that in this case the page must remain open to crawling; if bots cannot read the page, they also cannot see the noindex instruction.

2. Indexing: which pages should appear in search?

Not every page needs to be indexed. Thank-you pages, the basket, user accounts, search result pages and filter combinations are usually not wanted in the index. What matters is that pages that should be indexed are, and pages that should not be are not.

  • In the page indexing report in Search Console, review statuses such as "crawled but not indexed" and "discovered but not crawled".
  • Check that valuable pages do not accidentally carry noindex.
  • Merge, improve or remove thin or duplicate pages.
  • Verify how a page appears in the index with the URL inspection tool in Search Console.

3. XML sitemap

A sitemap is a list of the addresses you want indexed, and it speeds up discovery. A good sitemap has these characteristics:

  • It includes only the addresses you want indexed, which return a 200 status code and are canonical.
  • Redirected, deleted or noindex pages are not in the sitemap.
  • It is updated automatically as pages change, and the modification date (lastmod) changes only when the content has actually changed.
  • It is submitted to Search Console, and its address is specified in robots.txt.
  • Large sites are split into sub-sitemaps and gathered by an index file.

4. Canonical tags

When the same content can be reached at several addresses (parameterized addresses, tracking codes, print versions, or http/https and www differences), a canonical tag is used to tell the search engine which address is the main one.

  • Each page should carry a canonical that points to its own address.
  • The canonical address must be a full (absolute) address and must not point to a redirect chain or to a page that returns an error.
  • The address in the canonical, the sitemap and internal links must be the same; do not send contradictory signals.
  • On pagination, filter and sort pages, choose your strategy deliberately; making every page canonical to the first page can lead to content loss.

5. Hreflang (multilingual and multi-regional sites)

If your site has several language or regional versions, hreflang tags tell the search engine which version should be shown to which audience. Important rules:

  • Each version must mark all its counterpart pages, including itself; the relationship must be reciprocal.
  • Language codes and, where needed, region codes must be written correctly.
  • Every address marked must be crawlable, canonical and return a 200 status code.
  • x-default can be used to define a suitable default version.

For detailed setup, see our multilingual website guide.

6. Structured data (schema)

Structured data describes the content of a page (such as products, articles, businesses, questions and answers, or breadcrumb links) in a standard format that search engines understand. The most common method is JSON-LD.

  • Mark up only the real content visible on the page; hidden or invented information violates the rules.
  • Check validity with Google's rich results test and schema validation tools.
  • For local businesses, keep the business name, address, phone number and opening hours consistent.
  • There is no guarantee of rich results; the goal is correct understanding.

7. URL structure, redirects and error pages

  • Addresses should be short, readable and lowercase, and should not carry unnecessary parameters or session IDs.
  • Use 301 redirects for permanently moved pages, and clean up redirect chains and loops.
  • A page that truly does not exist should return a 404 status code; it should not be a "soft 404" (an error message served with a 200 code).
  • A custom 404 page should guide visitors to the main sections.
  • Make one version of the site (https, and either www or non-www) canonical and redirect the others to it. After an HTTPS switch, verify that all internal links also use the current address.

8. Speed and mobile compatibility

Search engines primarily base their evaluation on the mobile version of a page. Make sure the content and links on mobile are equivalent to desktop, and that meta tags and structured data are present in both versions. Core Web Vitals (LCP, INP, CLS) are monitored for page experience; for improvement steps, see our website speed optimization guide.

9. Basic on-page elements

During the technical audit, also review the basic on-page elements; these are as much a technical matter as content is. Each page should have a unique title tag and description, a single main heading (H1), and subheadings should follow a logical hierarchy. Use meaningful alt text for images, and write link texts that describe the destination rather than "click here". Repeating the same title and description across hundreds of pages is a common problem, especially on product and category pages; if you use template-based automatic generation, remember to add page-specific variables to each page.

Internal link structure is also part of this layer. Important pages should be reachable within a few clicks from the home page, related content should link to each other with natural text, and broken links should be cleaned up regularly. In lists that use pagination, each page should be reachable through crawlable links; lists that work only with a "load more" button and contain no links can make it harder for bots to reach later content.

Finally, security is indirectly related to technical SEO too: a site infected with malware or showing a security warning may be displayed with a warning in search results. So keep the site updated and include malware scanning in your audit routine.

Summary check table

HeadingCheck questionWhich tool?
CrawlingDoes robots.txt block important pages?robots.txt file, Search Console
IndexingAre valuable pages indexed and unwanted ones kept out?Page indexing report, URL inspection
SitemapDoes it contain only canonical 200 addresses?Search Console, crawling tool
CanonicalDoes every page point to the correct main address?Crawling tool, page source
HreflangAre the relationships reciprocal and correctly coded?Crawling tool, page source
Structured dataIs the markup valid and consistent with the visible content?Rich results test
Status codesAre there chains, loops or soft 404s?Crawling tool, server logs
Speed and mobileAre Core Web Vitals good, and is the mobile version complete?PageSpeed Insights, Search Console

Regular checking routine

  1. Weekly: check new error and warning notifications in Search Console.
  2. Monthly: crawl the site with a crawling tool; list broken links, redirect chains, and missing titles and tags.
  3. After every major release: re-verify robots.txt, noindex, canonical and the sitemap.
  4. Whenever a new language or section is added: update hreflang and the sitemap.
  5. Every three months: review bot behavior and error codes in the server logs.

Conclusion

Technical SEO lays a solid foundation, but it requires ongoing maintenance, because every new page and every version change brings a risk. Tying the checklist to a routine is the most effective approach. If you want to carry out a technical audit of your site together, you can get in touch with Erdem Software.

Frequently asked questions

Where should I start with technical SEO?

Start with crawling: robots.txt, server errors and the indexing report. Then look at the sitemap, canonical tags and status codes; the most critical problems usually show up at these stages.

Does robots.txt remove a page from the index?

No, it only blocks crawling; the address can still appear in the index. To remove a page from the index, use noindex, and the page must remain crawlable for that to work.

Is a canonical tag needed on every page?

It is recommended. If every page carries a canonical pointing to its own main address, parameterized and duplicate addresses do not create confusion.

When is hreflang needed?

When your site has several language or regional versions. The tags must be reciprocal, and all marked addresses must be crawlable.

How often should a technical SEO check be done?

Checking Search Console weekly and running a full site crawl monthly is a good routine. Also check after major releases and infrastructure changes.

Need help with this?

Google Business Profile, Apple and Yandex Maps listings, review management and visibility in nearby searches.

See the serviceGet a quote

Related posts

All →