Technical SEO is where most of the invisible failures in SEO live. A site can have excellent content and real authority and still underperform because a third of its pages are not indexed, its canonical tags point at the wrong versions, or its content only appears after a JavaScript bundle finishes executing. None of this shows up as an error. It shows up as a site that is "doing everything right" and not growing.
I organise technical SEO around the chain a search engine follows: it must crawl a page, render it, decide to index it, and then it can rank it. Each topic below sits somewhere on that chain, and the order in which I audit follows the chain too. This hub gives the mechanism and the common failure for each topic; the sub-guides linked at the end go deeper.
- Crawl
- Render
- Index
- Rank
Crawlability
Crawlability is whether Googlebot can discover and fetch a URL. Discovery comes from links (internal and external) and sitemaps. Fetching depends on robots.txt permissions, server response, and crawl budget, which is Google's own cap on how many requests it will make to a host in a period. It becomes a real constraint for large catalogues, marketplaces and anything with faceted navigation, where millions of parameter combinations compete with the few thousand pages that matter.
Common failures: important pages reachable only through JavaScript-driven navigation, filters that generate unbounded URL spaces, pagination that Google cannot follow, and slow server responses that cause Google to throttle itself. The symptom is pages sitting at "Discovered, currently not indexed", or never appearing in Search Console at all. The crawlability and indexability guide covers the diagnosis.
Indexability
A crawled page is not automatically indexed. Google decides whether to store it based on directives (noindex, canonical), duplication (is this a copy of something already stored?), and quality (is this worth storing?). The quality judgement is the one people underestimate. "Crawled, currently not indexed" is usually not a technical problem; it is Google saying the page is not distinct or valuable enough. Fixing it means consolidating, improving or removing pages, not resubmitting them.
Check indexability with URL inspection in Search Console, not with a site: search, which is unreliable. On a large site, segment the coverage report by template (product, category, article, tag) so you can see which types are being dropped. Dropped templates point to structural problems; dropped individual pages point to content problems.
Rendering and JavaScript
Google fetches HTML first and runs JavaScript later, in a second rendering wave that may lag the crawl. If the content, internal links, canonical tag or meta robots directive only exist after JavaScript runs, Google's first view of the page is incomplete, and any indexing decision made on that view is wrong. Client-side rendered frameworks are the usual cause, but so are tag managers that inject canonicals and lazy-loaders that never fire in a headless browser.
The test is simple: compare the raw HTML response to the rendered DOM, and compare both to what Search Console's URL inspection shows as the rendered page. If critical elements are missing from the raw HTML, move them server-side or pre-render. I documented my method in the JavaScript rendering test, and the JavaScript SEO guide covers the architecture options.
Site architecture
Architecture is how pages are grouped and how many clicks they sit from the homepage. It determines what Google crawls first, how link equity flows, and whether the site reads as being about a coherent set of topics. A flat architecture where every important page is within three or four clicks of the home page crawls better and ranks more evenly than a deep one. Hub pages that group related content signal topical structure; orphan pages, which nothing links to, signal the opposite.
I plan architecture from keyword clusters outward. Each cluster gets a hub, the hub links to its spokes, the spokes link back and sideways, and the nav exposes the hubs. This is the same structure this Learn SEO section uses. The site architecture guide covers the design decisions in detail, including URL structure and how to handle large catalogues.
Internal linking
Internal links are the strongest lever you fully control. They decide which pages get crawled, how often, how much authority they receive, and what context Google associates with them. Most sites waste the lever: they link heavily to the homepage and a few nav items, and leave the pages that need links, the deep money pages and new articles, with almost nothing.
The practical rules I follow: every important page is linked from at least a few relevant pages, not just the nav; anchor text describes the destination; links sit inside content, where they carry topical context; and the system scales, meaning it does not rely on someone remembering to add links manually. My internal linking at scale experiment tested automated approaches, and the internal linking guide turns it into a process.
Canonicals
A canonical tag tells Google which URL is the preferred version among duplicates. It is a hint, not a directive, and Google overrides it when the signals disagree. Common failures: canonicals pointing at the wrong protocol or hostname after a migration, paginated pages canonicalised to page one (which tells Google to drop pages two onward), every variant of a product canonicalised to one that is out of stock, and canonicals injected by JavaScript that Google never sees at first crawl.
The check is to crawl the site and compare declared canonicals with the "Google-selected canonical" in Search Console. Where they differ, Google is telling you its signals disagree with yours, and the cause is usually internal links or redirects pointing at the non-canonical version. The canonical tags guide goes through the cases.
Redirects
Redirects move users and search engines from one URL to another and, done properly, pass most of the original URL's signals to the new one. Use a 301 (or 308) for permanent moves. The failures are chains (A to B to C, each hop losing crawl efficiency), loops, soft 404s where a missing page redirects to the homepage, and redirects that strip parameters or change the destination by device. After any restructuring, crawl the old URL list and confirm each one resolves in a single hop to a live, indexable page.
Pagination
Paginated series (category pages 2, 3, 4) need to be crawlable so that deep products or articles get discovered. Each page in the series should be self-canonical, linked with ordinary anchor links (not just "load more" buttons), and should not be noindexed unless you have a separate discovery path for the items.
XML sitemaps
A sitemap is a list of URLs you want indexed, with last-modified dates. It helps discovery; it does not force indexing. The useful discipline is that the sitemap should contain only canonical, indexable, 200-status URLs. Sitemaps full of redirects, noindexed pages and 404s tell Google your site is poorly maintained. On large sites, split sitemaps by template so the coverage report shows you which section has problems.
Robots.txt
Robots.txt controls crawling, not indexing. A URL blocked in robots.txt can still be indexed if it is linked from elsewhere, just without content, which is how "indexed, though blocked by robots.txt" happens. Use it to stop crawl waste (internal search results, infinite filter combinations, admin paths), never to hide pages you want out of the index; that is what noindex is for. Test every change before deploying, because a single misplaced slash can block the whole site. I built a robots.txt tester for exactly this.
Core Web Vitals
Core Web Vitals are Google's user-experience metrics: Largest Contentful Paint (loading), Interaction to Next Paint (responsiveness) and Cumulative Layout Shift (visual stability). They are a ranking input, but a modest one; they are far more important as a conversion input and as a crawl-efficiency input, since a fast server lets Google fetch more pages. In India, where many users are on mid-range Android devices and variable mobile networks, the field data is often worse than the lab data suggests.
Work from field data (the Chrome UX Report, surfaced in Search Console) not lab scores. The usual wins are image sizing and formats, removing render-blocking scripts, reserving space for ads and embeds to stop layout shift, and reducing main-thread work from third-party tags. The Core Web Vitals guide explains how to find the actual cause of each metric rather than chasing a score.
Structured data
Structured data (schema.org markup in JSON-LD) describes what a page is about in a machine-readable way. It makes pages eligible for rich results such as product prices, ratings, FAQs and breadcrumbs, and it helps search engines connect a page to entities. It does not improve ranking directly. The failures are markup that describes things not visible on the page, which violates guidelines, and markup that never validates because it is injected incorrectly. I tested the real effect of rich results in the schema markup experiment; the semantic role is covered in the semantic SEO guide.
Log file analysis
Server logs are the only record of what Googlebot actually did, as opposed to what a crawler simulated or what Search Console summarised. Reading them tells you which sections get crawled, how often, which URLs waste budget, which return errors to Google but not to users, and whether new pages are being discovered at all. The technical SEO at scale note covers why this becomes decisive on large sites.
Migrations
A migration is any change to URLs, domain, platform or template structure. It is the single most common way a site loses traffic through technical SEO. The work is in the preparation: a full crawl and URL inventory before the change, a one-to-one redirect map, a staging crawl that confirms the new site renders, canonicalises and links correctly, and a launch-day checklist that includes removing the staging noindex. Afterwards, monitor crawl stats, coverage and the old URLs' redirect status for weeks.
How I audit: priority first
A technical audit that lists three hundred issues of equal weight gets ignored. The useful output is a short list of what blocks the chain, then what wastes effort, then what is hygiene. Here is the rough priority I apply; the technical SEO audit framework is the full version.
| Priority | Issue type | Why it sits here | How to check |
|---|---|---|---|
| 1. Blocking | Noindex on live pages, robots.txt blocking key sections, content missing from raw HTML, redirect loops, server errors | Stops crawl, render or index outright. Nothing else counts until fixed. | Crawl plus URL inspection on templates; compare raw HTML to rendered DOM; check server logs for 5xx |
| 2. Dilution | Wrong or conflicting canonicals, duplicate URL variants, cannibalising pages, faceted crawl waste, orphan pages | Google indexes the wrong things or spreads signals thin. Often the cause of "doing everything right but flat." | Declared vs Google-selected canonical; coverage report by template; log analysis for crawl distribution |
| 3. Structure | Deep architecture, weak internal linking, poor hub pages, uncrawlable pagination | Limits how far content and authority can carry. Compounding fixes. | Crawl depth report; inlink counts per important page; click-path from home to money pages |
| 4. Experience | Core Web Vitals failures, mobile usability, intrusive interstitials | Modest ranking input, significant conversion and crawl-efficiency input | Field data in Search Console; lab tools to locate the cause |
| 5. Enhancement | Missing structured data, sitemap hygiene, image alt text, hreflang details | Adds eligibility and clarity; rarely moves ranking on its own | Rich results testing; sitemap vs crawl diff; hreflang validation |
Inventory
Crawl the site, pull Search Console coverage, and sample server logs. You now know what exists, what Google has stored, and what Google actually fetches.
Chain check per template
For each page type, confirm crawl (reachable, allowed), render (content in raw HTML), index (Google-selected canonical matches yours). Problems at template level affect thousands of pages at once.
Prioritise
Sort findings into the five tiers above. Estimate the number of pages affected and the effort to fix. Blocking issues on money pages go first regardless of effort.
Specify and ship
Write each fix as a ticket with the current state, the required state and the way to verify. Then verify after deployment; a fix that was merged but not deployed is not a fix. This stage is where most audits die, which I wrote about in the implementation problem.
Monitor
Re-crawl, re-check coverage and watch logs for a few weeks. Technical fixes usually show in weeks, which makes this the fastest feedback loop in SEO.
Go deeper
- Crawlability and indexability. Diagnosing why pages are not crawled or not indexed, and what each Search Console status really means.
- Internal linking. Designing a linking system that distributes crawl and authority to the pages that need it, at scale.
- Core Web Vitals. LCP, INP and CLS explained by cause, with a method for finding what actually slows a page.
- Canonical tags. How canonicals work, when Google ignores them, and the patterns that go wrong on e-commerce and paginated sites.
- JavaScript SEO. Rendering options, how to test what Google sees, and when to move content server-side.
- Site architecture. Hubs, depth, URL structure and how to organise large sites so they crawl and rank evenly.
Frequently asked questions
How often should a technical SEO audit be done?
A full audit once a year or after any platform change, migration or redesign. Lightweight monitoring, meaning crawl stats, coverage and a scheduled crawl, should run continuously. Most damage happens between audits, during routine deployments.
Does fixing Core Web Vitals improve rankings?
Slightly, and mostly as a tie-breaker between otherwise comparable pages. The larger gains from speed are in conversion and in crawl efficiency. I prioritise it after blocking, dilution and structure issues unless the site is failing badly on mobile.
What does "Crawled, currently not indexed" mean?
Google fetched the page and chose not to store it. On most sites this is a quality or duplication judgement rather than a technical block. Look at whether the page is distinct, substantial and linked from relevant places before looking for a technical cause.
Can technical SEO alone grow traffic?
If the site was blocking or diluting content that deserved to rank, yes, and sometimes quickly. Once the site is clean, technical work stops adding traffic. It raises the ceiling; content and authority have to fill the space.
Is a JavaScript framework bad for SEO?
Not inherently. Server-side rendering, static generation or hybrid rendering all produce HTML Google can read at first crawl. Pure client-side rendering of SEO-critical content is the problem. Test what the raw response contains rather than trusting the framework's claims.
Where this connects
This guide is the reading behind the technical SEO audit framework, which turns it into a repeatable process. The rendering claims are tested in the JavaScript rendering test, and the whole approach is applied in the technical SEO and content architecture case study. If you need someone to run the audit and see the fixes through to deployment, that is my technical SEO service.