Testing Internal Linking at Scale

Hypothesis

Adding rule-based contextual links from high-authority hub pages to deep pages increases crawl frequency and search impressions for those deep pages, relative to a matched control group that receives no new links.

Method

Randomised test/control split of deep pages on one site; links injected by template rule, not by hand; measured via server logs and Search Console.

Measure

Googlebot hits per page per week (primary), impressions per page (secondary), average position and indexed status (tertiary).

Duration

6 weeks of baseline, then 8 weeks post-change.

Hypothesis

Deep pages that receive contextual links from a site's strongest hub pages will be crawled more often and earn more impressions than equivalent deep pages that are reachable only through pagination or category lists. The effect should be visible in crawl logs first and in Search Console impressions later.

What would falsify it: if the test group's crawl rate and impressions move in step with the control group across the whole post-change window, the links did nothing measurable. If only crawl rate moves and impressions do not, the hypothesis is half right and I will say so.

Why I tested it

Everyone agrees internal linking matters. Almost nobody isolates it. On most sites I audit, the internal links that get added are also accompanied by content rewrites, new sitemaps and a round of technical fixes, so the effect of the links alone is never known. I wanted a cleaner answer before I kept recommending link rules at scale through my internal linking framework.

The question also has a practical edge for large Indian sites: marketplaces, listing portals and programmatic builds often have tens of thousands of deep pages and a handful of genuinely strong hubs. If rule-based links from hubs work, that is cheap leverage. If they do not, the budget belongs elsewhere, as I argue in technical SEO at scale.

Method

This is a between-groups test on a single site, so both groups share the same domain authority, server, template and algorithm exposure.

1. Select the pool

   Deep pages at the same depth, in the same template, with at least one indexed month of history and no inbound internal links other than category pagination. Pages with external backlinks are excluded so a stray link does not skew a group.

2. Randomise

   The pool is split into a test group and a control group of equal size, stratified by current impressions so neither group starts with the stronger pages. The split is recorded before any change goes live.

3. Baseline

   I collect Googlebot hits per page from server logs and impressions per page from the Search Console API for a baseline window, long enough to see the normal weekly rhythm of crawling.

4. Apply the treatment

   Hub pages gain a contextual module that links to test-group pages only, chosen by a rule (shared entity, shared category attribute). Control pages receive nothing.

5. Measure

   Same metrics, same sources, for the post-change window. Nothing else on the site is deliberately changed during the test.

Controls and confounders I am tracking: a Google core or spam update inside the window, seasonality in the category, any CMS deploy that touches the template, and sitemap regeneration. If any of these lands mid-test, I note the date and read the before-and-after on both groups, not just the test group. The control group exists precisely so that a site-wide swing is visible as a swing in both lines.

Implementation

The links are generated by a template rule, not written by hand, because the point is to test something that can run at scale. Each hub page renders a short "related" block of [N] links to test-group pages that share an attribute with the hub. Anchor text is the deep page's own H1, so I am not also testing anchor-text variation.

<!-- hub template, test arm only -->
<section class="related-deep">
  <h2>Related in this category</h2>
  <ul>
    {% for page in related_test_pages(hub, limit=N) %}
      <li><a href="{{ page.url }}">{{ page.h1 }}</a></li>
    {% endfor %}
  </ul>
</section>

Two guardrails. First, the block is server-rendered HTML, not injected by JavaScript, so there is no rendering delay muddying the crawl signal; I cover why that matters in the JavaScript SEO guide. Second, the rule is deterministic, so the same hub links to the same deep pages on every crawl. Rotating links would make the crawl data unreadable.

How to read the results

The primary comparison is the change in median Googlebot hits per page per week, test minus control, before versus after. I use the median rather than the mean because a few pages crawled heavily can drag an average. Impressions are read the same way, with a lag: links are discovered on the next hub crawl, deep pages are recrawled after that, and impressions follow only if the page's ranking changes.

A result counts as meaningful if the gap between the groups is larger than the week-to-week noise in the baseline and persists for most of the post-change window, not just one spike after the deploy. One good week is not a result.

Data

Baseline log data established. Post-change crawl frequencies and impressions are actively recorded.

Result

This experiment is tracking crawl frequency and impression deltas between test and control groups. Full results will be published once the post-change window completes.

What I learned

Early log signals suggest crawl reallocation responds within days of hub link restructuring; complete analysis to follow.

Conclusion

Definitive rules for hub-to-deep linking architecture will be documented upon test conclusion.

Until then, the method above is the useful part: if you run a version of this on your own site, keep the control group, log the deploy date, and resist changing anything else. The internal linking guide explains the mechanism this test is probing, and the architecture case study shows what the rules look like in a live build.