- Hypothesis
- New pages that are discoverable only through the XML sitemap will be indexed more slowly, at a lower rate, and with worse retention than equivalent new pages that also receive internal links.
- Method
- Two matched groups of newly published pages on one site: sitemap-only (orphan) and sitemap-plus-links. Tracked daily via the URL Inspection API and server logs.
- Measure
- Days to first crawl, days to "indexed" status, share of each group indexed at the end of the window, share still indexed after a retention period.
- Duration
- 8 weeks from publication, plus a 4-week retention check.
Hypothesis
A sitemap tells Google a URL exists. It does not tell Google the URL matters. My expectation is that orphan pages will be crawled, some will be indexed, and a meaningful share will sit in "Discovered, currently not indexed" or drop out later, while linked pages will index faster and hold.
What would falsify it: if both groups reach the same indexed share within a similar number of days and retain it, then on this site, at this scale, the sitemap alone is doing the job and internal links are not a discovery or indexing lever. That would be a useful thing to know.
Why I tested it
Orphan pages are one of the most common findings in my audits, and the usual reaction is "but they are in the sitemap." Programmatic builds, migrated content and old campaign pages all end up in this state. I wanted an honest answer about how much that costs, because the fix, wiring pages into the site architecture, is real work and should be prioritised on evidence.
There is also a mechanism question. Indexing decisions appear to weigh signals about a page's importance, and internal links are one of the few importance signals a site owner fully controls. Testing it cleanly separates "Google can find it" from "Google thinks it is worth keeping."
Method
Create matched pages
[N] new pages in one template, similar length and structure, each on a distinct topic so they do not compete with each other. Topics are chosen to be comparably obscure, so neither group gets an easy win from demand.
Randomise into two arms
Half become orphans: listed in the sitemap, linked from nowhere on the site. Half are linked from one relevant category page and one relevant existing article. The assignment is recorded before publication.
Publish together
All pages go live in the same deploy and the sitemap is regenerated once. No manual "Request Indexing" for any page; that would be a third treatment.
Track daily
Server logs give first Googlebot hit per URL. The URL Inspection API gives index status per URL. Both are logged daily so I can plot time-to-event curves rather than a single end-of-window snapshot.
Retention check
After the main window closes, I leave everything untouched and re-check index status at the end of a second window. Indexing and staying indexed are different outcomes.
Controls and confounders: both arms share the sitemap, server, template and timing, so crawl budget and site-level trust are held constant. Things that could still distort the result include an algorithm update during the window, a change in sitemap processing frequency, and external links to any test page, which I check for and exclude. I also avoid publishing anything else on the site in the same week so the new pages are not competing with other fresh URLs for attention.
Implementation
The orphan arm is enforced, not assumed. Before publishing, I crawl the site with the new pages in place and confirm zero inlinks to each orphan URL, including from tag pages, author archives, "recent posts" widgets and the HTML sitemap page. Most accidental orphan-test failures come from a widget nobody remembered.
The linked arm receives exactly two contextual links per page, placed in body copy with the page's own title as anchor text. Two is deliberate: enough to be a real signal, few enough that I am not testing link volume.
# daily status pull (pseudocode)
for url in test_urls:
status = url_inspection(url) # Search Console API
first_hit = logs.first_googlebot(url) # server log parse
append(date, url, arm, status.verdict, status.coverage, first_hit)How to read the results
The clearest view is two curves: the cumulative share of each arm that has been crawled, and the cumulative share indexed, by day since publication. If the linked curve rises earlier and ends higher, the hypothesis holds. If the curves overlap, it does not.
The retention check is read as a simple comparison: of pages indexed at the end of window one, what share of each arm is still indexed at the end of window two. A large gap here would matter more to me than a small gap in speed, because it points at the "worth keeping" judgement rather than queue timing.
Data
Baseline tracking active. Daily status logs and indexation rates per arm are currently being recorded.
Result
This experiment is actively logging crawl frequency and index rates. Findings on indexing retention and crawl curves will be published once the retention window closes.
What I learned
Early crawl patterns indicate that sitemap-only discovery suffers from longer queue times; detailed lessons will follow.
Conclusion
Conclusive recommendations for internal linking versus sitemap discovery will be posted upon experiment completion.
Whatever the numbers show, the design is reusable. If you suspect orphan pages are hurting a site, run this on a small batch before rebuilding the architecture. The crawlability and indexability guide covers the mechanisms involved, and the audit framework includes the orphan-detection step I use before any test like this.