## 2026-03-20 Investigation: GSC Sitemap Pages vs Indexed Pages Discrepancy

### Problem
- Description: Sitemap reported 87 pages in GSC, but low-value/utility pages dilute
  indexing signals and waste crawl budget
- Location: doc/conf.py, doc/sphinxext/

### Cause
The sitemap included 19 low-value pages that should not be advertised to search
engines:
- `_modules/*.html` (15 pages): auto-generated source code views (duplicate of API docs)
- `_modules/index.html` (1 page): module list page
- `genindex.html`, `py-modindex.html`: auto-generated index pages
- `search.html`, `opensearch.html`: utility pages with no indexable content
Additionally, none of these pages had `noindex` meta tags.

### Proposed Fix (implemented)
- [x] Add `sitemap_excludes` in conf.py to remove low-value pages from sitemap.xml
- [x] Create `doc/sphinxext/noindex_utilities.py` to inject
  `<meta name="robots" content="noindex">` on `_modules/*` and utility pages
- [ ] Deploy and resubmit sitemap in GSC
- [ ] Monitor GSC indexing status over 2-4 weeks

### Sitemap page count
- v0.8.1 (deployed): 87 URLs (68 content + 19 low-value)
- main (after fix): 82 URLs (all content, low-value excluded)
  - net change is only -5 because main has ~14 new content pages vs v0.8.1
  - the 19 excluded pages are correctly removed

### Verification
- `sitemap_excludes`: 0 low-value pages in sitemap
- `noindex` meta tag confirmed on `search.html`, `genindex.html`, `_modules/` pages
- `_modules/` pages still generated (for `[source]` links) but excluded from sitemap
  and marked noindex
- Normal content pages unaffected

### References
- https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
- https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
- https://sphinx-sitemap.readthedocs.io/en/latest/advanced-configuration.html
