# Story 12.11: Sitemap Generation for RTD (Stable/Canonical) and GH Pages (Latest/Edge)

**Status:** Complete
**Priority:** P1 (High)
**Depends on:** 12.12 (Custom Domain Setup) - for correct canonical URLs
**Epic:** 12.0 Cookbook (SEO-Driven Documentation)

---

## Context / Background

fapilog documentation is deployed to two platforms with different update cadences:

| Platform | Content | Updates | URL |
|----------|---------|---------|-----|
| **Read the Docs** | Stable/release docs | On release only | `docs.fapilog.dev/en/stable/` |
| **GitHub Pages** | Latest/edge docs | Every commit to main | `chris-haste.github.io/fapilog/` |

Both are publicly accessible and serve different purposes:
- **RTD**: Canonical reference for SEO, stable content users can rely on
- **GH Pages**: Preview unreleased documentation changes

For SEO to work correctly:
1. RTD needs a valid sitemap submitted to search engines
2. GH Pages must not compete with RTD for rankings (duplicate content penalty)
3. GH Pages should include canonical tags pointing to RTD stable URLs

**Why this matters for SEO:**
- Sitemaps help search engines discover and index pages
- Duplicate content across domains dilutes ranking signals
- Canonical tags tell search engines which URL is authoritative (RTD stable)

---

## Scope (In / Out)

### In Scope

- Configure RTD to generate correct sitemap.xml
- Configure GH Pages build to add canonical tags pointing to RTD stable
- GH Pages robots.txt to discourage indexing (prevents duplicate content SEO issues)
- CI pipeline updates for both deployment targets
- Verify sitemap submission to Google Search Console
- Link to GH Pages from docs (e.g., "Preview unreleased changes")

### Out of Scope

- Search Console account setup (user responsibility)
- RTD custom domain configuration (separate task)
- Page-level SEO optimizations (covered in 12.1-12.10)
- Analytics integration

---

## Acceptance Criteria

### AC1: RTD Sitemap Generated Correctly

**Description:** RTD deployment includes valid sitemap.xml with correct URLs.

**Validation:**
```bash
# Sitemap accessible
curl -s https://docs.fapilog.dev/sitemap.xml | head -20

# Contains expected pages
curl -s https://docs.fapilog.dev/sitemap.xml | grep -c "<url>"
# Should match number of documentation pages
```

### AC2: GH Pages Has Canonical Tags

**Description:** Every GH Pages HTML file includes canonical link to RTD equivalent.

**Validation:**
```html
<!-- Every page on GH Pages includes: -->
<link rel="canonical" href="https://docs.fapilog.dev/en/stable/{path}" />
```

```bash
# Verify canonical tag present
curl -s https://chris-haste.github.io/fapilog/cookbook/fastapi-request-id-logging.html \
  | grep -o '<link rel="canonical"[^>]*>'
# Should output: <link rel="canonical" href="https://docs.fapilog.dev/en/stable/cookbook/fastapi-request-id-logging.html" />
```

### AC3: GH Pages Discourages Search Indexing (Not Hidden)

**Description:** GH Pages is publicly accessible but discourages search engine indexing to avoid duplicate content. Users can still browse and link to it for previewing unreleased docs.

**Validation:**
```
# Option A: robots.txt (allows browsing, blocks indexing)
User-agent: *
Disallow: /

# Option B: Meta tag on each page (in addition to canonical)
<meta name="robots" content="noindex, follow">
```

Note: The site remains publicly accessible - only search engine crawling is discouraged.

### AC4: CI Pipeline Handles Both Targets

**Description:** Documentation CI builds correctly for each platform.

**Validation:**
- RTD build: Standard sphinx build, sitemap extension enabled
- GH Pages build: Sphinx build with canonical URL injection
- Both builds succeed in CI
- Artifacts deployed to correct locations

### AC5: Sitemap Registered with Search Console

**Description:** RTD sitemap submitted to Google Search Console.

**Validation:**
- [ ] Sitemap URL added to Search Console
- [ ] No crawl errors reported
- [ ] Pages being indexed (may take days/weeks to verify)

---

## Implementation Notes

### RTD Configuration

RTD automatically generates sitemaps if using Sphinx. Verify in `conf.py`:

```python
# docs/conf.py
extensions = [
    # ... other extensions
    'sphinx_sitemap',  # May not be needed - RTD has built-in support
]

# For sphinx_sitemap extension (if used)
html_baseurl = 'https://docs.fapilog.dev/en/stable/'
```

RTD may handle this automatically. Check RTD dashboard settings.

### GH Pages Canonical Injection

Option 1: Sphinx `html_context` in conf.py:
```python
# docs/conf.py
html_context = {
    'canonical_url': 'https://docs.fapilog.dev/en/stable/',
}
```

Then in theme template:
```html
<!-- _templates/layout.html or theme override -->
{% if canonical_url %}
<link rel="canonical" href="{{ canonical_url }}{{ pagename }}.html" />
{% endif %}
```

Option 2: Post-processing in CI:
```bash
# In GH Pages workflow, after sphinx build
find _build/html -name "*.html" -exec sed -i \
  's|<head>|<head>\n<link rel="canonical" href="https://docs.fapilog.dev/en/stable/{{ pagename }}.html" />|' {} \;
```

### GH Pages robots.txt

Create `docs/_static/robots.txt`:
```
# This is a mirror. Canonical docs at docs.fapilog.dev
User-agent: *
Disallow: /
```

Or use meta tag approach for more granular control.

### File Changes

```
docs/conf.py (MODIFIED - add canonical URL config)
docs/_static/robots.txt (NEW - for GH Pages)
docs/_templates/layout.html (NEW or MODIFIED - canonical tag injection)
.github/workflows/docs.yml (MODIFIED - GH Pages build with canonical injection)
.readthedocs.yaml (VERIFY - sitemap generation enabled)
```

---

## Tasks

### Phase 1: RTD Sitemap Verification

- [ ] Verify RTD is generating sitemap.xml
- [ ] Check sitemap contains all expected pages
- [ ] Confirm URLs use correct base (versioned or latest)
- [ ] Submit sitemap to Google Search Console

### Phase 2: GH Pages Canonical Tags

- [ ] Add canonical URL configuration to conf.py
- [ ] Create/modify template to inject canonical tags
- [ ] Test locally that canonical tags appear in HTML output
- [ ] Update GH Pages workflow to use canonical config

### Phase 3: GH Pages Indexing Prevention

- [ ] Add robots.txt to _static directory
- [ ] Or add noindex meta tag to template
- [ ] Verify robots.txt deployed correctly
- [ ] Test with Google's robots.txt tester

### Phase 4: CI Pipeline Updates

- [ ] Update GH Pages workflow for canonical injection
- [ ] Ensure RTD and GH Pages builds don't interfere
- [ ] Add build verification step
- [ ] Document the dual-deployment setup

---

## Tests

### Manual Verification

- [ ] RTD sitemap accessible and valid XML
- [ ] GH Pages pages have canonical tags
- [ ] GH Pages robots.txt blocks crawlers
- [ ] Both sites render correctly

### Automated (CI)

- [ ] Sitemap XML validation in RTD build
- [ ] Canonical tag presence check in GH Pages build
- [ ] robots.txt present in GH Pages deploy

---

## Definition of Done

### Infrastructure Complete

- [ ] RTD sitemap generated and accessible
- [ ] GH Pages has canonical tags on all pages
- [ ] GH Pages discourages indexing
- [ ] CI handles both deployments correctly

### SEO Verified

- [ ] Sitemap submitted to Search Console
- [ ] No duplicate content warnings
- [ ] Canonical tags validated

### Documented

- [ ] Dual-deployment strategy documented
- [ ] CI workflow documented
- [ ] Search Console setup steps documented

---

## Risks / Rollback

### Risks

1. **Risk:** RTD doesn't generate sitemap automatically
   - **Mitigation:** Add sphinx_sitemap extension explicitly

2. **Risk:** Canonical tags break theme styling
   - **Mitigation:** Test locally before deploying

3. **Risk:** robots.txt too aggressive, blocks legitimate access
   - **Mitigation:** Use `noindex` meta instead, allows browsing but not indexing

4. **Risk:** GH Pages users confused by "not indexed" messaging
   - **Mitigation:** Add banner linking to canonical RTD docs

### Rollback Plan

- Remove canonical tags: revert conf.py and template changes
- Remove robots.txt: delete from _static
- Both deployments continue to work independently

---

## Related Stories

- **Depends on:** 12.12 Custom Domain Setup (docs.fapilog.dev)
- **Related:** 12.0 Cookbook Epic (SEO strategy)
- **Related:** 10.15 Documentation Correctness CI Enforcement

---

## Code Review

**Date:** 2026-01-21
**Reviewer:** Claude
**Verdict:** OK to PR

### Summary

Implemented dual-mode Sphinx configuration for RTD sitemap generation and GH Pages SEO controls (canonical tags, robots.txt, noindex meta).

### AC Verification

| Criterion | Evidence |
|-----------|----------|
| AC1: RTD Sitemap | `docs/conf.py:181-184` - html_baseurl, sitemap_filename, sitemap_url_scheme configured |
| AC2: Canonical Tags | `docs/_templates/layout.html:16` - canonical link injection via template |
| AC3: Indexing Prevention | `docs/_extra/robots.txt:8-9` + `layout.html:17` - robots.txt and noindex meta |
| AC4: CI Pipeline | `.github/workflows/docs-deploy.yml:57-88` - env var config and verification step |
| AC5: Search Console | N/A - manual user step |

### Quality Gates

- [x] ruff check passed
- [x] ruff format passed
- [x] RTD build succeeds with sitemap
- [x] GH Pages build succeeds with canonical tags
- [x] CI verification step validates SEO elements

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-21 | Initial draft | Claude |
| 2026-01-21 | Implementation complete | Claude |
