# Migration Plan: Convert Documentation to markdown-exec

## Problem Statement

All documentation markdown files currently contain static code blocks with manually-written outputs. These need to be converted to use markdown-exec's executable code blocks so that:
1. Code actually runs during documentation build
2. Outputs are real and stay in sync with code changes
3. Build fails if examples break

The old system used context names (e.g., `exec="simple_signal"`) to inject pre-built setups. We need to replicate this functionality using markdown-exec's session feature while minimizing code duplication.

## Current State Analysis

### Files Requiring Conversion (by priority)

**High Priority** (user-facing guides with many examples):
- `docs/user-guide/backend-storage.md` (22 code blocks)
- `docs/getting-started/basic-concepts.md` (13 blocks)
- `docs/user-guide/datasets.md` (7 blocks)
- `docs/user-guide/visualization.md` (7 blocks)
- `docs/user-guide/processing-steps.md` (6 blocks)
- `docs/user-guide/signals.md` (4 blocks) ✅ Already done as example
- `docs/user-guide/time-series.md` (4 blocks)
- `docs/user-guide/saving-loading.md` (5 blocks)

**Medium Priority** (examples and getting started):
- `docs/examples/basic-workflow.md` (4 blocks)
- `docs/examples/custom-processing.md`
- `docs/examples/real-world-cases.md`
- `docs/getting-started/quickstart.md` (4 blocks)

**Low Priority** (API reference, mostly static):
- API reference files (can remain static for now)
- Metadata dictionary files (mostly auto-generated)

### Old Context System

The old system provided these pre-built contexts:
- `simple_signal`: One temperature signal with basic setup
- `dataset`: Three signals (temp, pH, DO) in a dataset
- `processed_signal`: Signal with processing already applied
- `problematic_data`: Signal with gaps/outliers for processing demos
- `multi_signals`: Multiple signals without dataset wrapper
- `processing`: Specialized for processing function demos
- `custom_functions`: For writing custom processing functions

## Solution Approach

### Option 1: Session-Based with Centralized Setup Files ⭐ RECOMMENDED

**Approach:**
1. Create reusable "snippet" markdown files in `docs/snippets/` directory
2. Use markdown-exec's `source="file"` feature to include common setups
3. Each doc page starts by executing a setup snippet in a session
4. Continue using that session for all examples on the page

**Pros:**
- ✅ Zero code duplication - setup code in one place
- ✅ Easy to maintain - update snippet, all docs update
- ✅ Clear separation between setup and examples
- ✅ Can mix-and-match snippets for different scenarios

**Cons:**
- ❌ Requires markdown-exec to support file inclusion (need to verify)
- ❌ Slightly more complex for contributors to understand

**Structure:**
```
docs/
  snippets/
    setup_imports.md        # Base imports
    setup_simple_signal.md  # Single signal setup
    setup_dataset.md        # Multi-signal dataset setup
    setup_problematic_data.md  # Data with issues
  user-guide/
    signals.md   # Uses snippets + own examples
    datasets.md  # Uses snippets + own examples
```

### Option 2: Hidden Code Blocks for Setup ⭐ FALLBACK

**Approach:**
1. Each doc page has an initial hidden setup block
2. Use `source="below"` or `html="0"` to hide setup code
3. Continue using same session for visible examples

**Pros:**
- ✅ Works with standard markdown-exec features
- ✅ Setup is in same file as usage
- ✅ Contributors see setup when editing

**Cons:**
- ❌ Code duplication across files
- ❌ Maintenance burden - update setup in many places
- ❌ Large hidden blocks at start of each file

**Example:**
```markdown
<!-- Hidden setup block -->
```python exec="1" source="below" session="signals"
import numpy as np
import pandas as pd
from meteaudata import Signal, DataProvenance
np.random.seed(42)
timestamps = pd.date_range('2024-01-01', periods=100, freq='h')
# ... lots more setup code ...
```

<!-- Now the visible examples -->
```python exec="1" session="signals"
print(f"Signal: {signal.name}")
```
```

### Option 3: Macro-Based Setup

**Approach:**
1. Use mkdocs-macros plugin to define Python macros
2. Macros return setup code as strings
3. Include macro output in code blocks

**Pros:**
- ✅ Setup code in one place
- ✅ Can be parameterized

**Cons:**
- ❌ Requires adding another plugin
- ❌ More complex for contributors
- ❌ Mixes template syntax with markdown

## Recommended Solution: Option 2 (Hidden Blocks)

**Rationale:** After checking markdown-exec docs, the `source="file"` feature is not available. Option 2 is the most practical approach that:
- Works with current markdown-exec capabilities
- Keeps setup code visible to editors (in source)
- Allows zero-duplication for common patterns through templates/includes

### Implementation Strategy

1. **Create setup templates** in `docs/_templates/` (not rendered):
   - `_setup_simple_signal.py` - Single signal setup
   - `_setup_dataset.py` - Multi-signal dataset setup
   - `_setup_problematic.py` - Data with quality issues

2. **Document template usage** in contributor guide:
   - How to copy appropriate template
   - How to customize for specific page needs
   - Encourage sharing common setups

3. **Conversion workflow** for each file:
   1. Identify what data the examples need
   2. Copy appropriate template to start of file
   3. Add `exec="1" source="tabbed-left" session="pagename"`
   4. Make examples executable with same session
   5. Remove manual **Output:** blocks
   6. Test build to verify

4. **Prioritize conversion:**
   - Start with user-guide files (most visible)
   - Then getting-started
   - Then examples
   - API reference can wait (mostly auto-generated)

## Detailed Conversion Plan

### Phase 1: Create Templates (Week 1, Day 1)
- [ ] Create `docs/_templates/` directory (excluded from site build)
- [ ] Create `_setup_simple_signal.py` based on old `simple_signal` context
- [ ] Create `_setup_dataset.py` based on old `dataset` context
- [ ] Create `_setup_problematic.py` based on old `processing` context
- [ ] Document template usage in `docs/development/executable-code-docs.md`

### Phase 2: Convert User Guide (Week 1, Days 2-3)
- [x] signals.md ✅ (already done)
- [ ] datasets.md
- [ ] time-series.md
- [ ] processing-steps.md
- [ ] visualization.md
- [ ] saving-loading.md
- [ ] backend-storage.md (largest file, do last)

### Phase 3: Convert Getting Started (Week 1, Day 4)
- [ ] quickstart.md
- [ ] basic-concepts.md
- [ ] installation.md (minimal changes needed)

### Phase 4: Convert Examples (Week 2, Day 1)
- [ ] basic-workflow.md
- [ ] custom-processing.md
- [ ] real-world-cases.md

### Phase 5: Cleanup (Week 2, Day 2)
- [ ] Remove any remaining old template references
- [ ] Update CHANGELOG
- [ ] Update contributor documentation
- [ ] Test full documentation build
- [ ] Commit changes

## Testing Strategy

For each converted file:
1. Run `uv run mkdocs build`
2. Check for markdown-exec warnings
3. Verify output makes sense in rendered docs
4. Check for missing variables/imports

Final validation:
1. Clean build: `rm -rf site && uv run mkdocs build --strict`
2. Serve and manually review: `uv run mkdocs serve`
3. Check all code blocks executed successfully

## Risk Mitigation

**Risk:** Templates become out of sync with actual code
- **Mitigation:** Keep templates simple, document well, use in many places so breakage is obvious

**Risk:** Too much duplication still exists
- **Mitigation:** Identify common patterns during conversion, extract to templates

**Risk:** Build becomes slow with many executable blocks
- **Mitigation:** Consider caching strategy, or make some blocks static if needed

**Risk:** Contributors confused by new system
- **Mitigation:** Clear documentation, examples in executable-code-docs.md

## Alternative Considered: mkdocs-gen-files

We could use mkdocs-gen-files to programmatically generate markdown with setup code injected. However:
- More complex than needed
- Harder for contributors to understand
- Loses direct editability of markdown
