# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

pyiwfm is a Python package for working with IWFM (Integrated Water Flow Model) models developed by the California Department of Water Resources. It provides tools for reading/writing IWFM model files (ASCII, binary, HDF5, HEC-DSS formats), mesh manipulation, simulation execution, PEST++ parameter estimation, and interactive web visualization.

## Commands

### Installation
```bash
pip install -e .              # Basic install (includes matplotlib, geopandas, shapely)
pip install -e ".[dev]"       # Development (pytest, mypy, ruff)
pip install -e ".[webapi]"    # Web viewer (FastAPI + vtk.js + deck.gl)
pip install -e ".[all]"       # All optional dependencies
```

### Testing
```bash
pytest tests/unit/                     # Run unit tests (what CI runs on every push)
pytest tests/unit/test_mesh.py -v      # Single test file
pytest tests/ --cov=pyiwfm             # With coverage
pytest tests/ -m slow                  # Run slow-marked tests
pytest tests/ -m integration           # Run integration tests (need model data, see below)
pytest tests/ -m roundtrip             # Run roundtrip read→write→read tests
pytest tests/ -m property              # Run property-based (Hypothesis) tests
```

Integration tests require model data via environment variables:
- `IWFM_SAMPLE_MODEL` — path to IWFM Sample Model dir (auto-downloads from CNRA if unset)
- `C2VSIMCG_DIR` — path to C2VSimCG model dir (required for C2VSimCG-specific tests)

### Code Quality
```bash
ruff format src/ tests/                # Format code
ruff check src/ tests/ --fix           # Lint with auto-fix
mypy src/pyiwfm/                       # Type checking
pre-commit run --all-files             # Run all pre-commit hooks
```

### Documentation
```bash
sphinx-build docs docs/_build          # Build Sphinx docs (requires .[docs] extra)
```

### CLI
```bash
pyiwfm viewer --model-dir /path/to/model   # Launch the web viewer
pyiwfm export --model-dir /path/to/model   # Export to VTK/GeoPackage
pyiwfm budget <control_file>               # Export budgets to Excel
pyiwfm zbudget <control_file>              # Export zone budgets to Excel
pyiwfm iwfm2obs --obs o.smp --sim s.smp --output out.smp   # Explicit SMP interpolation
pyiwfm iwfm2obs --model C2VSimFG.in --obs-gw o.smp --output-gw out.smp  # Model discovery mode
pyiwfm calctyphyd --water-levels wl.smp --weights w.txt --output typhyd.smp  # Typical hydrographs
pyiwfm calctyphyd --config CalcTypeHyd.in  # Fortran config mode with PEST output
```

### Frontend
```bash
cd frontend && npm install && npm run build   # Build React frontend to webapi/static/
cd frontend && npm run dev                    # Vite dev server with hot reload
cd frontend && npm run lint                   # ESLint check
```

### Docker
```bash
docker-compose up --build               # Build and start web viewer
docker run -p 8080:8080 -v /path/to/model:/model pyiwfm  # Run with model mounted
docker build -f dss-build/Dockerfile -t pyiwfm-dss .      # Full image with HEC-DSS support
docker-compose --profile dss up --build viewer-dss        # Compose with HEC-DSS
```
See `DOCKER.md` for full configuration (env vars: PORT, TITLE, MODE, MODEL_PATH).

## Architecture

### Source Layout
```
src/pyiwfm/
├── core/              # Mesh, Stratigraphy, TimeSeries, IWFMModel, BaseComponent ABC, model_factory
├── components/        # Groundwater, Stream, Lake, RootZone, SmallWatershed, UnsaturatedZone (all inherit BaseComponent)
├── io/                # 50+ file type readers/writers (ASCII, binary, HDF5, HEC-DSS), writer_config_base
│                      # Also: head_loader, hydrograph_reader, hydrograph_loader, area_loader,
│                      # cache_builder, cache_loader (generic data loaders, no web deps)
├── calibration/       # IWFM2OBS interpolation, model file discovery, obs well spec,
│                      # multi-layer T-weighted averaging, CalcTypHyd, fuzzy c-means clustering
├── runner/            # IWFMRunner (subprocess execution), PEST++ integration, Scenario manager
├── visualization/
│   ├── webapi/        # FastAPI viewer: config.py, server.py, routes/, services/, static/
│   │                  # Also contains slicing, properties (re-export shims for moved loaders)
│   ├── vtk_export.py  # VTKExporter (2D/3D mesh, PyVista)
│   └── ...            # Matplotlib plots, GIS export
├── templates/         # Jinja2 templates for IWFM file generation
│   ├── engine.py      # Hybrid Jinja2 + NumPy template engine
│   ├── filters.py     # Custom Jinja2 filters (fortran_float, fortran_int, etc.)
│   └── iwfm/          # Subdirs per component: groundwater, streams, lakes, rootzone, etc.
├── roundtrip/         # Read→write→read verification pipeline (config, differ, script generator)
├── mesh_generation/   # Triangle and Gmsh wrappers
├── comparison/        # Model diffing and metrics
└── cli/               # CLI entry points + _model_finder.py, _model_loader.py helpers

frontend/              # React + TypeScript + vtk.js + deck.gl (builds to webapi/static/)
```

### Key Classes
- **AppGrid**: Main mesh container with nodes, elements, faces, subregions (mirrors IWFM's Class_AppGrid)
- **IWFMModel**: Orchestrates all components (grid, stratigraphy, groundwater, streams, lakes, rootzone, small watersheds, unsaturated zone). Construction helpers live in `core/model_factory.py`.
- **BaseComponent**: ABC in `core/base_component.py` — all 6 component classes inherit from it (`validate()`, `n_items`)
- **BaseReader/BaseWriter**: Abstract I/O classes in `io/base.py` and `io/writer_base.py`
- **BaseComponentWriterConfig**: Shared config base in `io/writer_config_base.py` — inherited by all 6 component writer configs
- **CommentAwareReader/CommentAwareWriter**: Extended base classes for roundtrip comment preservation

### I/O System
The `io/` module handles 50+ IWFM file formats. Key patterns:
- **`iwfm_reader.py`** is the central module for all IWFM line-reading utilities. It provides: `COMMENT_CHARS`, `is_comment_line()`, `strip_inline_comment()`, `next_data_value()`, `next_data_line()`, `next_data_or_empty()`, `resolve_path()`, `parse_version()`, `version_ge()`, and `LineBuffer`. All 19+ io/ readers import from this module — never duplicate these helpers.
- Readers parse IWFM ASCII/binary files into Python objects
- Writers use Jinja2 templates via `templates/engine.py` (hybrid Jinja2 headers + NumPy array output)
- Comment preservation during roundtrip (read → write) via `comment_extractor.py` and `comment_writer.py`
- `load_complete_model()` / `save_complete_model()` for full model I/O (in `io/preprocessor.py`)
- `CompleteModelLoader` / `CompleteModelWriter` for advanced loading (in `io/model_loader.py` / `io/model_writer.py`)
- Each component (groundwater, streams, lakes, rootzone, etc.) has dedicated reader and writer modules
- `io/__init__.py` uses PEP 562 `__getattr__` for lazy submodule access; add new public classes/functions to its `__all__` and import blocks
- `head_all_converter` is intentionally NOT in `io/__init__.py`; import directly: `from pyiwfm.io.head_all_converter import convert_headall_to_hdf`

### Web Viewer Architecture
The viewer is a FastAPI backend + React SPA frontend with 4 tabs: Overview, 3D Mesh (vtk.js), Results Map (deck.gl + MapLibre), and Budgets (Plotly).

**Backend** (`visualization/webapi/`):
- `config.py` — `ModelState` singleton that holds the loaded `IWFMModel` and provides lazy getters for head data, budget data, stream reach boundaries, etc. Caches `node_id_to_idx`, `elem_id_to_idx`, and hydrograph locations for performance.
- `server.py` — FastAPI app creation with CRS configuration and static file serving
- `routes/` — 13 route modules: model (+ comparison), mesh, results (+ drawdown pagination, statistics), groundwater, streams, lakes, rootzone, small_watersheds, budgets, export (+ GeoPackage, matplotlib plots), observations, slices, properties
- Data loaders (`head_loader`, `hydrograph_reader`, `hydrograph_loader`, `area_loader`, `cache_builder`, `cache_loader`) now live in `io/`; backward-compat shims remain in `webapi/`
- Coordinate reprojection: server-side via `pyproj` (model CRS → WGS84), `--crs` CLI flag

**Frontend** (`frontend/`):
- State management: Zustand store in `stores/viewerStore.ts`
- API client: `api/client.ts` (typed fetch wrappers)
- 3D rendering: vtk.js in `components/Viewer3D/`
- 2D map: deck.gl + MapLibre in `components/ResultsMap/`
- Charts: Plotly in `components/BudgetDashboard/`
- UI: MUI (Material UI) components
- URL hash routing for tab navigation (#overview, #3d, #results, #budgets)
- Builds to `src/pyiwfm/visualization/webapi/static/` via Vite
- Path alias: `@` → `frontend/src/` (configured in `vite.config.ts`)
- Dev proxy: `/api` → `http://localhost:8080` (backend must be running separately)

### Test Fixtures
`tests/conftest.py` provides shared fixtures and helpers:
- `make_simple_grid()` — creates a 2x2 quad AppGrid (9 nodes, 4 elements) for unit tests
- `make_simple_stratigraphy()` — creates uniform-layer Stratigraphy for unit tests
- `fixtures_path` / `small_model_path` — paths to test fixture data in `tests/fixtures/`
- `mock_model_dir` — temp directory with Simulation/ and Preprocessor/ subdirs
- Integration tests (roundtrip, preprocessor/simulation runs) live in `tests/integration/`

### Calibration Pipeline
`calibration/` modules provide IWFM2OBS and CalcTypHyd functionality:
- `model_file_discovery.py` — `discover_hydrograph_files()` parses simulation main file → component main files → `.out` file paths and hydrograph metadata (bore IDs, layers, coordinates). Uses `iwfm_reader` utilities.
- `obs_well_spec.py` — `read_obs_well_spec()` reads multi-layer well specification files (name, x, y, element, screen top/bottom).
- `iwfm2obs.py` — Core interpolation (`interpolate_to_obs_times`, `interpolate_batch`, `compute_multilayer_weights`, `compute_composite_head`, `iwfm2obs`), plus integrated workflow (`iwfm2obs_from_model`) that auto-discovers `.out` files, reads them via `IWFMHydrographReader`, interpolates, and writes multi-layer outputs (`write_multilayer_output`, `write_multilayer_pest_ins`). Timestamp alignment via `_compute_model_dates` and `_replace_timestamps` (replicates Fortran `ComputeDate`). Layer expansion via `expand_obs_to_layers`, deduplication via `deduplicate_smp`.
- `calctyphyd.py` — Typical hydrograph computation by cluster. Config file parsing (`read_calctyphyd_config`, `CalcTypHydFileConfig`), Fortran-matching time-series output (`compute_typical_hydrographs_timeseries`), PEST `.out`/`.ins` writer (`write_pest_output`), date-range filtering, header-aware `read_cluster_weights(n_clusters=)`.
- `clustering.py` — Fuzzy c-means clustering of observation wells.
- `io/hydrograph_reader.py` — `IWFMHydrographReader` reads IWFM `.out` files. `get_columns_as_smp_dict()` bridges `.out` data to the interpolation pipeline as `SMPTimeSeries` dicts.
- `cli/iwfm2obs.py` — CLI subcommand with explicit SMP mode (`--obs/--sim/--output`), model-discovery mode (`--model`, `--obs-gw/--output-gw`, `--obs-stream/--output-stream`, `--well-spec`, `--multilayer-out/--multilayer-ins`), and `--deduplicate-smp` mode.

### PEST++ Integration
`runner/pest*.py` modules provide parameter estimation workflow:
- Template/instruction file generation
- Parameter bounds and groups
- Observation management
- Ensemble methods (prior/posterior)

## Versioning and Build

Version is derived from git tags via [hatch-vcs](https://github.com/ofek/hatch-vcs) — no hardcoded version string. `src/pyiwfm/_version.py` is auto-generated and gitignored.

- Tagged commit `v1.0.4` → version `1.0.4`
- 3 commits after tag → `1.0.5.dev3+gabcdef1`
- Release: `git tag -a v1.0.4 -m "Release v1.0.4" && git push && git push origin v1.0.4`

`hatch_build.py` is a custom build hook that auto-builds the React frontend during `hatch build` (requires npm; falls back to pre-built assets).

## CI/CD Pipeline

CI runs on every push/PR to `master` (see `.github/workflows/ci.yml`):
1. **Lint** — `ruff check` and `ruff format --check` (Python 3.12)
2. **Typecheck** — `mypy src/pyiwfm/` (Python 3.12)
3. **Test** — `pytest tests/unit/` on Ubuntu/Windows × Python 3.10–3.13; coverage on 3.12/Ubuntu (coverage `fail_under = 70`)
4. **Integration** — `pytest tests/integration/` with `.[all,dev]` (caches IWFM sample model)

**Publish** (`publish.yml`): triggered by `v*` tags — builds with `hatch build`, publishes to PyPI, and pushes Docker image to `ghcr.io`.

All lint, typecheck, and unit test jobs must pass before merge. Run `ruff check src/ tests/ && ruff format --check src/ tests/ && mypy src/pyiwfm/ && pytest tests/unit/` locally to verify before pushing.

## Code Style

- Python 3.10+ with type hints required (mypy strict mode)
- Line length: 100 characters
- NumPy-style docstrings
- Uses 1-based IDs to match IWFM Fortran conventions
- Pre-commit hooks: trailing whitespace, end-of-file-fixer, check-yaml, check-toml, check-added-large-files, ruff lint/format, mypy
- Ruff rules: E, W, F, I (isort), B (bugbear), C4 (comprehensions), UP (pyupgrade), NPY

## IWFM Domain Conventions

- All node/element/reach IDs are 1-based (Fortran convention)
- Datetime format in IWFM files: `MM/DD/YYYY_HH:MM` (note `_24:00` means end of day → next day 00:00)
- C2VSimFG CRS: UTM Zone 10N, NAD83, US survey feet (`+proj=utm +zone=10 +datum=NAD83 +units=us-ft +no_defs`). This is NOT EPSG:2227.
- Monthly budget timesteps (`1MON`) vary 28–31 days; use `dateutil.relativedelta`
- Stream `StrmNode` often has `x=0, y=0`; look up coordinates via `node.gw_node` → `grid.nodes[gw_node].x/y`

## TypeScript / Frontend Notes

- TypeScript strict mode enabled: unused imports/variables cause build failures (TS6133)
- Target is ES2020 — `new Map<K,V>()` with generics may not work; use `Record<string, T>` for caches
- Plotly title objects must use `{ text: 'Title' }` format, not bare strings
- All API responses should be typed; see existing patterns in `api/client.ts`
- Build step is `tsc && vite build` — TypeScript errors block the build

## Optional Dependencies

Core dependencies include matplotlib, geopandas, shapely, and pyogrio (always installed). The following extras have optional imports — handle ImportError gracefully:
- `mesh`: triangle, gmsh
- `viz`: vtk
- `webapi`: fastapi, uvicorn, pydantic, pyvista, vtk, pyproj, python-multipart
- `dss`: bundled HEC-DSS 7 C library (`io/dss/lib/hecdss.dll`) with ctypes wrapper. On Linux, build `libhecdss.so` from source via `dss-build/` (see `dss-build/build_hecdss.py`). Set `HECDSS_LIB` env var to override library path.
- `pest`: scipy
