# NetCDF Output Specification for CruisePlan Phase 3c

## Overview

This specification defines CF-1.8 compliant NetCDF output formats for the CruisePlan system. The structure aligns with discrete sampling geometries from cfconventions.org, organizing cruise data by operation geometry type (point vs line) and providing a complete ship trajectory timeline.

## Key Design Principles

1. **CF-1.8 Compliance**: Strict adherence to CF discrete sampling geometries conventions
2. **Operation Geometry Organization**: Separate files based on spatial operation type
3. **Clean Schema Separation**: Pure navigation (`transits`) vs scientific operations (`surveys`)
4. **Ship Trajectory**: Complete time-ordered vessel path including all activities
5. **CruiseConfig Source**: All operation metadata extracted directly from YAML configuration

## CF Convention Alignment

- **FeatureType "point"**: Discrete observations at fixed locations (stations, moorings)
- **FeatureType "trajectory"**: Sequential observations along paths (surveys, ship schedule)
- **Coordinate Requirements**: All data variables include `coordinates` attribute
- **Standard Names**: Use CF standard name table for oceanographic variables

## YAML Schema Organization

The revised YAML schema cleanly separates operation types:

```yaml
# Point operations (featureType: point)
stations: [...]        # All fixed-location operations (CTD profiles, water sampling, calibrations, mooring deployments/recoveries)
# Each station entry includes an `operation_type` field, e.g. "station" or "mooring"

# Line operations (featureType: trajectory)
surveys: [...]         # Scientific operations along paths (ADCP, towing, bathymetry)

# Area operations (featureType: trajectory or specialized)
area_surveys: [...]    # Systematic coverage patterns, grid surveys

# Pure navigation (schedule file only)
transits: [...]        # Non-scientific vessel movement between operations
```

## Output File Structure

The system generates four NetCDF files organized by operation geometry and CF featureType:

### 1. Point Operations (`{cruise_name}_points.nc`)

**Purpose**: All discrete scientific operations at fixed locations
**Data Source**: `config.stations[]` + `config.moorings[]`
**FeatureType**: `point` (CF discrete sampling geometry)
**Use Case**: Station lists, mooring inventories, sampling locations

```python
# Dimensions
obs = N_point_operations  # Total stations + moorings

# Coordinate variables (spatial only)
lon(obs): float32
    standard_name = "longitude"
    long_name = "longitude of the operation"
    units = "degrees_east"

lat(obs): float32
    standard_name = "latitude"
    long_name = "latitude of the operation"
    units = "degrees_north"

# Oceanographic coordinate
waterdepth(obs): float32
    long_name = "water depth at operation location"
    standard_name = "sea_floor_depth_below_sea_surface"
    vocabulary = "http://vocab.nerc.ac.uk/collection/P07/current/CFV13N17/"
    units = "m"
    positive = "down"
    axis = "Z"
    _FillValue = -9999.0

# Operation metadata
name(obs): |S64
    long_name = "operation identifier from cruise plan"
    coordinates = "lat lon waterdepth"

operation(obs): |S64
    long_name = "type of oceanographic operation"
    flag_values = "CTD_profile water_sampling calibration Mooring_deployment Mooring_recovery"
    coordinates = "lat lon waterdepth"

duration(obs): float32
    long_name = "planned operation duration"
    units = "hours"
    coordinates = "lat lon waterdepth"
    _FillValue = -1.0

comment(obs): |S256
    long_name = "operation comments from cruise plan"
    coordinates = "lat lon waterdepth"

# Global attributes
:featureType = "point"
:title = "Point Operations: {cruise_name}"
:institution = "Generated by CruisePlan software"
:source = "YAML configuration file"
:Conventions = "CF-1.8"
:cruise_name = "{cruise_name}"
:creation_date = "{iso_timestamp}"
:coordinate_system = "WGS84"
:depth_datum = "Mean Sea Level"
```

### 2. Line Operations (`{cruise_name}_lines.nc`)

**Purpose**: Scientific operations along defined paths
**Data Source**: `config.surveys[]`
**FeatureType**: `trajectory` (CF discrete sampling geometry)
**Use Case**: ADCP surveys, tow-yo operations, bathymetry transects

```python
# Dimensions
operations = N_line_operations    # Number of discrete line operations
endpoints = 2                    # Start and end points (0=start, 1=end)

# Coordinate variables (2D arrays for start/end points)
longitude(operations, endpoints): float32
    long_name = "longitude coordinates defining line operation"
    units = "degrees_east"
    comment = "endpoints dimension: 0=start point, 1=end point"

latitude(operations, endpoints): float32
    long_name = "latitude coordinates defining line operation"
    units = "degrees_north"
    comment = "endpoints dimension: 0=start point, 1=end point"

# Operation metadata
operation_name(operations): |S64
    long_name = "line operation identifier"

operation_type(operations): |S32
    long_name = "type of line operation"
    flag_values = "underway towing"
    flag_meanings = "underway_instruments towed_instruments"

action(operations): |S32
    long_name = "specific survey instrument or method"
    flag_values = "ADCP bathymetry thermosalinograph tow_yo seismic microstructure"

vessel_speed(operations): float32
    long_name = "planned vessel speed for operation"
    units = "knots"
    _FillValue = NaN

estimated_duration(operations): float32
    long_name = "estimated operation duration"
    units = "hours"
    _FillValue = NaN

# Global attributes
:featureType = "trajectory"
:title = "Line Operations: {cruise_name}"
:institution = "Generated by CruisePlan software"
:source = "YAML configuration file"
:Conventions = "CF-1.8"
:cruise_name = "{cruise_name}"
:creation_date = "{iso_timestamp}"
```

### 3. Area Operations (`{cruise_name}_areas.nc`)

**Purpose**: Scientific operations over defined areas (future implementation)
**Data Source**: `config.area_surveys[]`
**FeatureType**: `trajectory` (coverage patterns) or specialized
**Use Case**: Grid surveys, systematic mapping, search patterns

```python
# Dimensions
operations = N_area_operations      # Number of area operations  
max_vertices = 20                   # Maximum vertices in any polygon

# Coordinate variables (2D arrays with ragged fill)
longitude(operations, max_vertices): float32
    long_name = "longitude vertices of area boundary polygon"
    units = "degrees_east"
    _FillValue = NaN
    comment = "Vertices define closed polygon boundary, unused positions filled with NaN"

latitude(operations, max_vertices): float32
    long_name = "latitude vertices of area boundary polygon"
    units = "degrees_north" 
    _FillValue = NaN
    comment = "Vertices define closed polygon boundary, unused positions filled with NaN"

# Vertex count for each operation
n_boundary_vertices(operations): int32
    long_name = "number of vertices in boundary polygon"
    valid_range = [3, 20]
    comment = "Actual number of vertices used (rest are NaN)"

# Operation metadata
area_name(operations): |S64
    long_name = "area operation identifier"

pattern_type(operations): |S32
    long_name = "area survey pattern"
    flag_values = "grid_survey systematic_mapping search_pattern random_sampling"

estimated_duration(operations): float32
    long_name = "estimated operation duration"
    units = "hours" 
    _FillValue = NaN

# Derived geometric properties (useful for analysis)
area_extent_km2(operations): float32
    long_name = "approximate area coverage"
    units = "km^2"
    _FillValue = NaN

# Global attributes
:featureType = "trajectory"  # or specialized area type
:title = "Area Operations: {cruise_name}"
:institution = "Generated by CruisePlan software"
:source = "YAML configuration file"
:Conventions = "CF-1.8"
:cruise_name = "{cruise_name}"
:creation_date = "{iso_timestamp}"
```

### 4. Ship Schedule (`{cruise_name}_schedule.nc`)

**Purpose**: Complete ship trajectory including all operations and transits
**Data Source**: Timeline from scheduler (`List[ActivityRecord]`)
**FeatureType**: `trajectory` (ship's continuous path)
**Use Case**: Operational timeline, logistics, voyage tracking

```python
# Dimensions
obs = N_timeline_events  # All scheduled activities in time order

# Coordinate variables (CF required)
time(obs): float64
    standard_name = "time"
    long_name = "time of ship position"
    units = "days since 1970-01-01 00:00:00"

lon(obs): float32
    standard_name = "longitude"
    long_name = "ship longitude"
    units = "degrees_east"

lat(obs): float32
    standard_name = "latitude"
    long_name = "ship latitude"
    units = "degrees_north"

# Schedule metadata
activity_name(obs): |S64
    long_name = "activity identifier"
    coordinates = "time lat lon"

activity_type(obs): |S32
    long_name = "type of cruise activity"
    flag_values = "point_operation line_operation area_operation transit"
    coordinates = "time lat lon"

operation_file_ref(obs): |S64
    long_name = "reference to operation in catalog files"
    coordinates = "time lat lon"
    comment = "Links to point/line/area operation files by name"

leg_assignment(obs): |S32
    long_name = "cruise leg identifier"
    coordinates = "time lat lon"

duration(obs): float32
    long_name = "activity duration"
    units = "hours"
    coordinates = "time lat lon"

vessel_speed(obs): float32
    long_name = "vessel speed"
    units = "knots"
    coordinates = "time lat lon"

# Global attributes
:featureType = "trajectory"
:title = "Ship Schedule: {cruise_name}"
:institution = "Generated by CruisePlan software"
:source = "Scheduler computation from YAML configuration"
:Conventions = "CF-1.8"
:cruise_name = "{cruise_name}"
:total_duration_days = {total_days}
:creation_date = "{iso_timestamp}"
```

## Implementation Requirements

### File Location and Output Directory
- All NetCDF files should be written to `tests_output/` for test cases
- Production outputs should allow user-specified output directory
- Filename format: `{cruise_name}_{type}.nc` where type is `points`, `lines`, `areas`, or `schedule`

### Integration with Current System

**Input Sources:**
1. **Point Operations**: Built from `config.stations[]` + `config.moorings[]`
2. **Line Operations**: Built from `config.surveys[]` (future schema enhancement)
3. **Area Operations**: Built from `config.area_surveys[]` (future implementation)
4. **Ship Schedule**: Built from `List[ActivityRecord]` returned by `generate_timeline()`

**Key Functions to Implement:**

```python
def generate_point_operations(config: CruiseConfig, output_path: Path) -> Path:
    """Generate point operations NetCDF from stations and moorings."""

def generate_line_operations(config: CruiseConfig, output_path: Path) -> Path:
    """Generate line operations NetCDF from surveys."""

def generate_area_operations(config: CruiseConfig, output_path: Path) -> Path:
    """Generate area operations NetCDF from area_surveys."""

def generate_ship_schedule(timeline: List[ActivityRecord], config: CruiseConfig, output_path: Path) -> Path:
    """Generate ship schedule NetCDF from timeline."""

def generate_all_netcdf_outputs(config: CruiseConfig, timeline: List[ActivityRecord], output_dir: Path) -> List[Path]:
    """Generate all NetCDF files and return paths."""
```

### Schema Enhancement Requirements

**Immediate (Phase 3c):**
1. Current system can generate point operations and ship schedule files
2. Line operations file requires new `surveys[]` list in YAML schema
3. Area operations file is placeholder for future development

**YAML Schema Updates Needed:**
```yaml
# Add to CruiseConfig validation
surveys: Optional[List[SurveyDefinition]] = []
area_surveys: Optional[List[AreaSurveyDefinition]] = []

class SurveyDefinition(BaseModel):
    name: str
    operation_type: str  # "underway" or "towing"
    action: str         # "ADCP", "bathymetry", "tow_yo", etc.
    route: List[GeoPoint]
    vessel_speed: Optional[float] = None

class AreaSurveyDefinition(BaseModel):
    name: str
    pattern_type: str   # "grid_survey", "search_pattern", etc.
    boundary: List[GeoPoint]
    # Additional area-specific fields
```

### CF Compliance Requirements

1. **FeatureType Specification**: Each file must have exactly one `featureType` global attribute
2. **Coordinate Variables**: Must include proper `units`, `standard_name`, and `long_name` attributes
3. **Coordinates Attribute**: All data variables must include `coordinates = "time lat lon ..."` attribute
4. **Time Variables**: Use CF-standard time encoding (`days since 1970-01-01 00:00:00`)
5. **Global Attributes**: Include mandatory CF attributes (`Conventions`, `title`, `institution`, `featureType`)
6. **Standard Names**: Use CF standard name table for oceanographic variables
7. **Fill Values**: Use appropriate `_FillValue` for missing or optional data

### Data Type Specifications

- **Coordinates**: `float64` for time precision, `float32` for spatial coordinates
- **Time**: `float64` (days since epoch for CF compliance)
- **Measurements**: `float32` for efficiency
- **Strings**: Variable length based on content (`|S32`, `|S64`, `|S256`)
- **Flags/Categories**: `|S32` for string enumerations

### Controlled Vocabulary

**Point Operation Types:**
- `"CTD_profile"` - CTD casts and water sampling
- `"water_sampling"` - Discrete water sample collection
- `"calibration"` - Instrument calibration activities
- `"Mooring_deployment"` - Mooring deployment operations
- `"Mooring_recovery"` - Mooring recovery operations

**Line Operation Types:**
- `"underway"` - Instruments operating while ship underway (nothing over the side)
- `"towing"` - Towed instruments or devices

**Line Action Types:**
- `"ADCP"` - Acoustic Doppler Current Profiler
- `"bathymetry"` - Multibeam or singlebeam mapping
- `"thermosalinograph"` - Underway temperature/salinity
- `"tow_yo"` - Tow-yo CTD operations
- `"seismic"` - Seismic surveys
- `"microstructure"` - Microstructure profilers

### Validation Requirements

Each generated file must:
1. Pass CF checker validation (`cf-checker` tool)
2. Include all required CF attributes for the specified featureType
3. Have consistent dimension sizes across variables
4. Include proper coordinate references for all data variables
5. Use valid CF standard names and units
6. Contain realistic values within expected ranges

### Cross-Reference System

**Operation Linking:**
- Use operation `name` fields as primary keys between files
- Ship schedule `operation_file_ref` links to catalog files
- Maintain naming consistency across all files

**File Relationships:**
1. Point operations file ← Ship schedule (via `operation_file_ref`)
2. Line operations file ← Ship schedule (via `operation_file_ref`)
3. Area operations file ← Ship schedule (via `operation_file_ref`)

## Integration with Test Framework

**Test Output Location**: All test-generated NetCDF files should be written to `tests_output/`

**Test Requirements**:
1. Generate NetCDF outputs from existing test fixtures (`tests/fixtures/*.yaml`)
2. Validate CF compliance using `cf-checker` or equivalent
3. Test cross-reference integrity between catalog and schedule files
4. Verify coordinate consistency and realistic value ranges

**Test Coverage**:
- Point operations only (stations and moorings)
- Mixed operations (points + transits)
- Complex multi-leg cruises
- Edge cases (empty operation lists, missing optional fields)

**Implementation Priority**:
1. **Phase 3c**: Point operations and ship schedule files (immediate)
2. **Future**: Line operations file (requires schema enhancement)
3. **Future**: Area operations file (placeholder implementation)

This specification provides a complete foundation for CF-1.8 compliant NetCDF outputs that properly separate operation types by geometry while maintaining scientific data standards and enabling cross-file analysis.
