# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This repository implements SHACL-based data quality validation for Cognite Data Fusion (CDF). It validates **DMS instances** (graph data), **time series data**, and **RAW table rows** using Cognite Functions and Workflows. All validation results are posted to the CDF Records API.

**Key Components:**
- Instance validation (real-time and historic)
- RAW table validation (scheduled and historic/partitioned)
- Time series quality validation (scheduled)
- Multi-environment support (separate configs per environment)
- Automated CI/CD deployment via GitHub Actions

**Version Information:**
- `cognite-data-quality`: 0.0.0 (function handlers, inlined SHACL validation, and deployment automation)

## Architecture

For deep architectural details, see `ARCHITECTURE.md`. Key patterns:

**Instance Validation Modes:**
1. **Real-time**: Data-change triggered workflows validate instances within seconds
2. **Historic**: Orchestrator calculates partitions, triggers parallel validation functions with cursor-based resumption

**RAW Table Validation Modes:**
1. **Incremental**: Scheduled workflows validate rows updated since last run (using `lastUpdatedTime`)
2. **Historic/Partitioned**: Uses CDF getCursors API to fetch N cursors, distributes to parallel workers with cursor resumption on timeout

**Time Series Validation:**
- Scheduled CRON-based workflows
- SHACL rules with CDF SDK and INDSL functions for quality metrics
- Supports backfill mode for historic data

**Partitioning Strategy:**
- DMS: Orchestrator queries DMS once for data distribution, creates N range-filtered partitions
- RAW: Uses getCursors API to distribute cursors based on table size and time ranges
- Each partition validates independently with cursor resumption on 480-second timeout

## Environment Configuration

**CRITICAL**: This codebase uses multi-environment configuration via the `CONFIG_ENV` environment variable.

```bash
# Always set CONFIG_ENV before running scripts
export CONFIG_ENV="cog-ai"     # or "abp-dev"
```

Environment configs are in `config/environments/{CONFIG_ENV}/`:
- `settings.yaml` - Function, workflow, and records settings
- `views/` - Instance validation configs (one YAML per view)
- `shacl_rules/` - SHACL Turtle files
- `timeseries/` - Time series validation configs

**GitHub Environments:**
- `cog-ai-arn` (uses CONFIG_ENV=cog-ai)
- `abp-dev` (uses CONFIG_ENV=abp-dev)

## Deployment

### Function Code Packaging

**Function handlers are deployed via PyPI package:**
- All validation handlers are packaged in `cognite-data-quality` PyPI package (includes `cognite_data_quality/_function_code/`)
- Functions deployed to CDF specify `cognite-data-quality==0.0.0` as a dependency
- CDF automatically installs the package from PyPI when executing functions
- Function code updates require: (1) updating package version, (2) publishing to PyPI, (3) redeploying function

**Publishing to PyPI:**
```bash
# 1. Update version in pyproject.toml
# version = "0.0.1"  # Increment this

# 2. Clean previous builds
rm -rf dist/ build/ *.egg-info

# 3. Build package
python -m build

# 4. Upload to PyPI (requires credentials)
twine upload dist/*

# 5. Verify package is available
pip install cognite-data-quality==0.0.1 --upgrade

# 6. Deploy function to CDF (pulls new version)
python scripts/deploy_function.py --force
```

**Testing Before Publishing:**
- Always test function code locally using notebooks (e.g., `test_raw_shacl_validation.ipynb`)
- Validate with `pytest` before publishing
- Consider publishing to TestPyPI first for verification

**Deployment Flow:**
1. Update function code in `cognite_data_quality/_function_code/`
2. Increment version in `pyproject.toml`
3. Publish package to PyPI: `python -m build && twine upload dist/*`
4. Deploy function to CDF (automatically pulls latest package)

### CI/CD Deployment (Primary Method)

**Automatic:**
- Push to `main` → deploys all changes automatically
- Pull request → validates configurations only

**Manual Dispatch:** Workflow supports manual trigger with options:
- Force function/workflow redeployment
- Deploy specific views, timeseries, or RAW table configs
- Dry-run mode
- Skip time series or demo data deployment

**Smart Deployment:** Uses content hashing to skip unchanged deployments:
- Functions: Only redeploy if code/dependencies change (checks package versions)
- Workflows: Only redeploy if configuration changes
- SHACL rules: Uploaded to CDF Files dataset on every workflow deployment

### Local Deployment

**Prerequisites:**
```bash
# Required environment variables
export COGNITE_CLUSTER=api
export COGNITE_PROJECT=my-project
export COGNITE_CLIENT_ID=your-client-id
export COGNITE_CLIENT_SECRET=your-client-secret
export AZURE_TENANT_ID=your-tenant-id
export CONFIG_ENV=cog-ai

# Install dependencies
pip install cognite-sdk pyyaml rdflib
```

**Deployment Commands:**
```bash
cd scripts

# Validate configuration first
python validate_config.py

# Deploy step-by-step
python ensure_container.py                       # 1. Create Records API containers (includes RawValidationState)
python deploy_datamodels.py                      # 2. Deploy data models and views
python deploy_function.py                        # 3. Deploy all functions
python deploy_workflows.py                       # 4. Deploy instance validation workflows
python deploy_timeseries_workflows.py            # 5. Deploy time series validation workflows
python deploy_raw_workflows.py                   # 6. Deploy RAW table validation workflows

# Deploy specific components
python deploy_function.py --function instances
python deploy_function.py --function timeseries
python deploy_function.py --function orchestrator
python deploy_function.py --function partitioned
python deploy_workflows.py --views equipment maintenance_order
python deploy_timeseries_workflows.py --configs demo_timeseries_quality
python deploy_raw_workflows.py --tables iot_sensors  # Deploy specific RAW table workflows

# Deployment flags
python deploy_function.py --dry-run              # Preview changes
python deploy_function.py --force                # Force redeployment
python deploy_workflows.py --skip-trigger        # Skip trigger deployment
```

**Deploy Demo Data Generator:**
```bash
# Only if demo_data.enabled: true in settings.yaml
python deploy_demo_data.py
```

## Development Workflow

### Configuration Validation

**Always validate before deploying:**
```bash
export CONFIG_ENV=cog-ai
python scripts/validate_config.py
```

This checks:
- YAML syntax in all config files
- SHACL Turtle syntax in .ttl files
- Required fields and file references
- View and data model structure

### Local Testing with Notebooks

Development notebooks in `test_and_deploy/`:
- `testing_and_exploration.ipynb` - Test validation logic locally
- `test_raw_shacl_validation.ipynb` - Test RAW table validation (template generation, incremental, historic)
- `deploy.ipynb` - Manual deployment
- `test_and_deploy.ipynb` - End-to-end testing

Root-level notebooks:
- `historic_data_validation.ipynb` - Test historic validation
- `explore_validation_records.ipynb` - Analyze validation results
- `test_orchestrator_monitor.ipynb` - Monitor orchestrator executions

**Testing RAW Validation Locally:**

Before deploying RAW validation to CDF Functions, test locally with `test_raw_shacl_validation.ipynb`:
1. Generate SHACL template from RAW table schema
2. Test validation with `with_shacl_raw()`
3. Test incremental and cursor-based processing
4. Test handler functions locally (simulating CDF Function execution)
5. Verify cursor state management

This allows you to validate functionality before publishing new package versions to PyPI.

### Validation Dashboard

```bash
cd test_and_deploy
streamlit run validation_dashboard.py
```

Shows validation pass/fail rates, failed constraints, and per-instance history.

## Adding New Validations

### Adding Instance Validation

1. **Create SHACL rules** in `config/environments/{CONFIG_ENV}/shacl_rules/my_view_shacl.ttl`
2. **Create view config** in `config/environments/{CONFIG_ENV}/views/my_view.yaml`:

```yaml
view:
  space: "my_space"
  external_id: "MyView"
  version: "v1"

instance_space: "my_instances"

shacl_rules:
  file: "my_view_shacl.ttl"
  external_id: "my_view_shacl"

datamodel:
  space: "my_space"
  external_id: "MyDataModel"
  version: "v1"

validation:
  auto_load_depth: 2  # Auto-load referenced instances (0-3)
  verbose: true

records:
  rule_set_id: "MyViewSHACLv1"
  rule_set_version: "1.0"
```

3. **Deploy:**
```bash
export CONFIG_ENV=cog-ai
python scripts/validate_config.py
python scripts/deploy_workflows.py --views my_view
```

### Adding Time Series Validation

1. **Create SHACL rules** in `config/environments/{CONFIG_ENV}/shacl_rules/my_timeseries_quality.ttl`
2. **Create config** in `config/environments/{CONFIG_ENV}/timeseries/my_sensors_quality.yaml`:

```yaml
name: "my-sensors-quality"
description: "Quality checks for my sensors"

# Option 1: DMS filter (server-side filtering)
filter:
  instance_space: "my_space"
  view:
    space: "cdf_cdm"
    external_id: "CogniteTimeSeries"
    version: "v1"
  expression:
    type: "prefix"
    property: ["node", "externalId"]
    value: "sensor_"
  limit: 500

# Option 2: Explicit instance IDs
# instance_ids:
#   - space: "my_space"
#     external_id: "sensor_001"

shacl_rules:
  file: "my_timeseries_quality.ttl"
  external_id: "my_sensors_shacl"

datamodel:
  space: "cdf_cdm"
  external_id: "CogniteCore"
  version: "v1"

validation:
  auto_load_depth: 0
  verbose: true

records:
  rule_set_id: "MySensorsQuality"
  rule_set_version: "1.0"

schedule:
  cron: "0 * * * *"  # Hourly
```

3. **Deploy:**
```bash
export CONFIG_ENV=cog-ai
python scripts/validate_config.py
python scripts/deploy_timeseries_workflows.py --configs my_sensors_quality
```

### Adding RAW Table Validation

**Step 1: Generate SHACL Template**

Use `cognite_data_quality` to auto-generate a SHACL template from your RAW table schema:

```python
from cognite_data_quality._neat import NeatSession
from cognite.client import CogniteClient

client = CogniteClient()
neat = NeatSession(client)

# Generate template by analyzing first 1000 rows
shacl_template = neat.validate_instances.generate_shacl_template_for_raw(
    db_name="iot",
    table_name="sensors",
    sample_size=1000,
    required_columns=["device_id", "timestamp"],  # Columns that must be present
    verbose=True,
)

# Save template for editing
with open("iot_sensors_shacl_template.ttl", "w") as f:
    f.write(shacl_template)
```

**Step 2: Customize SHACL Rules**

Edit the generated template to add custom constraints:

```turtle
# config/environments/{CONFIG_ENV}/shacl_rules/iot_sensors_shacl.ttl
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix raw_iot_sensors: <http://purl.org/cognite/raw/iot/sensors/> .

# Auto-generated rules...
raw_iot_sensors:DeviceIdShape
    a sh:NodeShape ;
    sh:targetClass raw_iot_sensors:sensors ;
    sh:property [
        sh:path raw_iot_sensors:device_id ;
        sh:minCount 1 ;
        sh:datatype xsd:string ;
    ] .

# Custom rule: Temperature must be in valid range
raw_iot_sensors:TemperatureRangeShape
    a sh:NodeShape ;
    sh:targetClass raw_iot_sensors:sensors ;
    sh:property [
        sh:path raw_iot_sensors:temperature ;
        sh:datatype xsd:double ;
        sh:minInclusive -40.0 ;
        sh:maxInclusive 120.0 ;
        sh:message "Temperature must be between -40 and 120 degrees" ;
    ] .
```

**Step 3: Create RAW Table Config**

```yaml
# config/environments/{CONFIG_ENV}/raw_tables/iot_sensors.yaml
name: iot_sensors
description: "IoT sensor data quality validation"

raw:
  db_name: iot
  table_name: sensors
  foreign_keys:  # Optional: columns that reference other rows
    - asset_id
  table_type: null  # Optional: custom RDF type (defaults to table_name)

shacl_rules:
  file: iot_sensors_shacl.ttl
  external_id: iot-sensors-shacl-rules

validation:
  batch_size: 10000  # Rows per incremental run
  verbose: true

records:
  rule_set_id: IoTSensorsSHACL
  rule_set_version: "1.0"
  data_domain_external_id: RawIoTSensors

schedule:
  cron: "*/15 * * * *"  # Every 15 minutes
```

**Step 4: Deploy**

```bash
export CONFIG_ENV=cog-ai
python scripts/validate_config.py
python scripts/deploy_raw_workflows.py --tables iot_sensors
```

**Validation Modes:**

1. **Incremental (Scheduled):** Validates rows updated since last run
   - Uses `lastUpdatedTime` field on RAW rows
   - Stores state in `RawValidationState` container
   - Runs on cron schedule (e.g., every 15 minutes)

2. **Historic/Partitioned:** Validates all existing rows using parallel workers
   - Uses CDF getCursors API to distribute work
   - Each worker processes independently with cursor resumption
   - Stops after 480 seconds and resumes on next invocation
   - Run manually via workflow trigger or orchestrator

## SHACL Rule Development

### RAW Table URI Scheme

RAW rows are converted to RDF using the following pattern:

**URI Pattern:**
- Subject: `http://purl.org/cognite/raw/{db_name}/{table_name}/#{row_key}`
- Type: `rdf:type http://purl.org/cognite/raw/{db_name}/{table_name}/#{table_name}`
- Properties: `http://purl.org/cognite/raw/{db_name}/{table_name}/#{column_name}`

**Example RDF for RAW row:**
```turtle
@prefix raw_iot_sensors: <http://purl.org/cognite/raw/iot/sensors/> .

raw_iot_sensors:sensor_001
    a raw_iot_sensors:sensors ;
    raw_iot_sensors:device_id "sensor_001" ;
    raw_iot_sensors:temperature 22.5 ;
    raw_iot_sensors:humidity 45.0 ;
    raw_iot_sensors:active true .
```

**SHACL Rules for RAW Tables:**
```turtle
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix raw_iot_sensors: <http://purl.org/cognite/raw/iot/sensors/> .

# Target all rows in the sensors table
raw_iot_sensors:SensorValidationShape
    a sh:NodeShape ;
    sh:targetClass raw_iot_sensors:sensors ;
    sh:property [
        sh:path raw_iot_sensors:device_id ;
        sh:minCount 1 ;
        sh:datatype xsd:string ;
        sh:message "device_id is required" ;
    ] ;
    sh:property [
        sh:path raw_iot_sensors:temperature ;
        sh:datatype xsd:double ;
        sh:minInclusive -40.0 ;
        sh:maxInclusive 120.0 ;
        sh:message "Temperature must be between -40 and 120 degrees" ;
    ] .
```

### Time Windows in SHACL Rules

Use placeholders that get replaced at runtime:
- `"60m-ago"`, `"90m-ago"`, `"24h-ago"`, `"7d-ago"` - Relative time
- `"now"` - Current time

**Example:**
```turtle
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix cdf_sdk: <http://purl.org/cognite/cdf_sdk#> .

:DataDensityShape a sh:NodeShape ;
    sh:targetClass <http://purl.org/cognite/cdf_cdm/v1/CogniteTimeSeries> ;
    sh:sparql [
        sh:message "Time series must have at least 50 datapoints in the last hour" ;
        sh:select """
            SELECT $this ?count
            WHERE {
                BIND(cdf_sdk:datapoints_count($this, "60m-ago", "now") AS ?count)
                FILTER (?count < 50)
            }
        """ ;
    ] .
```

### CDF SDK Functions in SHACL

Available functions for time series validation:
- `cdf_sdk:datapoints_count(?ts, start, end)` - Count datapoints
- `cdf_sdk:datapoints_average(?ts, start, end)` - Average value
- `cdf_sdk:datapoints_min(?ts, start, end)` - Minimum value
- `cdf_sdk:datapoints_max(?ts, start, end)` - Maximum value
- `cdf_sdk:timeseries_exists(?ts)` - Check existence

### INDSL Functions in SHACL

Available INDSL quality functions:
- `cdf_indsl:extreme_outliers(?ts, alpha)` - Detect outliers
- `cdf_indsl:gaps_identification(?ts, threshold)` - Detect gaps
- `cdf_indsl:value_decrease_check(?ts)` - Check for decreases
- `cdf_indsl:out_of_range(?ts)` - Detect out-of-range values

## Key File Paths

**Cognite Functions (in cognite_data_quality package):**
- `cognite_data_quality/_function_code/handler.py` - Unified validation function dispatcher (routes by validation_type)
- `cognite_data_quality/_function_code/handlers/instance_validation.py` - Real-time instance validation
- `cognite_data_quality/_function_code/handlers/partitioned_validation.py` - Historic partition validation
- `cognite_data_quality/_function_code/handlers/raw_validation.py` - RAW table validation (incremental and historic)
- `cognite_data_quality/_function_code/handlers/timeseries_validation.py` - Time series validation
- `cognite_data_quality/_function_code/handlers/orchestrator.py` - Orchestrator for historic validation
- `cognite_data_quality/_function_code/handlers/test_rule.py` - Test SHACL rules without writing
- `cognite_data_quality/_function_code/common/` - Shared utilities (cursor_state, records_api, shacl_utils, time_window, raw_state)
- `cognite_data_quality/_generate_demo_data/handler.py` - Demo data generator

**Validation Types (for handler routing):**
- `instance` - Real-time DMS instance validation
- `partitioned` - Historic DMS partition validation
- `raw_incremental` - Scheduled RAW table validation (timestamp-based)
- `raw_historic` - Historic RAW table validation (cursor-based with partitioning)
- `timeseries` - Time series quality validation
- `orchestrator` - Orchestrator for historic DMS validation
- `test` - Test SHACL rules without creating records

**Deployment Scripts:**
- `scripts/validate_config.py` - Validate all configurations
- `scripts/ensure_container.py` - Create Records API containers (includes RawValidationState)
- `scripts/deploy_datamodels.py` - Deploy data models and views
- `scripts/deploy_function.py` - Deploy Cognite Functions
- `scripts/deploy_workflows.py` - Deploy instance validation workflows
- `scripts/deploy_timeseries_workflows.py` - Deploy time series workflows
- `scripts/deploy_raw_workflows.py` - Deploy RAW table validation workflows
- `scripts/deploy_demo_data.py` - Deploy demo data generator
- `scripts/shared.py` - Common utilities

**Utility Scripts:**
- `scripts/delete_state_instances.py` - Clean up cursor state
- `scripts/test_partition_filter.py` - Test partition filtering
- `scripts/test_reverse_relations.py` - Test reverse relation validation

**Root-level utilities:**
- `check_property_coverage.py` - Analyze SHACL property coverage
- `inspect_single_instance.py` - Inspect a single instance validation
- `manual_verification.py` - Manual validation testing
- `verify_validation_results.py` - Verify validation results in Records API
- `validation_dashboard.py` - Streamlit dashboard (deprecated, use test_and_deploy/)

## Important Implementation Details

### Reverse Direct Relations

The system supports validating reverse direct relations in SHACL rules. Use the `neat:reverseDirectRelation` property to validate relationships from the target back to the source.

**Example:**
```turtle
@prefix neat: <http://purl.org/cognite/neat#> .

:EquipmentShape
    sh:property [
        sh:path neat:reverseDirectRelation ;
        sh:node :ValidMaintenanceOrder ;
    ] .
```

### Auto-load Depth

Controls how many levels of referenced instances to load:
- `0` - No auto-loading (fastest, use for time series)
- `1` - Load direct references only
- `2` - Load references and their references (recommended)
- `3+` - Multiple levels (can be slow)

### Backfill Mode

Time series validation supports backfilling historic data:

```yaml
backfill:
  enabled: true
  start_time: "30d-ago"      # Or ISO: "2024-01-01T00:00:00Z"
  end_time: "now"            # Or ISO: "2024-01-31T23:59:59Z"
  window_minutes: 60         # Chunk size (match SHACL rule window)
```

Features:
- Processes in hour-aligned chunks
- Saves cursor for resumption
- Switches to normal mode when complete

### State Management Containers

The system uses DMS containers in the `dataQuality` space to track validation state:

**FunctionValidationState** - Stores cursor state for historic/partitioned validation:
- `function_call_id`: Cognite Function call ID
- `validation_type`: Type of validation (instance, partitioned, raw_historic)
- `cursor`: Current cursor position
- `total_validated`: Total instances/rows validated so far
- `start_time`: Timestamp when validation started

**RawValidationState** - Stores incremental validation state for RAW tables:
- `db_name`: RAW database name
- `table_name`: RAW table name
- `last_processed_timestamp`: Last processed `lastUpdatedTime` (milliseconds)
- `last_run_time`: Timestamp of last validation run
- `rows_processed`: Number of rows processed in last run
- `last_validation_conforms`: Whether last validation passed
- `last_violation_count`: Number of violations in last run

**Container Creation:**
```bash
python scripts/ensure_container.py  # Creates both containers if they don't exist
```

### Dataset Organization

SHACL rule files are organized in a CDF dataset specified in `settings.yaml`:

```yaml
shacl_rules:
  dataset_external_id: "dq_shacl_rules"
```

**Important:** The dataset must exist before deployment. Create it manually in CDF or via cognite-toolkit.

## CDF Permissions Required

**Deployment (Service Principal):**
- `functionsAcl`: READ, WRITE
- `workflowsAcl`: READ, WRITE
- `filesAcl`: READ, WRITE
- `datasetsAcl`: READ, WRITE
- `streamsAcl`: CREATE, READ
- `streamRecordsAcl`: READ, WRITE

**Runtime Validation:**
- `dataModelsAcl`: READ (for data model spaces)
- `dataModelInstancesAcl`: READ (for instance spaces)
- `dataModelInstancesAcl`: WRITE (for dataQuality space - cursor storage)
- `timeSeriesAcl`: READ (for time series validation)

## Monitoring and Troubleshooting

### View Function Logs

```python
from cognite.client import CogniteClient
client = CogniteClient()

# Get recent function calls
calls = client.functions.calls.list(
    function_external_id="validate-instances-shacl",
    limit=10
)

# Get logs for a specific call
logs = client.functions.calls.get_logs(call_id=calls[0].id)
print(logs)
```

### Check Workflow Status

```python
executions = client.workflows.executions.list(
    workflow_external_id="dq-shacl-equipment",
    limit=10
)

for exe in executions:
    print(f"{exe.id}: {exe.status}")
```

### Query Validation Results

```python
# Enable alpha version for Records API
client.config.headers["cdf-version"] = "alpha"

# Query failed validations
response = client.get(
    f"/api/v1/projects/{client.config.project}/streams/test_stream_1/records",
    params={
        "filter": {
            "equals": {
                "property": ["dataQuality", "DataQualityValidationRecord/passedValidation"],
                "value": False
            }
        },
        "limit": 100
    }
)
```

## Testing

**Configuration validation:**
```bash
export CONFIG_ENV=cog-ai
python scripts/validate_config.py
```

**Unit tests:**
```bash
# Run all tests
pytest

# Run specific test files
pytest tests/test_handler_routing.py
pytest tests/test_validation.py

# Run with coverage
pytest --cov=cognite_data_quality --cov-report=html
```

**Test partition filtering:**
```bash
python scripts/test_partition_filter.py
```

**Test reverse relations:**
```bash
python scripts/test_reverse_relations.py
```

**Use notebooks for interactive testing:**
- Local validation logic: `test_and_deploy/testing_and_exploration.ipynb`
- RAW validation: `test_and_deploy/test_raw_shacl_validation.ipynb`
- Historic validation: `historic_data_validation.ipynb`
- Results analysis: `explore_validation_records.ipynb`

**Testing RAW Validation:**

The `test_raw_shacl_validation.ipynb` notebook provides comprehensive testing:
1. **Setup**: Creates test RAW table with intentional violations
2. **Template Generation**: Tests `generate_shacl_template_for_raw()`
3. **Basic Validation**: Tests `with_shacl_raw()` with simple SHACL rules
4. **Incremental Validation**: Tests timestamp-based filtering
5. **Cursor-based Validation**: Tests cursor mode for historic processing
6. **Handler Testing**: Tests `handle_raw_validation_incremental()` and `handle_raw_validation_historic()` locally
7. **State Management**: Tests cursor state save/load functions

Run this notebook before deploying RAW validation to production.

## Common Patterns

### Incrementing Rule Set Version

When updating SHACL rules, increment the `rule_set_version` in view configs:

```yaml
records:
  rule_set_id: "EquipmentSHACLv1"
  rule_set_version: "1.2"  # Increment this
```

This ensures validation records are tracked separately per version.

### Hash-based Deployment

Functions use content hashing to skip unchanged deployments. The hash is stored in function metadata. To force redeployment:

```bash
python scripts/deploy_function.py --force
```

Workflows also use hash-based deployment via workflow descriptions.

### Partition Count Selection

For historic validation, choose partition count based on data size:
- 1M instances: 10 partitions
- 10M instances: 20 partitions
- 100M instances: 50 partitions

Each partition processes independently with 10-minute timeout and 5-minute safety margin.

## Common Issues

**"Failed to load SHACL rules"**
- Ensure SHACL file exists in CDF Files
- Check file external_id matches config
- Verify dataset exists and file is in the dataset
- Validate Turtle syntax with `validate_config.py`

**"No instances to validate"**
- Check DMS filter in time series config
- Verify instances exist in specified space
- Ensure view and version match

**Workflow not triggering**
- Check trigger status in CDF Console
- Verify instance space matches trigger filter
- Ensure workflow version matches trigger version
- Check that `--skip-trigger` was not used during deployment

**RAW validation issues:**

**"RAW table not found"**
- Verify RAW database and table names in config
- Check that the RAW table exists and has rows
- Ensure service principal has read access to RAW

**"No rows updated since last run"**
- Normal for incremental validation when no changes occurred
- Check `RawValidationState` container for `last_processed_timestamp`
- Verify RAW rows have been updated since that timestamp

**"Cursor state not saving"**
- Ensure `dataQuality` space exists
- Verify service principal has write access to `dataQuality` space
- Check `FunctionValidationState` container exists (created by `ensure_container.py`)

**Function dependency errors:**

**"ModuleNotFoundError: No module named 'cognite_data_quality'"**
- Ensure `cognite-data-quality` package is published to PyPI
- Force redeploy function: `python scripts/deploy_function.py --force`

**"SHACL validation functions not available"**
- Ensure you are using the latest `cognite-data-quality` version
- Redeploy functions to CDF
