# Data Quality Validation Architecture

## Overview

The Data Quality Validation system provides comprehensive validation for industrial data in Cognite Data Fusion (CDF), covering both **data modeling instances** and **time series data**. The system supports two execution modes:

1. **Incremental/Real-time Processing**: Validates data changes as they occur using CDF Workflows
2. **Historic Processing**: Validates large datasets in parallel using Cognite Functions orchestration

All validation results are stored in the **CDF Records API** for data quality tracking, reporting, and visualization.

---

## Architecture Components

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         CDF Data Quality System                          │
└─────────────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────┐
│                          DATA SOURCES                                     │
├──────────────────────────────────────────────────────────────────────────┤
│  • CDF Data Modeling (DMS) - Instances                                   │
│  • CDF Time Series - Sensor data, measurements                           │
└──────────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    │                               │
                    ▼                               ▼
┌─────────────────────────────────┐   ┌─────────────────────────────────┐
│   INSTANCE VALIDATION (SHACL)   │   │  TIME SERIES VALIDATION         │
│                                  │   │  (SHACL + INDSL)                │
├─────────────────────────────────┤   ├─────────────────────────────────┤
│ • Real-time (Workflow-triggered)│   │ • Scheduled (CRON workflows)    │
│ • Historic (Orchestrated)        │   │ • Quality metrics via INDSL     │
│ • Auto-loads referenced nodes    │   │ • Statistical analysis          │
└─────────────────────────────────┘   └─────────────────────────────────┘
                    │                               │
                    └───────────────┬───────────────┘
                                    ▼
            ┌───────────────────────────────────────────┐
            │       CDF RECORDS API                     │
            │  (Data Quality Validation Records)        │
            ├───────────────────────────────────────────┤
            │ • Space: dataQuality                      │
            │ • Container: DataQualityValidationRecord  │
            │ • Streams: per-environment                │
            └───────────────────────────────────────────┘
                                    │
                                    ▼
            ┌───────────────────────────────────────────┐
            │    VISUALIZATION & REPORTING              │
            ├───────────────────────────────────────────┤
            │ • Dashboards (Streamlit/Power BI)        │
            │ • Alerts and notifications                │
            │ • Data quality metrics                    │
            └───────────────────────────────────────────┘
```

---

## 1. Instance Validation with SHACL

### 1.1 SHACL (Shapes Constraint Language)

**SHACL** is a W3C standard for validating RDF graphs. In our system, it validates CDF Data Modeling instances against business rules and data model constraints.

#### SHACL Rule Example

```turtle
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix ex: <http://example.org/ontology#> .

ex:MaintenanceOrderShape
    a sh:NodeShape ;
    sh:targetClass ex:MaintenanceOrder ;
    
    # Required field
    sh:property [
        sh:path ex:scheduledStartTime ;
        sh:minCount 1 ;
        sh:maxCount 1 ;
        sh:datatype xsd:dateTime ;
        sh:name "Scheduled start time" ;
        sh:message "Maintenance order must have exactly one scheduled start time" ;
    ] ;
    
    # Valid reference
    sh:property [
        sh:path ex:asset ;
        sh:class ex:Asset ;
        sh:maxCount 1 ;
        sh:message "Must reference a valid Asset" ;
    ] ;
    
    # Business rule
    sh:property [
        sh:path ex:status ;
        sh:in ("planned" "in_progress" "completed" "cancelled") ;
        sh:message "Status must be one of: planned, in_progress, completed, cancelled" ;
    ] .
```

#### Validation Process

1. **Instance Retrieval**: Fetch instances from CDF Data Modeling
2. **Auto-loading**: Automatically load referenced instances (e.g., Assets for Maintenance Orders)
   - Controlled by `auto_load_depth` parameter (default: 2)
   - Resolves `sh:class` and `sh:node` constraints
3. **RDF Conversion**: Convert instances to RDF graph
4. **SHACL Validation**: Apply SHACL rules using `pyshacl` library
5. **Result Recording**: Post validation results to Records API

### 1.2 Real-time Instance Validation

**Triggered by**: Data changes in CDF Data Modeling

**Architecture**:

```
CDF Data Modeling Instance Change
          │
          ▼
┌─────────────────────────┐
│  CDF Workflow Trigger   │
│  (DataModelingTrigger)  │
├─────────────────────────┤
│ • Batches changes       │
│ • Batch size: 10        │
│ • Timeout: 60s          │
└─────────────────────────┘
          │
          ▼
┌─────────────────────────┐
│  Validation Workflow    │
│  (dq-shacl-{view})      │
├─────────────────────────┤
│ • Calls function        │
│ • Retries: 3            │
│ • Timeout: 600s         │
└─────────────────────────┘
          │
          ▼
┌──────────────────────────────────────┐
│  validate-instances-shacl (Function) │
├──────────────────────────────────────┤
│ • Receives instances from workflow   │
│ • Loads SHACL rules from CDF Files   │
│ • Auto-loads referenced instances    │
│ • Validates with inlined SHACL engine│
│ • Posts to Records API               │
└──────────────────────────────────────┘
          │
          ▼
┌──────────────────────────┐
│   CDF Records API        │
│   (Stream per view)      │
└──────────────────────────┘
```

**Configuration** (per view in `config/environments/{env}/views/`):

```yaml
view:
  view_space: "sp_enterprise_process_industry"
  view_external_id: "MaintenanceOrder"
  view_version: "v1"
  instance_space: "your_instance_space"
  shacl_file: "maintenance_order_shacl_rules.ttl"
  rule_set_id: "MaintenanceOrderSHACL"
  rule_set_version: "1.0"
  auto_load_depth: 2

datamodel:
  space: "sp_enterprise_process_industry"
  external_id: "YourOrgMaintenanceOrder"
  version: "v1"
```

**Key Features**:
- ✅ Near real-time validation (within seconds of data change)
- ✅ Automatic batching of changes
- ✅ Auto-loading of referenced instances
- ✅ Retry logic for transient failures
- ✅ Per-view configuration

### 1.3 Historic Instance Validation

**Purpose**: Validate large existing datasets (millions of instances) efficiently and in parallel.

**Architecture**:

```
┌────────────────────────────────────────────────────────────────┐
│                  User/Scheduler Triggers                        │
└────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────────┐
│        validate-historic-orchestrator (Function)                │
├────────────────────────────────────────────────────────────────┤
│ 1. Query DMS for data distribution (1 aggregate query)         │
│    └─> Get min/max values of partition_field                   │
│    └─> Get total instance count                                │
│                                                                 │
│ 2. Calculate N partition ranges                                │
│    └─> Split date/numeric range into equal chunks              │
│    └─> Example: 10 partitions = 10 date ranges                 │
│                                                                 │
│ 3. Trigger N validation functions in parallel                  │
│    └─> Each gets a specific partition range                    │
│    └─> Non-blocking (fire and forget)                          │
└────────────────────────────────────────────────────────────────┘
                              │
                              ▼
        ┌─────────────────────┴─────────────────────┐
        │         Parallel Execution (N workers)     │
        └─────────────────────┬─────────────────────┘
                              │
        ┌─────────────────────┴─────────────────────┐
        │                     │                      │
        ▼                     ▼                      ▼
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│   Partition 0    │  │   Partition 1    │  │   Partition N    │
│   (Range 0)      │  │   (Range 1)      │  │   (Range N)      │
└──────────────────┘  └──────────────────┘  └──────────────────┘
        │                     │                      │
        ▼                     ▼                      ▼
┌───────────────────────────────────────────────────────────────┐
│    validate-instances-shacl-partitioned (Function)            │
├───────────────────────────────────────────────────────────────┤
│ 1. Load saved cursor (if resuming)                            │
│                                                                │
│ 2. Query DMS with range filter                                │
│    └─> WHERE partition_field IN [min_value, max_value)        │
│    └─> Server-side filtering (efficient!)                     │
│    └─> Fetch in chunks (default: 2000 instances)              │
│                                                                │
│ 3. For each chunk:                                             │
│    └─> Validate with SHACL                                    │
│    └─> Post results to Records API                            │
│    └─> Save cursor (for resumption)                           │
│    └─> Check timeout (5-min safety margin)                    │
│                                                                │
│ 4. On completion:                                              │
│    └─> Delete cursor state                                    │
│    └─> Return summary                                         │
│                                                                │
│ 5. On timeout:                                                 │
│    └─> Save cursor                                            │
│    └─> Exit (can be retried later to resume)                  │
└───────────────────────────────────────────────────────────────┘
        │                     │                      │
        └─────────────────────┴──────────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │  CDF Records API │
                    │  (All results)   │
                    └──────────────────┘
```

**Partitioning Strategy**:

1. **Aggregate Query** (1 DMS call):
   ```python
   # Discover data distribution
   stats = query_dms_for_stats(
       view="MaintenanceOrder",
       field="scheduledStartTime"
   )
   # Returns: {min: "2020-01-01", max: "2024-12-31", count: 1000000}
   ```

2. **Calculate Ranges** (10 partitions example):
   ```python
   # Split into 10 equal date ranges
   ranges = [
       {"min": "2020-01-01", "max": "2020-06-01"},  # Partition 0
       {"min": "2020-06-01", "max": "2020-11-01"},  # Partition 1
       {"min": "2020-11-01", "max": "2021-04-01"},  # Partition 2
       # ... 7 more partitions ...
   ]
   ```

3. **Range-filtered Queries** (N DMS calls, one per partition):
   ```python
   # Each partition queries only its range
   instances = query_dms(
       view="MaintenanceOrder",
       filter={
           "range": {
               "property": "scheduledStartTime",
               "gte": "2020-01-01",
               "lt": "2020-06-01"  # Exclusive upper bound
           }
       }
   )
   ```

**Cursor-based Resumption**:

State stored in `dataQuality.FunctionValidationState` container:

```json
{
  "space": "dataQuality",
  "externalId": "partition_state_0",
  "sources": [{
    "source": {
      "type": "container",
      "space": "dataQuality",
      "externalId": "FunctionValidationState"
    },
    "properties": {
      "partitionId": "0",
      "syncCursor": "base64-encoded-cursor-string",
      "processedCount": 15000,
      "lastUpdated": 1704067200000
    }
  }]
}
```

**Key Features**:
- ✅ **Parallel processing**: N partitions run simultaneously
- ✅ **Efficient DMS usage**: 1 aggregate query + N range-filtered queries (vs N * all-data queries)
- ✅ **Fault tolerance**: Cursor-based resumption survives function timeouts
- ✅ **Independent partitions**: One failure doesn't affect others
- ✅ **Progress tracking**: Know exactly how many instances processed
- ✅ **5-minute safety margin**: Stops at 9 minutes to avoid 10-min timeout

**Example Usage**:

```python
from cognite.client import CogniteClient

client = CogniteClient()

# Call orchestrator
result = client.functions.call(
    external_id="validate-historic-orchestrator",
    data={
        # Target function
        "validation_function_id": "validate-instances-shacl-partitioned",
        "partition_count": 10,
        "partition_field": "scheduledStartTime",
        
        # View to query
        "view_space": "sp_enterprise_process_industry",
        "view_external_id": "MaintenanceOrder",
        "view_version": "v1",
        "instance_space": "your_instance_space",
        
        # Data model for validation
        "datamodel_space": "sp_enterprise_process_industry",
        "datamodel_external_id": "YourOrgMaintenanceOrder",
        "datamodel_version": "v1",
        
        # SHACL rules
        "shacl_file_external_id": "maintenance_order_shacl_rules",
        "rule_set_id": "MaintenanceOrderSHACL",
        "rule_set_version": "1.0",
        
        # Records API
        "stream_id": "maintenance_order_stream",
    },
    wait=True
)

print(f"Triggered {result['partitions_triggered']} partitions")
print(f"Total instances: {result['distribution']['total_count']:,}")

# Monitor partition progress
for call_info in result['triggered_calls']:
    call = client.functions.calls.retrieve(call_id=call_info['call_id'])
    print(f"Partition {call_info['partition_id']}: {call.status}")
```

---

## 2. Time Series Validation with INDSL

### 2.1 INDSL (Industrial Data Science Library)

**INDSL** is Cognite's library for time series analysis, providing functions for:
- Statistical analysis (mean, std, min, max, percentiles)
- Anomaly detection
- Data quality checks (gaps, duplicates, outliers)
- Signal processing

### 2.2 Time Series Quality Validation

**Architecture**:

```
┌──────────────────────────┐
│   CRON Scheduler         │
│   (every 5 minutes)      │
└──────────────────────────┘
            │
            ▼
┌──────────────────────────┐
│  Validation Workflow     │
│  (dq-ts-shacl-{config})  │
├──────────────────────────┤
│ • Scheduled trigger      │
│ • Retries: 3             │
│ • Timeout: 1800s         │
└──────────────────────────┘
            │
            ▼
┌────────────────────────────────────┐
│  validate-timeseries-shacl         │
│  (Function)                        │
├────────────────────────────────────┤
│ 1. Load time series configuration  │
│ 2. Fetch data from CDF Time Series │
│ 3. Calculate quality metrics       │
│    (via INDSL functions)           │
│ 4. Convert metrics to RDF          │
│ 5. Validate with SHACL rules       │
│ 6. Post results to Records API     │
└────────────────────────────────────┘
            │
            ▼
┌──────────────────────────┐
│   CDF Records API        │
└──────────────────────────┘
```

**Configuration** (`config/environments/{env}/timeseries/`):

```yaml
timeseries_quality:
  timeseries:
    - external_id: "SENSOR_001"
      name: "Temperature Sensor 1"
      expected_frequency: "1m"
      
    - external_id: "SENSOR_002"
      name: "Pressure Sensor 1"
      expected_frequency: "5m"
  
  lookback_minutes: 60
  
  quality_checks:
    - check: "gap_detection"
      threshold_minutes: 10
      
    - check: "outlier_detection"
      method: "zscore"
      threshold: 3.0
      
    - check: "data_completeness"
      min_points_per_hour: 50
  
  shacl_file: "timeseries_quality_rules.ttl"
  rule_set_id: "TimeSeriesQuality"
  rule_set_version: "1.0"
```

**SHACL Rules for Time Series**:

```turtle
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix ex: <http://example.org/ontology#> .

ex:TimeSeriesQualityShape
    a sh:NodeShape ;
    sh:targetClass ex:TimeSeriesQuality ;
    
    # No gaps longer than threshold
    sh:property [
        sh:path ex:maxGapMinutes ;
        sh:maxInclusive 10 ;
        sh:message "Time series has gaps longer than 10 minutes" ;
    ] ;
    
    # Data completeness
    sh:property [
        sh:path ex:completenessPercent ;
        sh:minInclusive 95.0 ;
        sh:message "Time series completeness below 95%" ;
    ] ;
    
    # Outlier count
    sh:property [
        sh:path ex:outlierCount ;
        sh:maxInclusive 5 ;
        sh:message "Too many outliers detected" ;
    ] .
```

**Quality Metrics Calculated**:

- **Completeness**: Percentage of expected data points present
- **Gap Analysis**: Longest gap, gap count, gap distribution
- **Statistical**: Mean, std, min, max, percentiles
- **Outliers**: Z-score based outlier detection
- **Consistency**: Duplicate detection, value range checks

**Key Features**:
- ✅ Scheduled validation (CRON-based)
- ✅ Multiple time series per workflow
- ✅ INDSL integration for advanced analytics
- ✅ Configurable quality thresholds
- ✅ Historical trend analysis

---

## 3. CDF Records API Integration

### 3.1 Record Structure

All validation results are stored in the **CDF Records API** with a consistent schema:

```json
{
  "space": "dataQuality",
  "externalId": "dq_MaintenanceOrderSHACL_order_12345_20240101",
  "sources": [{
    "source": {
      "type": "container",
      "space": "dataQuality",
      "externalId": "DataQualityValidationRecord"
    },
    "properties": {
      "ruleSetId": "MaintenanceOrderSHACL",
      "ruleSetVersion": "1.0",
      "jobRunId": "validation_1704067200",
      "passedValidation": false,
      "resultSeverity": ["Violation", "Warning"],
      "failedConstraints": [
        "sh:MinCountConstraintComponent::scheduledStartTime",
        "sh:ClassConstraintComponent::asset"
      ],
      "focusNode": "http://purl.org/cognite/instance_space/order_12345",
      "focusNodeInstance": {
        "space": "instance_space",
        "externalId": "order_12345"
      },
      "sourceShape": ["http://example.org/ontology#MaintenanceOrderShape"],
      "dataDomainExternalId": "maintenance",
      "validationReport": {
        "violationCount": 2,
        "violations": [
          {
            "sourceConstraintComponent": "sh:MinCountConstraintComponent",
            "resultMessage": "Maintenance order must have exactly one scheduled start time",
            "resultSeverity": "Violation",
            "resultPath": "scheduledStartTime",
            "sourceConstraint": "1"
          },
          {
            "sourceConstraintComponent": "sh:ClassConstraintComponent",
            "resultMessage": "Must reference a valid Asset",
            "resultSeverity": "Warning",
            "resultPath": "asset"
          }
        ],
        "summary": "2 violation(s)"
      }
    }
  }]
}
```

### 3.2 Record Properties

| Property | Type | Description |
|----------|------|-------------|
| `ruleSetId` | string | Identifier for the validation rule set |
| `ruleSetVersion` | string | Version of the rule set |
| `jobRunId` | string | Unique ID for the validation job |
| `passedValidation` | boolean | True if no violations, false otherwise |
| `resultSeverity` | list[string] | Severity levels: "Violation", "Warning", "Info" |
| `failedConstraints` | list[string] | List of constraint identifiers that failed |
| `focusNode` | string | URI of the instance that was validated |
| `focusNodeInstance` | object | CDF instance reference (space + externalId) |
| `sourceShape` | list[string] | SHACL shapes that were applied |
| `dataDomainExternalId` | string | Optional domain classifier |
| `validationReport` | object | Detailed violation information |

### 3.3 Querying Records

```python
from cognite.client import CogniteClient

client = CogniteClient()

# Enable alpha version for Records API
client.config.headers["cdf-version"] = "alpha"

# Query failed validations
response = client.get(
    f"/api/v1/projects/{client.config.project}/streams/maintenance_order_stream/records",
    params={
        "filter": {
            "equals": {
                "property": ["dataQuality", "DataQualityValidationRecord/passedValidation"],
                "value": False
            }
        },
        "limit": 100
    }
)

records = response.json()
print(f"Found {len(records)} failed validations")
```

---

## 4. Deployment Architecture

### 4.1 Multi-Environment Setup

The system supports multiple environments (e.g., `cog-ai`, `cog-sail`) with isolated configurations:

```
config/environments/
├── cog-ai/
│   ├── settings.yaml              # Environment settings
│   ├── views/
│   │   ├── equipment.yaml
│   │   ├── maintenance_order.yaml
│   │   └── operation.yaml
│   ├── shacl_rules/
│   │   ├── equipment_shacl_rules.ttl
│   │   ├── maintenance_order_shacl_rules.ttl
│   │   └── operation_shacl_rules.ttl
│   └── timeseries/
│       └── sensor_quality.yaml
│
└── cog-sail/
    ├── settings.yaml
    ├── views/
    │   ├── largeboat.yaml
    │   ├── smallboat.yaml
    │   └── nmeatimeseries.yaml
    └── shacl_rules/
        ├── largeboat_shacl_rules.ttl
        └── nmeatimeseries_shacl_rules.ttl
```

**Environment Settings** (`settings.yaml`):

Each environment configures a dataset for organizing SHACL rule files:

```yaml
shacl_rules:
  source_dir: "config/environments/cog-ai/shacl_rules"
  mime_type: "text/turtle"
  dataset_external_id: "dq_shacl_rules"  # Dataset for file organization
```

**Note**: The dataset must exist before deployment. Create it manually in CDF or via Toolkit:

```bash
# Using cognite-toolkit or CDF Console
# Create dataset with external_id: "dq_shacl_rules"
```

This provides:
- ✅ **File Organization**: All SHACL rules grouped under one dataset
- ✅ **Access Control**: Dataset-level permissions for governance
- ✅ **Metadata**: Easy filtering and discovery of validation rules
- ✅ **Lineage**: Track which files belong to data quality validation

### 4.2 Cognite Functions

All functions are deployed from the `cognite_data_quality/_function_code/` directory:

| Function | External ID | Purpose | Runtime | Handler |
|----------|-------------|---------|---------|---------|
| Unified Validation | `data-quality-validation` | Unified function for all validation types | 10 min | `handler.py` with type dispatch |
| Instance Validation | - | Real-time validation (workflow-triggered) | 10 min | `handlers/instance_validation.py` |
| Time Series Validation | - | Scheduled time series quality checks | 30 min | `handlers/timeseries_validation.py` |
| Historic Orchestrator | - | Coordinate parallel historic validation | 10 min | `handlers/orchestrator.py` |
| Partitioned Validation | - | Validate a partition of historic data | 10 min | `handlers/partitioned_validation.py` |

**Note**: The unified function (`data-quality-validation`) routes to different handlers based on the `validation_type` parameter in the function data payload.

### 4.3 CDF Workflows

**Instance Validation Workflows** (per view):
- External ID: `dq-shacl-{view_external_id}`
- Trigger: Data changes in the view
- Function: `validate-instances-shacl`

**Time Series Validation Workflows** (per config):
- External ID: `dq-ts-shacl-{config_name}`
- Trigger: CRON schedule (e.g., `*/5 * * * *`)
- Function: `validate-timeseries-shacl`

### 4.4 CI/CD Pipeline

GitHub Actions workflow (`.github/workflows/deploy.yml`):

```yaml
jobs:
  deploy:
    strategy:
      matrix:
        environment:
          - name: cog-ai
            enabled: true
          - name: cog-sail
            enabled: true

    steps:
      1. Validate configuration
      2. Run unified deployment script: scripts/deploy_infrastructure.py
         - Ensures Records Container
         - Deploys unified validation function
         - Deploys instance validation workflows (uploads SHACL files)
         - Deploys time series validation workflows (uploads SHACL files)
         - Optionally deploys data models
```

**Deployment Triggers**:
- Push to `main` branch
- Manual workflow dispatch with options:
  - `--dry-run`: Preview changes without deploying
  - `--force`: Force redeployment even if no changes
  - `--views`: Deploy specific views only
  - `--skip-trigger`: Skip workflow trigger creation

**Hash-based Deployment**:
- Functions only redeploy if code changes (content hash comparison)
- Workflows redeploy if configuration changes
- SHACL files redeploy if content changes
- Function secrets are managed separately

---

## 5. Key Technologies

### 5.1 Core Libraries

- **cognite_data_quality._neat**: Inlined SHACL validation engine (previously thisisneat)
  - SHACL validation with CDF SPARQL functions
  - Auto-loading referenced instances (including reverse relations)
  - RDF graph conversion
  - Incremental sync with cursor management
  - CDF SDK and INDSL function wrappers for time series validation

- **pyshacl**: Python SHACL validator
  - W3C SHACL specification compliance
  - Detailed validation reports

- **rdflib**: RDF graph processing
  - Turtle/RDF parsing
  - SPARQL queries
  - Graph manipulation

- **INDSL 8.7.0+**: Industrial Data Science Library
  - Time series analytics
  - Statistical functions
  - Anomaly detection (extreme outliers, gaps, data quality)

- **cognite-sdk 7.83.0+**: Official Cognite Python SDK
  - CDF API access
  - Data Modeling, Functions, Workflows
  - Records API (alpha)

### 5.2 CDF Services

- **Data Modeling (DMS)**: Graph-based data modeling
- **Functions**: Serverless compute
- **Workflows**: Orchestration and scheduling
- **Records API**: Structured validation results
- **Files API**: SHACL rule storage (organized in datasets)
- **Data Sets**: Organize and govern files
- **Time Series API**: Sensor data storage

---

## 6. Data Flow Diagrams

### 6.1 Real-time Instance Validation Flow

```
┌─────────────────────┐
│  User/System        │
│  Creates/Updates    │
│  Instance           │
└─────────────────────┘
          │
          ▼
┌─────────────────────┐
│  CDF Data Modeling  │
│  (DMS)              │
│  Instance stored    │
└─────────────────────┘
          │
          ▼ (triggers within seconds)
┌─────────────────────┐
│  Workflow Trigger   │
│  Batches changes    │
│  (10 instances or   │
│   60 seconds)       │
└─────────────────────┘
          │
          ▼
┌─────────────────────┐
│  Validation         │
│  Workflow           │
│  (dq-shacl-view)    │
└─────────────────────┘
          │
          ▼
┌──────────────────────────────┐
│  Validation Function         │
├──────────────────────────────┤
│ 1. Receive instances         │
│ 2. Load SHACL rules          │
│ 3. Auto-load references      │
│ 4. Validate                  │
│ 5. Post to Records API       │
└──────────────────────────────┘
          │
          ▼
┌─────────────────────┐
│  CDF Records API    │
│  Validation result  │
│  stored             │
└─────────────────────┘
          │
          ▼
┌─────────────────────┐
│  Dashboard/Alert    │
│  User notified of   │
│  data quality issue │
└─────────────────────┘
```

### 6.2 Historic Validation Flow

```
┌─────────────────────┐
│  User/Scheduler     │
│  Initiates historic │
│  validation         │
└─────────────────────┘
          │
          ▼
┌───────────────────────────────┐
│  Orchestrator Function        │
├───────────────────────────────┤
│ • Query DMS stats (1 call)    │
│ • Calculate N partition ranges│
│ • Trigger N validation funcs  │
└───────────────────────────────┘
          │
          ▼
    ┌─────┴──────┐
    │  Parallel  │
    └─────┬──────┘
          │
    ┌─────┴─────┬─────────────┬──────────────┐
    │           │             │              │
    ▼           ▼             ▼              ▼
┌────────┐  ┌────────┐    ┌────────┐    ┌────────┐
│Part 0  │  │Part 1  │    │Part 2  │... │Part N  │
├────────┤  ├────────┤    ├────────┤    ├────────┤
│Range:  │  │Range:  │    │Range:  │    │Range:  │
│[min,   │  │[r1,    │    │[r2,    │    │[rN,    │
│ r1)    │  │ r2)    │    │ r3)    │    │ max)   │
└────────┘  └────────┘    └────────┘    └────────┘
    │           │             │              │
    └─────┬─────┴─────────────┴──────────────┘
          │
          ▼
┌────────────────────────────────┐
│  Partitioned Validation        │
│  Function (per partition)      │
├────────────────────────────────┤
│ While not done:                │
│   • Load cursor (if resuming)  │
│   • Query 2000 instances       │
│   • Validate chunk             │
│   • Post to Records API        │
│   • Save cursor                │
│   • Check timeout              │
│ Delete cursor on completion    │
└────────────────────────────────┘
          │
          ▼
┌─────────────────────┐
│  CDF Records API    │
│  All validation     │
│  results stored     │
└─────────────────────┘
```

### 6.3 Timestamp Collection Flow (equal-count partitioning)

Before triggering partitioned validation, the orchestrator needs to collect all
`lastUpdatedTime` timestamps from DMS to compute equal-count partition boundaries.
This can take longer than the 10-minute function timeout, so it is split across
multiple function invocations using a resumable state machine.

```
START: handle_orchestrator (no orchestration_id → initial mode)
│
├─ Load existing state: load_timestamp_collection_state()
│    ├─ No state                           → fresh start
│    ├─ phase="collecting", age < 10 min   → EXIT: "already_running"
│    ├─ phase="collecting", age ≥ 10 min   → resume from CogniteFile (stale/crashed)
│    └─ phase="checkpoint"                 → resume in-process (monitor re-triggered)
│
├─ Create function schedule (if not resuming)
│    └─ Fires every 2 min → monitor mode; auto-retries on timeout
│
├─ Save initial "collecting" state to DMS
│
├─ ┌──────────────────────────────────────────────────────────────────┐
│  │  SYNC LOOP (metadata-only — lastUpdatedTime per node)            │
│  │                                                                  │
│  │  while True:                                                     │
│  │    ├─ elapsed > 480s?                                            │
│  │    │    └─ save_timestamp_collection_state(phase="checkpoint")   │
│  │    │         CogniteFile: {timestamps: [...], cursor: "..."}     │
│  │    │         DMS:         phase="checkpoint"                     │
│  │    │         → RETURN                                            │
│  │    │         monitor fires ≤2 min later, resumes in-process      │
│  │    │                                                             │
│  │    ├─ DMS sync query (metadata only):                            │
│  │    │    filter:  space=X, HasData(view)                          │
│  │    │    limit:   2000 → 1000 → 500 → 200 → 100 → 10             │
│  │    │    cursor:  current_cursor (pagination)                     │
│  │    │                                                             │
│  │    ├─ Timeout / transient error retry:                           │
│  │    │    attempt 1: wait 30s, retry same limit                    │
│  │    │    attempt 2: step down limit, wait 30s                     │
│  │    │    all limits exhausted: save state, RETURN                 │
│  │    │                                                             │
│  │    ├─ Collect: all_timestamps += node.lastUpdatedTime            │
│  │    │                                                             │
│  │    ├─ Every 120s: save_timestamp_collection_state()              │
│  │    │    CogniteFile: {timestamps: [...], cursor: "..."}          │
│  │    │    DMS:         phase="collecting", count=N                 │
│  │    │                                                             │
│  │    └─ chunk_size == 0? → BREAK (all pages consumed)             │
│  └──────────────────────────────────────────────────────────────────┘
│
├─ Sort timestamps → mark_timestamp_collection_complete()
│    DMS: phase="complete" | DELETE CogniteFile
│
└─ Calculate equal-count partition boundaries → trigger partitioned validation


State storage
─────────────
DMS OrchestrationState node:
  timestamp_collection_phase   "collecting" | "checkpoint" | "complete"
  timestamp_collection_count   N (visibility / progress reporting)
  last_updated                 datetime (staleness detection — ISO string from DMS)
  timestamps_file              DirectRelation → CogniteFile node

CogniteFile content (JSON):
  { "timestamps": [ms, ms, ...], "cursor": "opaque_string" | null }


Controlled timeout (phase="checkpoint"):
  CogniteFile kept; monitor detects checkpoint; updates phase → "collecting";
  calls handle_orchestrator in-process via _resume_orchestration_id;
  collection resumes from saved cursor without a new function call.

Crash / unexpected kill (phase stays "collecting"):
  last_updated stops advancing; monitor detects age > 10 min (stale);
  deletes state + CogniteFile; triggers a fresh initial run.
```

### 6.4 Partitioned Validation Flow

```
PHASE 1 — TRIGGER (orchestrator initial mode, after timestamp collection)

  Compute equal-count partition ranges from sorted timestamps.

  For each partition — fire in parallel (wait=False):
    config: { orchestration_id, job_run_id, view, SHACL file,
              partition_id: i,
              partition_range: { min_value: ts_start, max_value: ts_end } }
    → trigger_validation_function()

  Save to DMS OrchestrationState:
    partition_call_ids:    { partition_id → call_id }
    partition_retry_count: { partition_id → 0 }
    status: "monitoring"


PHASE 2 — PARTITION EXECUTION (handle_partitioned_validation, parallel)

  Load cursor: load_cursor(orchestration_id, partition_id)
    DMS node "partition_state_orch_{id}_p{n}":
      syncCursor, processedCount (cumulative across resumes)

  Build DMS filter:
    And(HasData(view), space=instance_space,
        Range(lastUpdatedTime, [min_value, max_value)))
    Note: upper bound is exclusive — prevents overlap between adjacent partitions

  ┌──────────────────────────────────────────────────────────────────┐
  │  CHUNK LOOP                                                      │
  │                                                                  │
  │  while True:                                                     │
  │    ├─ elapsed > 480s?                                            │
  │    │    └─ save_cursor(cumulative) → RETURN status="partial"    │
  │    │         monitor retriggers partition; cursor loaded on resume│
  │    │                                                             │
  │    ├─ DMS sync query (cursor-based, limit=chunk_size):           │
  │    │    no data → BREAK (partition complete)                     │
  │    │                                                             │
  │    ├─ SHACL validate chunk via NEAT                              │
  │    │                                                             │
  │    ├─ Post to Records API                                        │
  │    │    job_run_id = orchestration_id (shared across all         │
  │    │    partitions → single queryable result stream)             │
  │    │    failure_rate > 10% → RAISE (fail this partition)        │
  │    │                                                             │
  │    └─ save_cursor(cumulative) — fail partition if save fails     │
  └──────────────────────────────────────────────────────────────────┘

  Completion:   delete_cursor() → return status="completed"
  Timeout/err:  save_cursor()   → return status="partial" | "error"


PHASE 3 — MONITOR (handle_orchestrator, every 2 min via schedule)

  Load OrchestrationState from DMS.
    Not found → orchestration already finalized, delete schedule, RETURN.

  For each partition — retrieve call → inspect result:

    Call status      Response status   Action
    ──────────────   ───────────────   ────────────────────────────────────────
    Completed        "completed"       Aggregate metrics → mark completed
    Completed        "partial"         Retry (controlled timeout in function)
    Completed        "error"           Retry up to max_retries (3)
    Failed / Timeout —                 Retry up to max_retries (3)
    Running          —                 Leave in progress, check next tick

  Retry limits:
    "partial" → max_partial_retries = 100   (many function timeouts expected)
    "error"   → max_retries         = 3     (real errors should be rare)

  Update DMS OrchestrationState → save.

  ├─ Still in progress → wait for next schedule tick (2 min)
  ├─ All completed     → delete schedule + workflow, status="completed"
  └─ Some exceeded max retries → delete schedule + workflow, status="failed"


Retry / failure matrix:

  Scenario               Cursor saved?      Max retries
  ─────────────────────  ─────────────────  ───────────
  Timeout (> 480s)       Yes (cumulative)   100
  Error (exception)      Yes (if in loop)   3
  Call crash / kill      No                 3
```

---

## 7. Performance Characteristics

### 7.1 Real-time Validation

- **Latency**: 5-60 seconds from data change to validation
- **Throughput**: ~100 instances per validation (batched)
- **Scalability**: Horizontal (one workflow per view)

### 7.2 Historic Validation

For 1,000,000 instances with 10 partitions:

- **DMS Queries**: 11 (1 aggregate + 10 range-filtered)
- **Parallel Functions**: 10 (running simultaneously)
- **Chunks per Partition**: 50 (2000 instances per chunk)
- **Total Time**: ~15-30 minutes (depends on SHACL complexity)
- **Fault Tolerance**: If 1 partition fails, 9 continue
- **Resumption**: Failed partition can resume from cursor

**DMS Efficiency**:
- 1 aggregate query to discover data distribution
- N range-filtered queries (one per partition with server-side filtering)
- Minimal data transfer overhead (only querying relevant ranges)

---

## 8. Security and Permissions

### 8.1 Required CDF Capabilities

**For Deployment** (Service Principal):
- `functionsAcl`: READ, WRITE
- `workflowsAcl`: READ, WRITE
- `filesAcl`: READ, WRITE (for SHACL rules)
- `datasetsAcl`: READ, WRITE (for organizing SHACL files)
- `streamsAcl`: CREATE, READ
- `streamRecordsAcl`: READ, WRITE

**For Runtime Validation**:
- `dataModelsAcl`: READ (for data model spaces)
- `dataModelInstancesAcl`: READ (for instance spaces)
- `dataModelInstancesAcl`: WRITE (for `dataQuality` space - cursor storage)
- `timeSeriesAcl`: READ (for time series validation)

### 8.2 Data Isolation

- Each environment has isolated configurations
- Records stored in separate streams per environment
- Function deployments are environment-specific

---

## 9. Monitoring and Observability

### 9.1 Function Logs

All functions log to CDF Functions logs:

```python
print("Validated 2000 instances, found 15 violations")
```

Access via:
- CDF Console: Functions → Function Name → Calls → Logs
- SDK: `client.functions.calls.get_logs(call_id=...)`

### 9.2 Workflow Status

Monitor workflow executions:

```python
executions = client.workflows.executions.list(
    workflow_external_id="dq-shacl-MaintenanceOrder",
    limit=10
)

for exe in executions:
    print(f"{exe.id}: {exe.status}")
```

### 9.3 Validation Metrics

Query Records API for data quality metrics:

```python
# Total validations today
total = count_records(
    stream_id="maintenance_order_stream",
    date_filter=today()
)

# Failed validations
failed = count_records(
    stream_id="maintenance_order_stream",
    date_filter=today(),
    passed=False
)

quality_score = (total - failed) / total * 100
print(f"Data quality score: {quality_score:.1f}%")
```

---

## 10. Best Practices

### 10.1 SHACL Rule Design

✅ **Do**:
- Use clear, descriptive messages
- Set appropriate severity levels
- Group related constraints in one shape
- Use `sh:name` for human-readable labels
- Test rules on sample data before deployment

❌ **Don't**:
- Create overly complex shapes (impacts performance)
- Use hardcoded values that change frequently
- Forget to handle optional fields (use `sh:minCount 0`)

### 10.2 Performance Optimization

- **Batch Size**: 10 instances for real-time, 2000 for historic
- **Auto-load Depth**: Keep at 2 (3+ can be slow)
- **Partition Count**: 10-20 for historic (balance parallelism vs overhead)
- **Chunk Size**: 2000 for historic validation

### 10.3 Error Handling

- **Retries**: Configure workflow retries (default: 3)
- **Timeouts**: Use cursor-based resumption for long jobs
- **Monitoring**: Set up alerts for repeated failures

---

## 11. Future Enhancements

### 11.1 Planned Features

- **Custom validation functions**: Python-based validators alongside SHACL
- **Real-time dashboards**: Live data quality metrics
- **Automated remediation**: Trigger corrections for common issues
- **ML-based anomaly detection**: Learn normal patterns from time series
- **Cross-instance validation**: Validate relationships across multiple instances

### 11.2 Scalability Improvements

- **Incremental SHACL**: Only validate changed properties
- **Cached reference loading**: Reduce DMS calls for frequently referenced instances
- **Distributed validation**: Validate partitions on different Cognite Functions regions

---

## Appendix A: Glossary

| Term | Definition |
|------|------------|
| **CDF** | Cognite Data Fusion - Industrial data platform |
| **DMS** | Data Modeling Service - Graph-based data modeling in CDF |
| **SHACL** | Shapes Constraint Language - W3C standard for RDF validation |
| **INDSL** | Industrial Data Science Library - Cognite's time series analytics library |
| **RDF** | Resource Description Framework - Graph data model |
| **Cursor** | Pagination token for incremental data fetching |
| **Partition** | Subset of data based on range (e.g., date range) |
| **Auto-loading** | Automatic fetching of referenced instances during validation |

---

## Appendix B: References

- [SHACL Specification](https://www.w3.org/TR/shacl/)
- [CDF Documentation](https://docs.cognite.com/)
- [INDSL Documentation](https://indsl.docs.cognite.com/)
- [SHACL validation code](cognite_data_quality/_neat/) (inlined, previously thisisneat)
- [Cognite Python SDK](https://cognite-sdk-python.readthedocs.io/)

---

**Document Version**: 1.0  
**Last Updated**: 2026-02-02  
**Maintained By**: Cognite Data Quality Team
