# Cognite Data Quality

[![Documentation Status](https://readthedocs.com/projects/cognite-data-quality/badge/?version=latest)](https://cognite-data-quality.readthedocs-hosted.com/en/latest/)
[![PyPI](https://img.shields.io/pypi/v/cognite-data-quality)](https://pypi.org/project/cognite-data-quality/)

`cognite-data-quality` is a Python package for **SHACL-based data quality validation** in Cognite Data Fusion (CDF). It validates **DMS instances** (structured data models) and **time series data** against declarative rules, stores results in the CDF Records API, and provides tools to deploy and run validation in CDF Functions and Workflows.

---

## Purpose

Industrial data in CDF must meet quality standards before it can be trusted for analytics, dashboards, or automation. Raw data often has gaps, outliers, missing required fields, or rule violations. Manual checks do not scale.

This package provides:

- **Declarative validation** using SHACL (W3C standard). You define *what* good data looks like; the package checks *whether* your data conforms.
- **Automated validation** in CDF (Functions, Workflows, Triggers) and locally in notebooks and scripts.
- **Traceable results** in the Records API for monitoring, dashboards, and drilling into failures.

---

## Ways to Test

1. **Unit tests** (no CDF): Run `uv run pytest` from the repo root.
2. **Interactive validation** (requires CDF): Use the environment notebook (e.g. `config/environments/cog-sail/test_cogsail_cognite_data_quality.ipynb`) to run SHACL validation against live data, deploy infrastructure, and run the validation pipeline.

---

## How It Works

1. **Define rules** in TTL (Turtle), JSON rule set, or YAML view config. Rules target views or time series.
2. **Run validation** locally with `run_validation()` or in CDF via deployed Functions triggered by Workflows.
3. **Post results** to the Records API (pass/fail, violation details, severity).

The package includes inlined SHACL validation with CDF SPARQL functions and integrates with CDF Data Modeling (DMS). It supports both CDF SDK and INDSL functions for time series quality checks.

---

## Workflows

### Development and testing

1. Create a TOML file (e.g. `cog-sail.toml`) with CDF credentials.
2. Write SHACL rules (TTL). Use `run_validation()` against live data.
3. Iterate on rules; optionally set `post_to_records=True` with a `RecordsConfig` to test the full pipeline.
4. Use `DataModelConfig` and `RecordsConfig` for datamodel, instance space, stream, and rule set.

### Production deployment

1. Deploy infrastructure with `deploy_validation_infrastructure(client, settings_path=..., views_dir=..., function_secrets=...)`. This ensures containers (Records, OrchestrationState, FunctionValidationState), deploys the unified validation function, workflows, and triggers.
2. Optionally run the validation pipeline for a view with `deploy_validation_pipeline(client, settings_path=..., view_external_id=..., wait=True)` to process historic data and set up sync and monitor schedules.
3. Results are written to the Records API automatically.

### Invoking from Python

Use the invoke helpers to call deployed functions from notebooks or scripts:

```python
from cognite_data_quality import call_validate_instances_shacl

result = call_validate_instances_shacl(client, data)
```

---

## Main Capabilities

| Capability | Description |
| --- | --- |
| **run_validation** | Run validation from DMS (no workflows). Supports TTL, JSON, or YAML. Use `datamodel`, `instance_space`, `records_config`, `limit`, `print_output`. |
| **deploy_validation_infrastructure** | Deploy all validation infrastructure (containers, function, workflows, triggers) from `settings_path` and `views_dir`. |
| **deploy_validation_pipeline** | Deploy and run the full validation pipeline for a view (historic partitions, sync trigger, monitor schedule). |
| **call_validation** | Invoke the unified function with type-based dispatch (instance, instance_sync_cursor, timeseries, orchestrator, partitioned, test). |
| **invoke helpers** | `call_validate_instances_shacl`, `call_validate_timeseries_shacl`, etc. |

---

## Other Features

- **Credentials:** Load from TOML (e.g. `cog-sail.toml`) with `load_cognite_client_from_toml()`, same pattern as pygen.
- **Rule sources:** TTL, JSON rule set, or YAML view config.
- **Records output:** Default container `dataQuality:DataQualityValidationRecord`; configurable stream and rule set in `RecordsConfig`.
- **Auto-loading:** References between instances are loaded automatically when validating DMS data (configurable depth).

---

## Next Steps

- **[Installation](installation.md):** Set up the package and credentials.
- **[Where to Start](quickstart/where_to_start.md):** Run your first validation and deploy.
- **[Usage](usage/run_validation.md):** Run validation, deploy, invoke, and configure rules.
