# cf-pipeline-engine

The Cogniflow pipeline engine implemented in C++ (compiler, scheduler, runtime).

This folder is structured as a Python package for consistency with the other
`cf_*` components, even though the core implementation is native C++ and built
via CMake.

## Build (CMake)

```bash
cmake -S . -B build
cmake --build build
```

## Python package wrapper

The Python package is intentionally thin and provides access to the packaged
native engine assets:

- `cf_pipeline_engine.cf_pipeline_v2_path()`
- `cf_pipeline_engine.cf_siggen_path()`
- `cf_pipeline_engine.cf_engine_include_path()`
- `cf_pipeline_engine.cf_type_registry_path()`
- `cf_pipeline_engine.resolve_cf_pipeline_v2_executable()`

Published distribution name:

```bash
pip install cf-pipeline-engine
```

The published wheel installs:

- `bin/cf_pipeline_v2(.exe)`
- `bin/cf_siggen(.exe)`
- `bin/type_registry.v0.json`
- `include/*.h`

## Python CLI

The legacy standalone Python wrapper package has been retired. Its CLI surface
now lives in this `cf_pipeline_engine` package.

Direct invocation remains policy-gated by `CF_ALLOW_DIRECT_ENGINE`:

```bash
CF_ALLOW_DIRECT_ENGINE=1 python -m cf_pipeline_engine.cli \
  --pipeline sandcastle/cf_pipeline/cf_pipeline_engine/examples/opcua_fifo_avg_to_duckdb_parquet_triggered.jsonld \
  --interval 1 \
  --duration 10
```

Equivalent module entrypoint:

```bash
CF_ALLOW_DIRECT_ENGINE=1 python -m cf_pipeline_engine --help
```

## Publishing

`cf_pipeline_engine` is published with the dedicated Windows workflow:

- Workflow: `.github/workflows/cf_pipeline_engine_windows_publish.yml`
- Package directory: `sandcastle/cf_pipeline/cf_pipeline_engine`
- PyPI tag: `cf-pipeline-engine-v<version>`
- TestPyPI tag: `cf-pipeline-engine-v<version>-test`

Local preflight:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/mimic_windows_python_publish_workflow.ps1 `
  -WorkflowFile .github/workflows/cf_pipeline_engine_windows_publish.yml `
  -PackageDir sandcastle/cf_pipeline/cf_pipeline_engine `
  -PythonExe py `
  -PythonVersion 3.13
```

Queue a dry-run dispatch:

```powershell
powershell -ExecutionPolicy Bypass -File scripts/queue_windows_python_publish_workflow.ps1 `
  -WorkflowFile .github/workflows/cf_pipeline_engine_windows_publish.yml `
  -PackageDir sandcastle/cf_pipeline/cf_pipeline_engine `
  -PublishTarget testpypi `
  -Ref main `
  -RequireLocalPass `
  -DryRun
```

## OPC UA Demo Pipeline Sink

The existing demo pipeline
`examples/opcua_fifo_avg_to_duckdb_parquet_triggered.jsonld` now uses
`cfsink:DataHiveParquetSinkStep` from `cf_basic_sinks`.

The sink writes through the C++ gatekeeper library `cf_datahive_cpp`,
producing one committed data hive run with 20 rows (`cycle_id` 0..19) and no
`archive.jsonl`.

Run the one-click demo:

```powershell
.\scripts\fresh_install.ps1 -Clean -RunDemo
```

Expected output layout after one demo session:

- `workspace/<data_hive>/opcua_fifo_avg/latest.txt`
- `workspace/<data_hive>/opcua_fifo_avg/runs/<run_id>/manifest.json`
- `workspace/<data_hive>/opcua_fifo_avg/runs/<run_id>/tables/measurements/part-*.parquet`
