# Pipeline Backend & Plugin Specification
## (Mixed C++ and Python Operations)

## 1. Overview

This specification defines a **modular pipeline execution engine** with:

- a **C++ core backend** (pipeline engine, data types, C++ operations),
- a **Python configuration layer** (defining pipelines, loading plugins),
- **operations** (steps) that can be:
  - high-performance **C++ functions** (built-in or from C++ plugins),
  - **Python callables** (Python-only plugins).

The key goals:

- Pipelines are **defined in Python**.
- Execution path stays **fully in C++** for C++ ops.
- Python-only operations are also possible and **registered into the same engine**.
- All operations (C++ and Python) are **discoverable and automatically validatable**.

---

## 2. Core Concepts

### 2.1 Data

All operations act on a shared C++ data type:

```cpp
namespace pipeline {

struct Data {
    // concrete implementation (e.g. spectra, arrays, etc.)
};

} // namespace pipeline
```

Properties:

* `Data` is defined in backend headers.
* All plugins must use the same `Data` ABI (same struct layout, compiler settings).
* Every operation receives a `Data&` and may modify it in place.

### 2.2 Operation Types

There are two operation kinds:

1. **C++ operation**

   * C++ function of type: `using OpFn = void(*)(Data&);`
   * Registered via a C API exposed by the backend (PyCapsule).
   * Executed entirely in C++.

2. **Python operation**

   * Python callable with a single parameter: `fn(data)`.
   * Registered via a Python API: `backend.register_py_op(name, fn)`.
   * When executed, C++ enters Python (GIL), calls the function.

### 2.3 Operation Registry

The backend maintains a single **registry** that holds all operations, regardless of implementation:

* Key: operation name (`std::string`)
* Value: operation entry with:

  * `impl`: `"cpp"` or `"python"`
  * C++ function pointer (for C++ ops) OR Python callable (for Python ops).

Conceptual C++ structure:

```cpp
namespace pipeline {

enum class ImplKind { Cpp, Python };

struct OpEntry {
    ImplKind impl;
    OpFn cpp_fn;             // valid if impl == Cpp
    pybind11::object py_fn;  // valid if impl == Python
};

void register_cpp_op(const std::string& name, OpFn fn);
void register_py_op(const std::string& name, pybind11::object fn);
const OpEntry& get_op(const std::string& name);
std::vector<std::string> list_op_names();
std::vector<OpEntry> list_op_entries(); // or Python-exposed 'describe_ops()'

} // namespace pipeline
```

### 2.4 Pipeline

A **pipeline** is an ordered list of operation names:

```cpp
namespace pipeline {

class Pipeline {
public:
    void add_step(const std::string& name);
    void run(Data& d) const;
};

} // namespace pipeline
```

Execution:

```cpp
void Pipeline::run(Data& d) const {
    for (auto const& name : steps_) {
        auto const& entry = get_op(name);
        if (entry.impl == ImplKind::Cpp) {
            entry.cpp_fn(d); // pure C++
        } else {
            pybind11::gil_scoped_acquire gil;
            entry.py_fn(d);  // jump to Python
        }
    }
}
```

Python usage:

```python
from pipeline_backend import Pipeline

p = Pipeline()
p.add_step("normalize_cpp")
p.add_step("smooth_py")
p.run(data)
```

---

## 3. Backend Requirements

The **backend** is the Python extension module implementing the engine.

### 3.1 Required C++ Components

Backend C++ must define:

```cpp
namespace pipeline {

struct Data { /* ... */ };

using OpFn = void(*)(Data&);

enum class ImplKind { Cpp, Python };

struct OpEntry {
    ImplKind impl;
    OpFn cpp_fn;
    pybind11::object py_fn;
};

void register_cpp_op(const std::string& name, OpFn fn);
void register_py_op(const std::string& name, pybind11::object fn);

const OpEntry& get_op(const std::string& name);
std::vector<std::string> list_op_names();
// C++ side function exposed to Python as 'describe_ops'
std::vector<OpEntry> list_op_entries();

class Pipeline {
public:
    void add_step(const std::string& name);
    void run(Data& d) const;
};

} // namespace pipeline
```

### 3.2 C API for C++ Plugin Registration (PyCapsule)

The backend exposes a **C API** for C++ plugins to register operations at import time.

C function pointer type:

```cpp
extern "C" {
    typedef void (*register_op_c_api_t)(const char* name, pipeline::OpFn fn);
}
```

Backend must:

1. Implement:

   ```cpp
   static void register_op_c_api(const char* name, pipeline::OpFn fn) {
       pipeline::register_cpp_op(name, fn);
   }
   ```

2. Expose it as a PyCapsule in the module init:

   ```cpp
   PYBIND11_MODULE(pipeline_backend, m) {
       // bind Pipeline, Data, list_ops, describe_ops, register_py_op, ...

       m.attr("_register_op_capsule") = py::capsule(
           (void*) &register_op_c_api,
           "pipeline_backend.register_op_v1"
       );
   }
   ```

The capsule name **must match** the string in the plugin and validator:
`"pipeline_backend.register_op_v1"`.

### 3.3 Python API Requirements

On the Python side, the backend must provide:

* `__backend_metadata__` : `dict`
* `list_ops() -> List[str]`
* `describe_ops() -> List[dict]`
* `register_py_op(name: str, fn: Callable) -> None`

Example:

```python
# pipeline_backend/__init__.py
from ._backend_impl import (
    Pipeline,
    _register_py_op,   # C++ bound, internal
    _list_ops_impl,    # C++ bound, internal
    _describe_ops_impl # C++ bound, internal
)

def register_py_op(name, fn):
    """
    Register a pure-Python operation into the backend engine.
    fn must be callable(data).
    """
    _register_py_op(name, fn)

def list_ops():
    """Return a list of all registered operation names."""
    return _list_ops_impl()

def describe_ops():
    """
    Return a list of dicts describing operations:
    [
      {
        "name": "normalize",
        "impl": "cpp" | "python"
        // optionally, "callable": <py function> for python ops
      },
      ...
    ]
    """
    return _describe_ops_impl()

__backend_metadata__ = {
    "name": "pipeline_backend",
    "version": "X.Y.Z",
    "api_version": "1",
    "capsule_name": "pipeline_backend.register_op_v1",
    "data_abi_tag": "Data-v1",
    "supports_python_ops": True
}
```

On the C++ side, `_describe_ops_impl` will convert `OpEntry` into Python dicts.

---

## 4. Plugin Requirements

A **plugin** is a Python package (wheel) that:

* Provides operations (C++ or Python),
* Registers them with the backend registry on import,
* Ships a `pipeline_plugin.json` metadata file.

### 4.1 C++ Plugin (with C++ operations)

Requirements:

1. Use backend SDK headers (for `Data`, `OpFn`).
2. Implement one or more `void my_op(pipeline::Data&)` functions.
3. In module init:

   * Import `pipeline_backend`,
   * Get the capsule,
   * Call its function pointer to register C++ operations.

Example C++ registration:

```cpp
#include <Python.h>
#include "pipeline_data.h" // defines pipeline::Data, pipeline::OpFn

using register_op_c_api_t = void(*)(const char*, pipeline::OpFn);

void my_cpp_op(pipeline::Data& d) {
    // ...
}

static void register_with_backend() {
    PyObject* backend = PyImport_ImportModule("pipeline_backend");
    if (!backend) return; // handle error properly

    PyObject* capsule = PyObject_GetAttrString(backend, "_register_op_capsule");
    if (!capsule) {
        Py_DECREF(backend);
        return;
    }

    auto reg_fn = (register_op_c_api_t) PyCapsule_GetPointer(
        capsule,
        "pipeline_backend.register_op_v1"
    );
    if (!reg_fn) {
        Py_DECREF(capsule);
        Py_DECREF(backend);
        return;
    }

    // Register operation
    reg_fn("my_cpp_op", &my_cpp_op);

    Py_DECREF(capsule);
    Py_DECREF(backend);
}

PyMODINIT_FUNC PyInit_my_cpp_plugin(void) {
    register_with_backend();
    // create and return module object (pybind11 / CPython)
    return create_my_module(); 
}
```

Metadata example (`pipeline_plugin.json`):

```json
{
  "name": "my_cpp_plugin",
  "version": "1.0.0",
  "backend_api_version": "1",
  "backend_data_abi": "Data-v1",
  "operations": [
    {
      "name": "my_cpp_op",
      "type": "cpp",
      "description": "Example C++ operation."
    }
  ]
}
```

### 4.2 Python-only Plugin (no C++)

Requirements:

1. Implement Python functions `fn(data)`.
2. On import, call `pipeline_backend.register_py_op(name, fn)`.
3. Include matching metadata in `pipeline_plugin.json` with `"type": "python"`.

Example:

```python
# my_py_plugin/__init__.py
import json
from importlib.resources import files
import pipeline_backend as pb

def my_py_op(data):
    # pure Python or calling other Python packages (numpy, etc.)
    ...

# register at import
pb.register_py_op("my_py_op", my_py_op)

def get_metadata():
    meta_path = files("my_py_plugin").joinpath("pipeline_plugin.json")
    return json.loads(meta_path.read_text(encoding="utf-8"))
```

Metadata:

```json
{
  "name": "my_py_plugin",
  "version": "1.0.0",
  "backend_api_version": "1",
  "backend_data_abi": "Data-v1",
  "operations": [
    {
      "name": "my_py_op",
      "type": "python",
      "description": "Example Python-only operation."
    }
  ]
}
```

---

## 5. Validation

Validation has two phases:

1. **Static metadata validation**: checks JSON against schema, versions, ABI tags.
2. **Runtime validation**: checks that ops are actually registered and match the declared type.

### 5.1 Static Checks

* `pipeline_plugin.json` must exist.
* Must conform to JSON Schema.
* `backend_api_version` must equal backend’s `__backend_metadata__["api_version"]`.
* `backend_data_abi` must equal backend’s `__backend_metadata__["data_abi_tag"]`.

### 5.2 Runtime Checks

Given a backend and a plugin:

1. Import backend and load `__backend_metadata__`.
2. Import plugin (this triggers op registration).
3. Call `backend.describe_ops()` → list of dicts with `"name"` and `"impl"` (`"cpp"` / `"python"`).
4. For each operation declared in plugin metadata:

   * Ensure it exists in `describe_ops()` with the same `"impl"` type.
   * For `"python"` ops:

     * Ensure it has a callable associated and that it accepts exactly one parameter.

This ensures that:

* the plugin’s metadata is honest,
* registration actually happened,
* the backend and plugin are API/ABI compatible,
* Python-only ops have the correct signature.
