# dotpipe

**Declarative transformation pipelines**

Part of the Shape pillar, `dotpipe` enables building data transformations through a `Pipeline` class with a fluent interface, or via a declarative DSL.

## Overview

`dotpipe` provides a `Pipeline` class that transforms a single dictionary through a sequence of steps: assigning new fields (with optional function chains), plucking/selecting fields, deleting fields, and applying functions to existing fields. It also provides a declarative DSL format for defining pipelines as serializable data.

## Pipeline Class

### Basic Usage

```python
from shape.dotpipe import Pipeline

data = {
    "user": {
        "firstName": "Alice",
        "lastName": "Smith",
        "age": 30
    },
    "scores": [85, 92, 78]
}

result = (
    Pipeline(data)
    .assign("name", from_path="user.firstName")
    .assign("total_score", from_path="scores", then="sum")
    .pluck("name", "total_score")
    .apply()
)
# {"name": "Alice", "total_score": 255}
```

### Pipeline Methods

#### `assign(new_field, *, from_path, then=None)`

Adds a new field to the result, calculated from a path on the **original source data** (not the current state). The `from_path` uses the `dotpath` engine, so wildcards, slices, and descendant traversal are supported.

The optional `then` parameter applies a function or chain of functions to the selected value:

```python
pipeline = Pipeline(data)

# Simple assignment
pipeline.assign("name", from_path="user.firstName")

# With a named safe function
pipeline.assign("upper_name", from_path="user.firstName", then="upper")

# With a chain of functions (applied left-to-right)
pipeline.assign("unique_tags", from_path="tags", then=["unique", "len"])

# With a Python callable
pipeline.assign("doubled", from_path="value", then=lambda x: x * 2)
```

#### `pluck(*fields)`

Selects only the specified fields from the current result, discarding all others:

```python
pipeline.pluck("name", "email")  # Keep only these two fields
```

#### `delete(*fields)`

Removes the specified fields from the current result:

```python
pipeline.delete("temp_data", "internal_id")
```

#### `apply_to(field, func)`

Applies a function or function chain to an existing field in the current result:

```python
pipeline.apply_to("name", "upper")
pipeline.apply_to("scores", lambda x: sorted(x, reverse=True))
```

#### `apply()`

Finalizes the pipeline and returns the resulting transformed dictionary:

```python
result = pipeline.apply()
```

### Complete Example

```python
from shape.dotpipe import Pipeline

data = {
    "user": {"name": "alice smith", "email": "ALICE@EXAMPLE.COM"},
    "orders": [
        {"id": 1, "total": 50},
        {"id": 2, "total": 30},
        {"id": 3, "total": 80}
    ],
    "internal_ref": "xyz123"
}

result = (
    Pipeline(data)
    .assign("name", from_path="user.name", then="upper")
    .assign("email", from_path="user.email", then="lower")
    .assign("order_count", from_path="orders", then="len")
    .delete("internal_ref", "orders")
    .apply()
)
# {"user": {...}, "name": "ALICE SMITH", "email": "alice@example.com", "order_count": 3}
```

## SAFE_FUNCTIONS Registry

The pipeline's `then` clause can reference named functions from a built-in safe registry. These are the only string-referenced functions allowed:

| Name | Function | Description |
|------|----------|-------------|
| `len` | `len()` | Length of a sequence |
| `sum` | `sum()` | Sum of numeric values |
| `upper` | `str.upper()` | Uppercase a string |
| `lower` | `str.lower()` | Lowercase a string |
| `flatten` | custom | Flatten a list of lists into a single list |
| `unique` | custom | Deduplicate a list, preserving order |
| `first` | custom | First element of a sequence (or None) |
| `last` | custom | Last element of a sequence (or None) |

Referencing an unregistered function name raises `ValueError`.

## Declarative DSL

### from_dsl(data, pipeline_dsl)

For serializable, data-driven pipelines, use `from_dsl()` (also aliased as `pipe()`):

```python
from shape.dotpipe import from_dsl

data = {
    "user": {"name": "alice", "email": "ALICE@EXAMPLE.COM"},
    "scores": [85, 92, 78]
}

pipeline_dsl = [
    {"verb": "assign", "field": "name", "path": "user.name", "then": "upper"},
    {"verb": "assign", "field": "email", "path": "user.email", "then": "lower"},
    {"verb": "assign", "field": "total", "path": "scores", "then": "sum"},
    {"verb": "delete", "fields": ["scores"]},
    {"verb": "pluck", "fields": ["name", "email", "total"]}
]

result = from_dsl(data, pipeline_dsl)
# {"name": "ALICE", "email": "alice@example.com", "total": 255}
```

### DSL Verb Reference

| Verb | Keys | Description |
|------|------|-------------|
| `assign` | `field`, `path`, `then` (optional) | Add a new field from a source path |
| `pluck` | `fields` (list) | Keep only listed fields |
| `delete` | `fields` (list) | Remove listed fields |
| `apply_to` | `field`, `func` | Apply a function to an existing field |

## Command Line

```bash
# Apply a pipeline DSL file to a data file
dotpipe pipeline.json data.json

# Read data from stdin, pipeline from file
echo '{"user": {"name": "alice"}}' | dotpipe pipeline.json

# YAML pipeline and data
dotpipe pipeline.yaml data.yaml
```

The pipeline file is a JSON or YAML list of step objects:

```json
[
    {"verb": "assign", "field": "name", "path": "user.name", "then": "upper"},
    {"verb": "pluck", "fields": ["name"]}
]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `pipeline_file` | Path to JSON/YAML file containing the pipeline DSL, or `-` for stdin |
| `data_file` | Path to source JSON/YAML data file, or `-` for stdin (default) |

### Options

| Flag | Description |
|------|-------------|
| `--in`, `--input-format` | Input format: `json` or `yaml` (inferred from extension if omitted) |

Note: Both `pipeline_file` and `data_file` cannot be `-` (stdin) at the same time.

## How It Works

1. `Pipeline.__init__()` deep-copies the input data into both `_source_data` (immutable reference) and `_current_state` (mutable working copy)
2. `assign()` reads from `_source_data` using the `dotpath` engine and writes to `_current_state`
3. `pluck()`, `delete()`, and `apply_to()` modify `_current_state` directly
4. `apply()` returns the final `_current_state`

This separation means `assign()` always reads from the original data, regardless of prior mutations.

## Related Tools

- **[dotmod](dotmod.md)** - Individual modifications
- **[dotbatch](dotbatch.md)** - Atomic batch operations
- **[dotpluck](dotpluck.md)** - Extract and reshape data
