Operation Log (oplog)
---

The `aaiclick/oplog/` module captures data operation provenance inside ClickHouse with zero AI dependencies. Every object created or transformed within a session is recorded, enabling lineage tracing, debugging, and post-job data sampling. Activation via `data_context()` is planned for Phase 3 — see `docs/future.md`.

---

# ClickHouse Tables

**Implementation**: `aaiclick/oplog/models.py` — see `OPERATION_LOG_DDL`, `TABLE_REGISTRY_DDL`, `init_oplog_tables()`

## operation_log

Append-only audit log of every data operation. Fields: `id` (Snowflake, auto-generated by ClickHouse DEFAULT), `result_table`, `operation`, `args` (`Array(String)`), `kwargs` (`Map(String, String)`), `sql_template`, `task_id`, `job_id`, `created_at`. TTL-managed via `AAICLICK_OPLOG_TTL_DAYS` (default: 90 days).

`ORDER BY created_at` — nullable `job_id` cannot be in a MergeTree sorting key.

`args` holds positional inputs (e.g. N sources for `concat`); `kwargs` holds named inputs (e.g. `{"left": ..., "right": ...}` for binary ops). Forward lineage query: `has(args, t) OR mapContains(kwargs, t)`.

## table_registry

Maps every ephemeral table to its owning `job_id` for post-job cleanup. Persistent tables (`p_` prefix) have no `job_id` and are excluded from cleanup.

## Initialization

`init_oplog_tables(ch_client)` creates both tables idempotently then validates schema via `_validate_schema()`. Raises `RuntimeError` on column name/type mismatch (stale table detection).

---

# OplogCollector

**Implementation**: `aaiclick/oplog/collector.py` — see `OplogCollector`, `get_oplog_collector()`

Buffer-based event sink. Collects `OperationEvent` objects in memory; batch-inserts to both `operation_log` and `table_registry` on `flush()`. Accessed via ContextVar `_oplog_collector`.

## Activation ⚠️ NOT YET IMPLEMENTED (Phase 3)

The `data_context()` function does not yet accept an `oplog` parameter. OplogCollector activation
is planned for Phase 3 — see `docs/future.md` — "Operation Provenance Integration (Phase 3)".

Once implemented, the intended API is:

```python
# Bool — creates a plain OplogCollector()
async with data_context(oplog=True):
    ...

# Pre-configured instance — used by orchestration (task_id/job_id already set)
collector = OplogCollector(task_id=task_id, job_id=job_id)
async with data_context(oplog=collector):
    ...
```

`data_context` will store the collector in the ContextVar, call `flush()` on clean exit, and reset in `finally`. On exception, the buffer is discarded.

---

# Instrumentation Points

**Implementation**: `aaiclick/data/operators.py`, `aaiclick/data/ingest.py`, `aaiclick/data/object.py`, `aaiclick/data/data_context.py`

Each instrumentation is a 2-line addition: get collector from ContextVar, call `record()` if not None.

| Location                               | Operation              | `args`                       | `kwargs`                                     |
|----------------------------------------|------------------------|------------------------------|----------------------------------------------|
| `data_context.create_object()`         | (none — registry only) | —                            | —                                            |
| `data_context.create_object_from_value()` | `"create_from_value"` | `[]`                        | `{}`                                         |
| `operators._apply_operator_db()`       | `"add"`, `"sub"`, etc. | `[]`                         | `{"left": left.table, "right": right.table}` |
| `operators._apply_aggregation()`       | `"sum"`, `"mean"`, etc.| `[]`                         | `{"source": source.table}`                   |
| `ingest.concat_objects_db()`           | `"concat"`             | `[s.table for s in sources]` | `{}`                                         |
| `ingest.insert_objects_db()`           | `"insert"`             | `[]`                         | `{"source": src.table, "target": tgt.table}` |
| `object.Object.copy()`                 | `"copy"`               | `[]`                         | `{"source": self.table}`                     |

`create_object()` only calls `record_table()` (populates `table_registry`), not `record()`, to avoid double-counting — higher-level functions record the operation that produced the table.

---

# Graph Queries

**Implementation**: `aaiclick/oplog/graph.py` — see `backward_oplog()`, `forward_oplog()`, `oplog_subgraph()`, `OplogGraph.to_prompt_context()`

Iterative graph traversal over `operation_log`. `OplogGraph.to_prompt_context()` formats the graph as plain text for LLM consumption.

---

# Table Lifecycle & Cleanup

Post-job table sampling (replacing ephemeral tables with 10-row samples after job completion) is planned for Phase 3. See `docs/future.md` — "Oplog: Table Lifecycle & Cleanup".

---

# Environment Variables

| Variable                  | Default | Description                          |
|---------------------------|---------|--------------------------------------|
| `AAICLICK_OPLOG_TTL_DAYS` | `90`    | TTL for `operation_log` rows in days |
