# Job Plan / Dry-Run Mode

## The Problem

Backfill and materialized view (MV) refresh jobs are expensive — they provision Ray
clusters, create actor pools, and process data across distributed workers. In many
operational scenarios (CI/CD pipelines, cron schedules, manual checks), a job is
triggered when there is actually **no work to do**: all rows are already computed, all
fragments are checkpointed, or the source table hasn't changed.

Today there is no cheap way to answer: *"Does this job have pending work?"* without
launching the full distributed pipeline.

### Current Behavior

```python
# Must launch a full Ray job just to discover there's nothing to do
with conn.ray_context():
    table.backfill("embeddings")  # Spins up actors, plans, finds 0 tasks, exits
```

For MV refresh the situation is worse — the three-pass pre-processing
(`_build_dst_to_src_mapping`, `_identify_new_source_fragments`,
`_append_placeholder_fragments`) runs inside the Ray job before discovering there's
nothing to copy.

### Desired Behavior

```python
plan = table.plan_backfill("embeddings")
if plan.has_work:
    with conn.ray_context():
        table.backfill("embeddings")
```

```bash
mf plan backfill my_table embeddings
rc=$?
if [ $rc -eq 0 ]; then
  echo "Up to date, skipping cluster spin-up"
elif [ $rc -eq 2 ]; then
  mf backfill run my_table embeddings
fi
```

---

## Existing Infrastructure

The planning logic already exists — it just isn't exposed independently.

### Backfill Planning

`_plan_read()` (`src/geneva/apply/__init__.py:567`) iterates table fragments, checks
checkpoint coverage, and produces `ScanTask` objects for gaps. The result includes:

- **`tasks`** — iterator of `ScanTask` with fragment ID, offset, row count
- **`skipped_stats`** — `{"fragments": N, "rows": N}` for already-completed work
- **`skipped_fragments`** — fragment IDs with existing data file checkpoints

If `tasks` is empty after materialization, there is no work.

### MV Refresh Planning

`run_ray_copy_table()` (`src/geneva/runners/ray/pipeline.py:3042`) performs three
pre-processing passes before dispatching work:

1. **`_build_dst_to_src_mapping()`** — maps destination fragments to source fragments,
   validates checkpoints against current source data files
2. **`_identify_new_source_fragments()`** — finds source fragments with no
   representation in destination
3. **`_extract_new_row_ids_from_source_fragment()`** — gets new row IDs that need
   placeholder rows

These passes read Lance metadata and `__source_row_id` columns but don't require Ray.

### Mismatch Detection (Backfill)

`backfill_async()` (`src/geneva/table.py:1356`) already checks for:

- **UDF version mismatch** — checkpoint was written by an older UDF version
- **Source files mismatch** — input column data files changed (e.g., upstream column
  was re-backfilled)

These checks determine whether the default `WHERE <col> IS NULL` filter should be
dropped in favor of reprocessing all rows.

---

## Design

### Python API

Separate `plan_backfill()` and `plan_refresh()` methods on `Table` return plan
objects without dispatching work.  `backfill()` and `refresh()` retain their
original signatures and return types (`str` job-id and `None`, respectively).

```python
# Plan only (no Ray cluster needed)
plan = table.plan_backfill("col")
if plan.has_work:
    with conn.ray_context():
        table.backfill("col")

plan = table.plan_refresh()
if plan.has_work:
    with conn.ray_context():
        table.refresh()
```

**No breaking changes:** `backfill()` still returns `str` (job_id), `refresh()`
still returns `None`.

### Data Model

Plan types use a base class for shared fields and separate subclasses for
type-specific fields:

```python
@attrs.define(frozen=True)
class _BasePlan:
    """Common fields shared by all job plans."""
    table_name: str
    version: int
    has_work: bool
    total_tasks: int
    total_rows_pending: int
    skipped_fragments: int
    skipped_rows: int
    total_fragments: int
    total_rows: int

@attrs.define(frozen=True)
class BackfillPlan(_BasePlan):
    job_type: Literal["backfill"]  # set automatically, not passed to __init__
    column_name: str = ""
    where: str | None = None
    udf_mismatch: bool = False
    srcfiles_mismatch: bool = False

@attrs.define(frozen=True)
class RefreshPlan(_BasePlan):
    job_type: Literal["refresh"]  # set automatically, not passed to __init__
    new_source_fragments: int = 0
    stale_rows: int = 0
    invalidated_fragments: int = 0

JobPlan = BackfillPlan | RefreshPlan
```

`job_type` uses `Literal` typing and is set via `attrs.field(init=False, default=...)`
so callers never pass it — it's determined by the plan subclass.

### CLI

A top-level `plan` command group, extensible as new job types are added:

```
mf plan backfill <table> <column> [--json]
mf plan refresh <table> [--src-version VERSION] [--json]
```

### Exit Codes

Following `git diff --exit-code` and `terraform plan -detailed-exitcode` conventions:

| Code | Meaning |
|------|---------|
| **0** | Up to date — no work needed |
| **1** | Error — table not found, invalid column, validation failure |
| **2** | Work pending — tasks would be created if the job ran |

```bash
mf plan backfill my_table embeddings
rc=$?
case $rc in
  0) echo "Up to date, skipping" ;;
  2) echo "Work pending, launching backfill"
     mf backfill run my_table embeddings ;;
  *) echo "Error, aborting"; exit 1 ;;
esac
```

---

## Implementation

### `plan_backfill()`

Standalone method that runs validation, mismatch detection, WHERE resolution,
and fragment scanning — everything except Ray dispatch.

```python
def plan_backfill(
    self, col_name, *, udf=None, where=None, read_version=None,
    num_frags=None, task_size=None,
) -> BackfillPlan:
```

**Steps:**

1. `validate_backfill_args()` — same validation as real backfill
2. `fetch_udf()` + unmarshal — resolve UDF for version checking
3. `detect_backfill_mismatches()` — extracted helper in `apply/utils.py`
4. `resolve_backfill_where()` — extracted helper in `apply/utils.py`
5. `_plan_read()` — produces tasks and skipped stats
6. Materialize the task iterator, count tasks and sum rows
7. Return `BackfillPlan`

**What it does NOT do:**

- Start Ray or create actors
- Write any checkpoints
- Modify table state
- Perform admission control

### `plan_refresh()`

Standalone method that reads MV metadata and source/destination fragments
to determine what work a refresh would do.

```python
def plan_refresh(self, *, src_version=None) -> RefreshPlan:
```

**Steps:**

1. Check MV type — UDTF views return `has_work=True` immediately
2. `_open_mv_source_table()` — namespace-aware helper (shared with `refresh()`)
3. Build lightweight dst→src fragment mapping by reading `__source_row_id`
   from destination fragments
4. `_identify_new_source_fragments()` — finds source fragments not yet in destination
5. Single-pass fragment iteration to count total/new source rows
6. Return `RefreshPlan`

**What it does NOT do:**

- Start Ray or create actors
- Append placeholder fragments
- Delete stale rows
- Modify table state

**Note:** `run_ray_copy_table()` does its own richer analysis (checkpoint-aware mapping
with `_build_dst_to_src_mapping`, data file validation, source deletion detection).
The plan is a lightweight approximation; the pipeline does the full work.

**UDTF-backed MVs:** Returns `has_work=True` unconditionally since UDTF refresh is
always a full re-execution. No meaningful task-level breakdown.

### `backfill()` and `refresh()`

Unchanged signatures and return types. `backfill()` returns `str` (job_id),
`refresh()` returns `None`.

### Refactoring: Shared Helpers

Mismatch detection and filter defaulting were extracted from `backfill_async()` into
reusable helpers used by both `backfill_async()` and `plan_backfill()`:

```python
# src/geneva/apply/utils.py

def detect_backfill_mismatches(table, col_name, udf, read_version) -> tuple[bool, bool]:
    """Returns (udf_mismatch, srcfiles_mismatch).
    Opens checkpoint store once and reuses for both checks."""

def resolve_backfill_where(col_name, col_field, where, udf_mismatch, srcfiles_mismatch) -> str | None:
    """Compute effective WHERE filter for backfill."""
```

Namespace-aware source table opening was extracted into a shared method
used by both `refresh()` and `plan_refresh()`:

```python
# src/geneva/table.py

def _open_mv_source_table(self) -> Table | None:
    """Open source table using namespace-aware logic.
    Handles namespace connections, same-db reuse, and cross-db connections."""
```

### CLI Implementation

```python
# src/cli/muffuletta/cli.py

plan_app = typer.Typer(name="plan", help="Plan jobs without executing them")
app.add_typer(plan_app)

@plan_app.command("backfill")
def plan_backfill_cmd(table_name, column, json_output=False):
    plan = table.plan_backfill(column)
    ...

@plan_app.command("refresh")
def plan_refresh_cmd(table_name, src_version=None, json_output=False):
    plan = table.plan_refresh(src_version=src_version)
    ...
```

The CLI connects directly to the table via `geneva.connect()` (not via K8s
port-forward), so it works without a running cluster.

---

## Files Changed

| File | Change |
|------|--------|
| `src/geneva/plan.py` | New — `_BasePlan`, `BackfillPlan`, `RefreshPlan` |
| `src/geneva/table.py` | New `plan_backfill()`, `plan_refresh()`, `_open_mv_source_table()`; `backfill_async()` refactored to use extracted helpers |
| `src/geneva/apply/utils.py` | New `detect_backfill_mismatches()` and `resolve_backfill_where()` |
| `src/geneva/runners/ray/pipeline.py` | `run_ray_copy_table()` returns `job_id` (was `None`) |
| `src/cli/muffuletta/cli.py` | `plan` command group with `backfill` and `refresh` subcommands |
| `src/tests/test_plan.py` | New — unit tests for plan types and plan methods |
| `src/integ_tests/test_pipeline.py` | Plan assertions added to existing `test_ray_add_column_pipeline` and `test_matview_refresh_with_admission_control` |

---

## Cost Profile

Neither plan operation requires a Ray cluster. All work is local Lance
metadata reads and checkpoint store listing.

| Operation | Backfill Plan | Refresh Plan |
|-----------|---------------|--------------|
| Lance metadata read | Fragment list | Both src + dst fragment lists |
| Checkpoint store list | One `list()` call | No (skips checkpoint validation) |
| Fragment row counting | `frag.count_rows()` per fragment | `frag.count_rows()` per fragment |
| Source data scan | No | `__source_row_id` per dst fragment |
| Ray cluster | **No** | **No** |
| Checkpoint writes | **No** | **No** |
| Table mutations | **No** | **No** |

**Estimated wall-clock time:** For a table with ~1,000 fragments, backfill plan
completes in seconds. Refresh plan takes slightly longer (~10s) due to
`__source_row_id` scans across destination fragments.

---

## Usage Patterns

### CI/CD Gate

Skip cluster provisioning when there's no work:

```bash
mf plan backfill my_table embeddings
if [ $? -eq 0 ]; then
  echo "Column up to date, skipping cluster spin-up"
  exit 0
fi

# Only provision cluster if work is pending
mf clusters create ...
mf backfill run my_table embeddings
```

### Programmatic SDK

```python
plan = table.plan_backfill("embeddings")
if not plan.has_work:
    print(f"Already complete: {plan.skipped_rows} rows")
    return

print(f"Pending: {plan.total_tasks} tasks, {plan.total_rows_pending} rows")
with conn.ray_context():
    job_id = table.backfill("embeddings")
    print(f"Completed job {job_id}")
```

### Monitoring / Alerting

```bash
# Cron job: alert if MV is stale
mf plan refresh analytics_view --json | jq '.has_work'
```

---

## Deferred / Future Work

1. **Progress estimation** — use historical job metrics (rows/sec from
   past runs) to estimate wall-clock time for pending work.
2. **UDTF MV refresh granularity** — UDTF-backed MVs always do a full
   refresh, so task-level breakdown isn't meaningful. A future version
   could compare source data hashes to detect no-op refreshes.
3. **Multi-column plan** — plan all backfillable columns on a table at
   once (e.g., `mf plan backfill my_table --all`).
4. **Richer refresh plan** — currently the plan uses a lightweight
   `__source_row_id` scan rather than the full `_build_dst_to_src_mapping`
   (which requires a `MapTask`). A future version could surface
   invalidated fragments from checkpoint/data-file validation.
