# UDF Type Detection Ambiguity

## The Problem

Today, the `@udf` decorator auto-detects whether a function is SCALAR, ARRAY, or RECORD_BATCH based on type hints. This detection is implicit and fails silently in common cases.

### Current Detection Logic

```python
# Priority order:
# 1. RECORD_BATCH: exactly 1 param annotated as pa.RecordBatch
# 2. ARRAY: return type is pa.Array AND all params are pa.Array
# 3. SCALAR: everything else (default fallback)
```

### The Danger: Silent Fallback to SCALAR

When detection fails, the UDF silently becomes SCALAR. This causes:
- **Wrong execution**: UDF called per-row instead of per-batch
- **Runtime crashes**: Array operations on scalar values
- **Silent wrong results**: Code runs but produces garbage

## Ambiguous Cases by Severity

| Signature | Detected | Expected | Severity | Problem |
|-----------|----------|----------|----------|---------|
| `(a: pa.Array, b: int) -> pa.Array` | SCALAR | ARRAY | 🔴 **High** | Mixed types → silent scalar fallback |
| `(x: pa.Array)` *(no return)* | SCALAR | ARRAY | 🔴 **High** | Missing return hint → silent fallback |
| `(x: pa.Array) -> int` | SCALAR | ARRAY | 🔴 **High** | Wrong return type → silent fallback |
| `(a: pa.Array, b: pa.Array) -> list` | SCALAR | ARRAY | 🟡 **Medium** | Wrong return type |
| `(x: pa.Array) -> pa.Array` | ARRAY | RECORD_BATCH? | 🟡 **Medium** | Single array param is ambiguous |
| `(x: int, y: int) -> int` | SCALAR | SCALAR | 🟢 Clear | Unambiguous |
| `(batch: pa.RecordBatch) -> pa.Array` | RECORD_BATCH | RECORD_BATCH | 🟢 Clear | Unambiguous |

### Example: Silent Failure

```python
# User writes this, expecting vectorized execution
@udf(data_type=pa.int64())
def add_arrays(a: pa.Array, b: int) -> pa.Array:  # b: int is the bug
    return pc.add(a, b)

# Geneva silently treats as SCALAR
# At runtime: a is a single int, not an array
# Result: crash or wrong output, no warning at definition time
```

---

## Proposed Solution: Explicit Decorators

Replace the single `@udf` decorator with three explicit decorators:

| Decorator | Input | Output | Use Case |
|-----------|-------|--------|----------|
| `@udf` | Scalar values | Scalar value | Per-row transformations |
| `@batch_udf` | `pa.Array` columns | `pa.Array` | Vectorized column operations |
| `@recordbatch_udf` | `pa.RecordBatch` | `pa.Array` | Full batch access |

### Before (Implicit Detection)

```python
# SCALAR - works
@udf
def file_size(image: bytes) -> int:
    return len(image)

# ARRAY - works, but only if you get ALL annotations exactly right
@udf(data_type=pa.int64())
def add_one(a: pa.Array) -> pa.Array:
    return pc.add(a, 1)

# ARRAY - SILENTLY BECOMES SCALAR (bug!)
@udf(data_type=pa.int64())
def add_columns(a: pa.Array, b: int) -> pa.Array:  # forgot b: pa.Array
    return pc.add(a, b)
```

### After (Explicit Decorators)

```python
# SCALAR - explicit
@udf
def file_size(image: bytes) -> int:
    return len(image)

# ARRAY - explicit, validates input types
@batch_udf(data_type=pa.int64())
def add_one(a: pa.Array) -> pa.Array:
    return pc.add(a, 1)

# ARRAY - ERROR at definition time!
@batch_udf(data_type=pa.int64())
def add_columns(a: pa.Array, b: int) -> pa.Array:  # Error: b must be pa.Array
    return pc.add(a, b)

# RECORD_BATCH - explicit
@recordbatch_udf(data_type=pa.float32())
def process_batch(batch: pa.RecordBatch) -> pa.Array:
    return batch.column(0)
```

---

## Benefits of Explicit Decorators

### 1. Fail Fast

```python
@batch_udf  # Decorator knows this MUST be batched
def bad(a: pa.Array, b: int) -> pa.Array:  # ERROR: b is not pa.Array
    ...
```

Errors at **definition time**, not runtime.

### 2. Self-Documenting

```python
# Reader immediately knows execution model
@udf                    # "This runs per-row"
@batch_udf              # "This runs vectorized on columns"
@recordbatch_udf        # "This gets the whole batch"
```

### 3. Relaxed Type Hints

With explicit decorators, type hints become optional documentation rather than required for detection:

```python
# Today: MUST have exact annotations or silent failure
@udf(data_type=pa.int64())
def add(a: pa.Array, b: pa.Array) -> pa.Array:  # All 3 annotations required
    return pc.add(a, b)

# With explicit decorator: annotations are optional
@batch_udf(data_type=pa.int64())
def add(a, b):  # Still works, decorator knows it's batched
    return pc.add(a, b)
```

### 4. Better IDE/Type Checker Support

Separate decorators can have distinct type signatures:

```python
# batch_udf expects Array inputs
@batch_udf
def process(a: pa.Array) -> pa.Array: ...  # Type checker validates

# udf expects scalar inputs
@udf
def process(a: int) -> int: ...  # Type checker validates
```

---

## Migration Path

### Option A: Deprecate Implicit Detection

1. Add new explicit decorators: `@batch_udf`, `@recordbatch_udf`
2. Keep `@udf` for scalar only (no more auto-detection)
3. Emit deprecation warning when `@udf` is used with array annotations
4. Remove implicit detection in next major version

### Option B: Keep Both, Prefer Explicit

1. Add new explicit decorators
2. Keep `@udf` with current auto-detection for backward compatibility
3. Document explicit decorators as preferred
4. Warn on ambiguous cases

### Option C: Add Validation to Current System

1. Keep single `@udf` decorator
2. Add `mode="scalar"|"array"|"recordbatch"` parameter
3. When mode is explicit, validate annotations match
4. When mode is implicit, warn on ambiguous fallback

---

## Comparison

| Approach | Breaking Change | Explicitness | Verbosity |
|----------|-----------------|--------------|-----------|
| **Separate decorators** | Yes (migration) | High | Low (decorator name) |
| **mode parameter** | No | Medium | Medium (`mode="array"`) |
| **Validation only** | No | Low | Low |

### Recommendation

**Separate decorators** (`@udf`, `@batch_udf`, `@recordbatch_udf`) because:

1. Most explicit — decorator name tells you the execution model
2. Familiar pattern — similar to `@pytest.fixture` vs `@pytest.mark.parametrize`
3. Type-checker friendly — each decorator has distinct expected signatures
4. Forces user to think about execution model upfront

---

## Naming Alternatives

| Scalar | Batched | RecordBatch | Notes |
|--------|---------|-------------|-------|
| `@udf` | `@batch_udf` | `@recordbatch_udf` | Explicit, matches internal terminology |
| `@udf` | `@vectorized_udf` | `@table_udf` | More descriptive |
| `@scalar_udf` | `@array_udf` | `@recordbatch_udf` | All explicit, parallel structure |
| `@udf` | `@udf.batch` | `@udf.recordbatch` | Namespaced, single import |
