# Multi-Column Output UDF Interface Design

## The Problem

Today, UDFs return a single value that maps to a single column. To output multiple related values (e.g., height and width from an image), users must either:

1. **Write two separate UDFs** — duplicating both IO and computation (e.g., reading and decoding the same image twice)
2. **Return a struct** — but then values are nested, not separate columns

### Current Behavior

```python
@udf(data_type=pa.struct([("height", pa.int32()), ("width", pa.int32())]))
def dimensions(image: bytes) -> tuple[int, int]:
    img = Image.open(io.BytesIO(image))
    return img.size

table.add_columns({"dims": dimensions})
```

**Result:** Single struct column, nested access required

| image | dims |
|-------|------|
| 0x89PNG... | {height: 480, width: 640} |
| 0x89PNG... | {height: 1080, width: 1920} |

### Desired Behavior

```python
@udf
def dimensions(image: bytes) -> Columns[Dimensions]:
    img = Image.open(io.BytesIO(image))
    return Dimensions(height=img.height, width=img.width)

table.add_columns(dimensions)
```

**Result:** Separate columns, direct access

| image | height | width |
|-------|--------|-------|
| 0x89PNG... | 480 | 640 |
| 0x89PNG... | 1080 | 1920 |

---

## How Other Systems Handle This

### Summary Table

| System | UDF Returns | Expansion Method | Type Declaration |
|--------|-------------|------------------|------------------|
| **Spark** | Tuple → StructType | `.select("col.*")` | Schema in decorator |
| **Snowflake** | Tuple (yield) | `RETURNS TABLE(...)` | SQL DDL |
| **DuckDB** | Dict → Struct | `SELECT col.*` | `struct_type({...})` |
| **Polars** | Dict → Struct | `.unnest("col")` | `pl.Struct([...])` |
| **Pandas** | List or Series | `result_type='expand'` | Column names in assignment |

**Common Pattern**: Most systems use **struct/dict return + explicit unpacking step**.

### PySpark / Spark

**Approach: Return a StructType, then select/expand fields**

```python
from pyspark.sql.types import StructType, StructField, IntegerType

schema = StructType([
    StructField("height", IntegerType()),
    StructField("width", IntegerType())
])

@udf(returnType=schema)
def dimensions(image):
    return (height, width)

# Apply UDF, then expand struct to columns
df.withColumn("dims", dimensions("image")) \
  .select("*", "dims.height", "dims.width")
```

**Pandas UDF variant** can return `pd.DataFrame`:
```python
@pandas_udf("height int, width int")
def dimensions(images: pd.Series) -> pd.DataFrame:
    return pd.DataFrame({"height": [...], "width": [...]})
```

[Source: Stack Overflow](https://stackoverflow.com/questions/35322764/apache-spark-assign-the-result-of-udf-to-multiple-dataframe-columns) | [Spark Docs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.pandas_udf.html)

### Snowflake

**Approach: UDTFs (User-Defined Table Functions) with explicit RETURNS TABLE**

```sql
CREATE FUNCTION dimensions(image BINARY)
RETURNS TABLE (height INT, width INT)
LANGUAGE PYTHON
AS $$
class Dimensions:
    def process(self, image):
        yield (height, width)
$$;

-- Called with TABLE() syntax
SELECT * FROM TABLE(dimensions(image_col));
```

[Source: Snowflake Docs](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-tabular-functions)

### DuckDB

**Approach: Return a dict → maps to STRUCT, then use `column.*` to expand**

```python
def dimensions(image):
    return {"height": h, "width": w}

con.create_function(
    'dimensions',
    dimensions,
    [duckdb.BLOB],
    duckdb.struct_type({'height': 'INTEGER', 'width': 'INTEGER'})
)
```

```sql
SELECT dims.* FROM (
    SELECT dimensions(image) AS dims FROM images
);
```

[Source: DuckDB Docs](https://duckdb.org/docs/stable/clients/python/function) | [MotherDuck Guide](https://motherduck.com/learn-more/duckdb-struct-nested-data/)

### Polars

**Approach: Return dict from `map_elements`, then `.unnest()`**

```python
def dimensions(row):
    return {"height": h, "width": w}

df.with_columns(
    pl.col("image")
    .map_elements(dimensions, return_dtype=pl.Struct([
        pl.Field("height", pl.Int32),
        pl.Field("width", pl.Int32)
    ]))
    .alias("dims")
).unnest("dims")
```

[Source: Polars User Guide](https://docs.pola.rs/user-guide/expressions/structs/) | [Polars Docs](https://docs.pola.rs/py-polars/html/reference/dataframe/api/polars.DataFrame.unnest.html)

### Pandas

**Approach: `apply()` with `result_type='expand'` or return `pd.Series`**

```python
def dimensions(row):
    return [height, width]

# Option 1: result_type='expand'
df[['height', 'width']] = df.apply(dimensions, axis=1, result_type='expand')

# Option 2: Return Series with named index
def dimensions(row):
    return pd.Series({'height': h, 'width': w})

df.join(df.apply(dimensions, axis=1))
```

[Source: Pandas Docs](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html) | [GeeksforGeeks](https://www.geeksforgeeks.org/python/return-multiple-columns-using-pandas-apply-method/)

---

## Proposed Interface Options

### Option A: Struct Unpacking Method

Build on existing struct pattern, add `.unpack()` method:

```python
@udf(data_type=pa.struct([("height", pa.int32()), ("width", pa.int32())]))
def dimensions(image: bytes) -> tuple[int, int]:
    return img.size

# Creates two columns: "height" and "width"
table.add_columns(dimensions.unpack())

# Or with prefix to avoid collisions
table.add_columns(dimensions.unpack(prefix="img_"))  # img_height, img_width
```

**Pros**: Builds on existing struct pattern, explicit, backward compatible
**Cons**: Requires PyArrow struct type definition

### Option B: Output Columns Parameter

New decorator parameter for multi-output:

```python
@udf(output_columns={"height": pa.int32(), "width": pa.int32()})
def dimensions(image: bytes) -> tuple[int, int]:
    return (h, w)

table.add_columns(dimensions)  # Creates height, width columns
```

**Pros**: Clean, declarative
**Cons**: New parameter, dict order must match return order

### Option C: Dict Return Type

Infer columns from dict keys at runtime:

```python
@udf
def dimensions(image: bytes) -> dict[str, int]:
    return {"height": h, "width": w}

table.add_columns(dimensions)  # Keys become column names
```

**Pros**: Most Pythonic, zero boilerplate
**Cons**: Runtime schema discovery, all values same type

### Option D: NamedTuple Return

Use Python NamedTuple for named fields:

```python
from typing import NamedTuple

class Dimensions(NamedTuple):
    height: int
    width: int

@udf
def dimensions(image: bytes) -> Dimensions:
    return Dimensions(height=h, width=w)

table.add_columns(dimensions)  # Uses NamedTuple field names
```

**Pros**: Type-safe, self-documenting, standard Python
**Cons**: Requires extra class definition

### Option E: Wrapper Type

Use a `Columns[T]` wrapper to indicate multi-column output:

```python
from geneva import Columns
from typing import NamedTuple

class Dimensions(NamedTuple):
    height: int
    width: int

@udf
def dimensions(image: bytes) -> Columns[Dimensions]:
    return Dimensions(height=h, width=w)

table.add_columns(dimensions)  # Creates height, width columns
```

**Pros**: Intent clear at definition, type-checker friendly, familiar pattern
**Cons**: New type to learn

---

## Disambiguation: Struct vs Multiple Columns

If we support NamedTuple/dict returns, how does the user indicate whether they want:
- A **struct column** (nested data, one column)
- **Multiple columns** (unpacked, separate columns)

### Disambiguation Option 1: Call-site (like DuckDB/Polars)

```python
@udf
def dimensions(image: bytes) -> Dimensions:
    return Dimensions(height=h, width=w)

# Keep as struct column
table.add_columns({"dims": dimensions})  # → dims.height, dims.width

# Unpack to separate columns
table.add_columns(dimensions.unpack())   # → height, width columns
```

| Pros | Cons |
|------|------|
| Same UDF works both ways | Easy to forget `.unpack()` |
| User decides at use-time | Intent not clear at definition |
| Follows industry pattern | |

### Disambiguation Option 2: Wrapper Type

```python
from geneva import Columns

# Unpacks to separate columns
@udf
def dimensions(image: bytes) -> Columns[Dimensions]:
    return Dimensions(height=h, width=w)

# Returns struct column (default)
@udf
def dimensions(image: bytes) -> Dimensions:
    return Dimensions(height=h, width=w)
```

| Pros | Cons |
|------|------|
| Intent clear at definition | New type to learn |
| Type checker validates | Slightly more verbose |
| Familiar pattern (`Optional[T]`, `list[T]`) | |

### Disambiguation Option 3: Decorator Parameter

```python
# Unpacks to separate columns
@udf(unpack=True)
def dimensions(image: bytes) -> Dimensions:
    ...

# Returns struct (default)
@udf
def dimensions(image: bytes) -> Dimensions:
    ...
```

| Pros | Cons |
|------|------|
| Explicit | Behavior hidden in decorator |
| Simple, no new types | Not visible in return type |

---

## Geneva Options Comparison

| Option | Geneva Interface | Aligns With | Trade-offs |
|--------|------------------|-------------|------------|
| **A: Struct Unpacking** | `dimensions.unpack()` | DuckDB, Polars | Explicit; requires struct type in decorator |
| **B: Output Columns** | `@udf(output_columns={...})` | Spark Pandas UDF | Declarative; new API parameter |
| **C: Dict Return** | `-> dict[str, int]` | DuckDB, Polars | Pythonic; runtime schema discovery |
| **D: NamedTuple** | `-> Dimensions` | Python stdlib | Type-safe; requires extra class |
| **E: Wrapper Type** | `-> Columns[Dimensions]` | Novel | Clear intent; type-checker friendly |

### Discussion Points

1. **Explicit vs Implicit**: Options A/B require declaring output columns upfront; C/D/E infer from return value
2. **Two-step vs One-step**: A follows industry pattern (return struct, then expand); B/C/D/E expand automatically
3. **Type Safety**: A/B/D/E have full type info at definition time; C discovers at runtime
4. **Learning Curve**: C/D are most Pythonic; A/B are most explicit about multi-column intent
5. **Struct vs Columns Disambiguation**:
   - A: Explicit `.unpack()` call
   - B: Always multi-column (no struct option)
   - C/D: Ambiguous without additional mechanism
   - E: `Columns[T]` = multi-column, plain `T` = struct

### Ergonomics Ranking for AI Engineers

| Rank | Option | Why |
|------|--------|-----|
| 1 | **E: Wrapper Type** | Type-safe, IDE-friendly, clear disambiguation |
| 2 | **D: NamedTuple** + call-site `.unpack()` | Familiar Python, flexible |
| 3 | **C: Dict Return** | Zero boilerplate, but runtime errors |
| 4 | **A/B: PyArrow types** | Requires PyArrow knowledge |
