# DevNote: EnumBuilder and pyo3-polars Enum→Categorical Workaround

## Context

Task 22.1 implemented `EnumBuilder` for decoding Avro enum types into Polars Enum columns.

## Decision

The Rust implementation correctly creates `DataType::Enum` with `FrozenCategories`, but pyo3-polars converts this to `Categorical` during DataFrame serialization across the Rust/Python boundary.

### Root Cause

pyo3-polars uses Arrow C FFI to transfer DataFrames between Rust and Python. Arrow's Dictionary type doesn't distinguish between Polars Enum (fixed categories from schema) and Categorical (runtime-inferred categories). When Python Polars reconstructs the Series from Arrow, it defaults to Categorical.

Key code path in pyo3-polars `types.rs`:
- `PySeries::into_pyobject` uses `export_series()` which goes through Arrow FFI
- The `PyDataType::into_pyobject` method DOES handle Enum correctly, but this is only used for type conversion, not Series data transfer

### Solution

We encapsulate the workaround in Rust (`src/python/reader.rs`) via the `dataframe_to_py_with_enums()` function:

1. Before returning a DataFrame to Python, identify Enum columns and their categories from the Rust schema
2. Export the DataFrame via Arrow FFI (Enum becomes Categorical)
3. Cast Categorical columns back to Enum types using Python Polars API

This approach:
- **Encapsulated**: Python users always see correct Enum types without any additional code
- **Transparent**: No workaround visible in the Python API
- **Correct**: Uses schema-defined categories from the Rust side
- **Removable**: Can be deleted when pyo3-polars adds native Enum support

### Research Findings

During investigation, we discovered:
- **Categorical→Enum cast allocates a new buffer** (different pointer addresses)
- **UInt→Enum cast is true zero-copy** (same buffer address, ~900 bytes metadata only)

We initially attempted to export Enum columns as their physical representation (UInt8/16/32) and cast UInt→Enum for better performance. However, `to_physical_repr()` on Enum columns triggered a "not implemented" panic in polars-core 0.52.0. The simpler Categorical→Enum cast works reliably.

### Implementation Details

The workaround is applied in both reader types:
- `AvroReaderCore.__next__()` - used by `scan()`
- `AvroReader.__next__()` - used by `open()`

Both call `dataframe_to_py_with_enums()` which handles the Enum reconstruction.

## Future

This workaround should be removed when pyo3-polars adds native Enum support in the Arrow FFI layer. Monitor:
- https://github.com/pola-rs/pyo3-polars/issues
- https://github.com/pola-rs/polars/issues/20089 (tracking issue for Enum/Categorical in Parquet)

## Verification

All 5 enum tests pass:
- `test_read_enum_file` - File reads without errors
- `test_enum_dtype` - Columns have `pl.Enum` type (not Categorical) via `scan()`
- `test_enum_dtype_open_api` - Columns have `pl.Enum` type via `open()`
- `test_enum_values` - Values are correct
- `test_enum_categories` - Categories match schema symbols

Both APIs return proper Enum types with clean Python code:
- `jetliner.scan()` ✓
- `jetliner.open()` ✓
