# 21.1 Investigation: Null Mask Error for Nullable Arrays/Records

## Problem Statement

When reading Avro files with nullable complex types (arrays or records), the following error occurs:

```
DecodeError: DataFrame builder error: Decode error: Invalid data: Failed to apply null mask: failed to determine supertype of list[i32] and i32
```

## Root Cause Analysis

The error originates in `src/reader/record_decoder.rs` in the `NullableBuilder::finish()` method:

```rust
fn finish(&mut self) -> Result<Series, DecodeError> {
    let inner_series = self.inner.finish()?;
    let validity = std::mem::take(&mut self.validity);

    let mask = BooleanChunked::new("mask".into(), &validity);

    let result = inner_series
        .zip_with(
            &mask,
            &Series::new_null(self.name.clone().into(), inner_series.len()),  // <-- BUG HERE
        )
        .map_err(|e| DecodeError::InvalidData(format!("Failed to apply null mask: {}", e)))?;

    Ok(result.with_name(self.name.clone().into()))
}
```

### The Issue

The `zip_with` method requires both series to have compatible types. When the inner series is a complex type like `List[i32]` or `Struct`, and we create a null series using `Series::new_null()`, the null series has type `Null`, not the same type as the inner series.

Polars cannot determine a supertype between `List[i32]` and `Null`, hence the error:
```
failed to determine supertype of list[i32] and i32
```

### Why It Works for Primitives

For primitive types (Int32, String, etc.), Polars can coerce `Null` to the primitive type. But for complex types like `List` and `Struct`, this coercion fails.

## Solution

Use `Series::full_null(name, len, dtype)` instead of `Series::new_null(name, len)`. This creates a null series with the correct dtype matching the inner series:

```rust
fn finish(&mut self) -> Result<Series, DecodeError> {
    let inner_series = self.inner.finish()?;
    let validity = std::mem::take(&mut self.validity);

    let mask = BooleanChunked::new("mask".into(), &validity);

    // Create null series with the SAME dtype as inner_series
    let null_series = Series::full_null(
        self.name.clone().into(),
        inner_series.len(),
        inner_series.dtype(),
    );

    let result = inner_series
        .zip_with(&mask, &null_series)
        .map_err(|e| DecodeError::InvalidData(format!("Failed to apply null mask: {}", e)))?;

    Ok(result.with_name(self.name.clone().into()))
}
```

## Verification

Python test confirming the fix approach:

```python
import polars as pl

# List type
list_series = pl.Series('test', [[1, 2], [3, 4], [5, 6]])
null_series = pl.Series('test', [None, None, None], dtype=list_series.dtype)
mask = pl.Series('mask', [True, False, True])
result = list_series.zip_with(mask, null_series)
# Result: [[1, 2], null, [5, 6]]

# Struct type
struct_series = pl.Series('test', [{'a': 1}, {'a': 2}, {'a': 3}])
null_series = pl.Series('test', [None, None, None], dtype=struct_series.dtype)
result = struct_series.zip_with(mask, null_series)
# Result: [{1}, null, {3}]
```

## Files to Modify

- `src/reader/record_decoder.rs`: Update `NullableBuilder::finish()` method

## Behavior Validation

The fix produces behavior consistent with established precedents:

### polars.read_avro
```python
# polars.read_avro output for nullable array:
shape: (4, 3)
┌─────┬────────────────┬─────────────────┐
│ id  ┆ nullable_array ┆ nullable_record │
│ --- ┆ ---            ┆ ---             │
│ i32 ┆ list[i32]      ┆ struct[1]       │
╞═════╪════════════════╪═════════════════╡
│ 0   ┆ [1, 2]         ┆ {10}            │
│ 1   ┆ null           ┆ null            │  # <-- proper null, not []
│ 2   ┆ [3, 4]         ┆ {20}            │
│ 3   ┆ null           ┆ null            │
└─────┴────────────────┴─────────────────┘
```

### fastavro
```python
{'id': 0, 'nullable_array': [1, 2]}
{'id': 1, 'nullable_array': None}  # <-- None, not []
{'id': 2, 'nullable_array': [3, 4]}
```

### Key Semantic Distinction
The fix correctly preserves the distinction between:
- `null` (absence of value) → `None`/`null` in DataFrame
- Empty collection `[]` → empty list `[]` in DataFrame

This is non-surprising behavior that matches user expectations and Avro semantics.

## Requirements Addressed

- 5.5: Preserve null values from Avro unions containing null
