# E2E Test Plan for Jetliner Avro Reader

## Executive Summary

This document provides a comprehensive plan for implementing an end-to-end test suite for Jetliner that ensures compatibility with all major Avro libraries and validates correct reading of real-world Avro files. The research analyzed test strategies from:

- **Apache Avro** (Java, Python, C++, Rust) - Official implementation
- **fastavro** - High-performance Python library
- **hamba/avro** - Modern Go implementation
- **linkedin/goavro** - Go implementation with extensive edge case tests

All test data and assets identified are licensed under either Apache License 2.0 or MIT License, both of which permit use in this project with proper attribution.

## Research Findings

### 1. Apache Avro (Official Implementation)

**Repository**: https://github.com/apache/avro
**License**: Apache License 2.0 ✅ (Permits use with attribution)
**Languages**: Java, Python, C++, Rust, C#, JavaScript, Ruby, PHP, Perl

#### Key Test Strategies

**Cross-Language Interoperability**:
- Shared test data at `share/test/data/` used by all language implementations
- Standard `interop.avsc` schema tests all Avro features in one comprehensive schema
- Each language implementation must read files generated by others
- Validates spec compliance across implementations

**Compression Codec Testing**:
- All codecs tested with same data for consistency validation
- Files: `weather.avro` (null), `weather-deflate.avro`, `weather-snappy.avro`, `weather-zstd.avro`
- Tests codec-specific features (e.g., snappy CRC32 validation)

**Sync Marker & Block Handling** (Critical for streaming):
- Sync point seeking for parallel processing
- File splitting at block boundaries
- Block-level error recovery
- Custom sync marker handling

**Schema Evolution & Compatibility**:
- Reader/writer schema differences (BACKWARD, FORWARD, FULL compatibility)
- Type promotions (int→long, int→float/double, string↔bytes)
- Field additions with defaults, field removals
- Projection (reading subset of fields)

**Edge Cases & Corruption**:
- Invalid magic bytes, missing sync markers
- Block count violations, invalid codecs
- EOF handling, truncated files
- Metadata edge cases

#### Test Files Downloaded

Located in `tests/data/apache-avro/`:
- `weather.avro` (358 bytes) - Baseline uncompressed file
- `weather-deflate.avro` (319 bytes) - Deflate compression
- `weather-snappy.avro` (330 bytes) - Snappy compression
- `weather-zstd.avro` (333 bytes) - Zstandard compression
- `weather-sorted.avro` (335 bytes) - Sorted blocks
- `interop.avsc` - Comprehensive schema with all Avro types
- `weather.avsc` - Simple 3-field weather schema

#### Python Test Analysis (test_datafile.py)

Key test patterns identified:
```python
# Parametrized testing across all codecs and type combinations
CODECS_TO_VALIDATE = ["null", "deflate", "snappy", "zstd", ...]
TEST_PAIRS = [
    (schema, datum) for all primitive + complex types
]

# Tests:
- test_round_trip: Write 10x, read all back
- test_append: Multiple append operations
- test_context_manager: Resource cleanup
- test_metadata: Custom key-value metadata
- test_empty_datafile: Empty block handling
```

**Comprehensive Type Coverage**:
- All 8 primitive types (null, boolean, int, long, float, double, bytes, string)
- Complex types: records, arrays, maps, unions, enums, fixed
- Recursive records (Lisp cons cell example)

#### Java Test Classes (Reading-Focused)

- `TestDataFile.java` - Core reading, codecs, sync points, seeking
- `TestDataFileReader.java` - Edge cases, EOF, validation
- `TestDataFileCorruption.java` - Corrupted files, sync recovery
- `TestDataFileConcat.java` - File concatenation, multi-codec
- `TestDataFileCustomSync.java` - Custom 16-byte sync markers

#### C++ DataFileTests.cc

20+ reading tests including:
- `testReadFull`, `testReadProjection` - Schema projection
- `testReaderGeneric`, `testReaderGenericByName` - Generic reading
- `testReaderSyncSeek`, `testReaderSyncDiscovery` - Sync navigation
- `testReaderSplits` - Parallel processing with random splits
- `testCompatibility{Null,Deflate,Snappy,Zstd}Codec` - Codec validation
- `testLastSync`, `testMetadata` - Advanced features

### 2. fastavro (High-Performance Python)

**Repository**: https://github.com/fastavro/fastavro
**License**: MIT License ✅ (Permits use with attribution)
**Focus**: Performance (10x faster than Apache Avro Python with C extensions)

#### Key Test Strategies

**Pre-Built Test Files** (30+ files):
- Unlike Apache Avro which generates files at build time, fastavro includes pre-built binary test files
- Compression variants at multiple levels: deflate (0,1,3,6,9), snappy, zstd
- Edge case files: `no-fields.avro`, `null.avro`, `recursive.avro`
- Interop files: `java-generated-uuid.avro` tests reading Java-produced files

**Block-Level Reading**:
- Tests with 2,000+ record datasets
- Block iteration with record count validation
- Memory-efficient streaming tests

**Schema Evolution** (50+ parametrized combinations):
- Extensive type promotion testing
- Field addition/removal with defaults
- Union type evolution
- Array/map value type changes

**Logical Types**:
- Comprehensive logical type tests
- Pandas integration tests (datetime conversion)
- UUID generation and validation

**Graceful Degradation**:
- Tests behavior when codec libraries are missing
- Separate schema validation testing
- Error handling for corrupted data

#### Test Files Downloaded

Located in `tests/data/fastavro/`:
- `no-fields.avro` (139 bytes) - Record with no fields edge case
- `null.avro` (3.2K) - Null type handling
- `recursive.avro` (218 bytes) - Self-referential records
- `java-generated-uuid.avro` (8.3K) - Java interop with UUID logical type

#### Test Pattern Analysis

From `test_fastavro.py`:
```python
# Extensive schema evolution tests
@pytest.mark.parametrize("writer_schema,reader_schema,value", [
    # Type promotions
    ("int", "long", 123),
    ("int", "float", 123),
    ("float", "double", 1.23),
    ("string", "bytes", "hello"),
    # Field changes
    (record_with_field_a, record_with_field_a_b_default, {...}),
    # Union evolution
    ...
])
def test_schema_migration(writer_schema, reader_schema, value):
    # Write with writer schema, read with reader schema
    ...

# Block reading tests
def test_block_reader():
    # Tests reading 2000+ records
    # Validates block count tracking
    # Tests record iteration
    ...

# Compression codec tests (7 codecs)
@pytest.mark.parametrize("codec", ["null", "deflate", "snappy", "zstd", ...])
def test_compression(codec):
    ...
```

### 3. Go Implementations

#### linkedin/goavro

**Repository**: https://github.com/linkedin/goavro
**License**: Apache License 2.0 ✅
**Status**: Legacy (LinkedIn moved to hamba/avro internally)

**Unique Value**: Edge case test files in `fixtures/` directory:
- `firstBlockCountNotGreaterThanZero.avro` - Invalid block count
- `blockCountExceedsMaxBlockCount.avro` - Overflow protection
- `cannotReadBlockSize.avro` - Size reading errors
- `blockSizeNotGreaterThanZero.avro` - Invalid block size

These malformed files test error handling and resilience - valuable for our skip mode testing.

#### hamba/avro

**Repository**: https://github.com/hamba/avro
**License**: MIT License ✅
**Status**: Actively maintained, used by LinkedIn internally
**Performance**: 238.7 ns/op decode, 196.2 ns/op encode

**Test Resources** in `testdata/`:
- `bad-schema.avsc` - Invalid schema scenarios
- `concurrent-schema.avsc` - Concurrent processing tests
- `superhero.avsc` - Complex nested structures
- `superhero.bin` - Binary test data

### 4. Rust apache-avro

**Repository**: https://github.com/apache/avro (lang/rust) & https://github.com/apache/avro-rs
**Crate**: apache-avro on crates.io
**License**: Apache License 2.0 ✅

**Key Features**:
- Recent arrow-avro integration for high-performance Arrow conversion
- Strong Serde integration for type-safe deserialization
- Uses shared Apache Avro test data from `share/test/`

**API Pattern**:
```rust
use apache_avro::Reader;
let reader = Reader::new(&input[..]).unwrap();
for value in reader {
    println!("{:?}", value.unwrap());
}

// With schema resolution
let reader = Reader::with_schema(&reader_schema, &input[..]).unwrap();
```

### 5. Interoperability Schema

The `interop.avsc` schema is used across all Apache Avro implementations:

```json
{
  "type": "record",
  "name": "Interop",
  "namespace": "org.apache.avro",
  "fields": [
    {"name": "intField", "type": "int"},
    {"name": "longField", "type": "long"},
    {"name": "stringField", "type": "string"},
    {"name": "boolField", "type": "boolean"},
    {"name": "floatField", "type": "float"},
    {"name": "doubleField", "type": "double"},
    {"name": "bytesField", "type": "bytes"},
    {"name": "nullField", "type": "null"},
    {"name": "arrayField", "type": {"type": "array", "items": "double"}},
    {"name": "mapField", "type": {
      "type": "map",
      "values": {"type": "record", "name": "Foo",
                 "fields": [{"name": "label", "type": "string"}]}
    }},
    {"name": "unionField", "type": ["boolean", "double", {"type": "array", "items": "bytes"}]},
    {"name": "enumField", "type": {"type": "enum", "name": "Kind", "symbols": ["A","B","C"]}},
    {"name": "fixedField", "type": {"type": "fixed", "name": "MD5", "size": 16}},
    {"name": "recordField", "type": {
      "type": "record",
      "name": "Node",
      "fields": [
        {"name": "label", "type": "string"},
        {"name": "children", "type": {"type": "array", "items": "Node"}}
      ]
    }}
  ]
}
```

This single schema tests:
- All 8 primitive types
- Arrays, maps, unions, enums, fixed, nested records
- Recursive types (Node contains array of Node)
- Complex union types

## License Requirements

### Apache License 2.0 Assets

**Sources**: Apache Avro (all files from share/test/), linkedin/goavro

**Requirements**:
1. Include copy of Apache License 2.0 in the project
2. Add attribution notice (example below)
3. Preserve copyright notices
4. Document any modifications

**Recommended Attribution**:
Create `tests/data/apache-avro/LICENSE` and `tests/data/apache-avro/NOTICE`:

```
Tests data files are from the Apache Avro project:
https://github.com/apache/avro

Copyright 2010-2025 The Apache Software Foundation

Licensed under the Apache License, Version 2.0
```

### MIT License Assets

**Sources**: fastavro, hamba/avro

**Requirements**:
1. Include copy of MIT license with the files
2. Include copyright notice

**Recommended Attribution**:
Create `tests/data/fastavro/LICENSE`:

```
Test data files are from the fastavro project:
https://github.com/fastavro/fastavro

Copyright (c) 2011 Miki Tebeka

Licensed under the MIT License
```

### License Compatibility

✅ Both MIT and Apache 2.0 licenses are permissive and compatible
✅ Can use assets from both in the same project
✅ Just need proper attribution for each

## Test Data Organization

### Current Structure

```
jetstream/
├── tests/
│   ├── data/
│   │   ├── apache-avro/          # Apache 2.0 licensed
│   │   │   ├── LICENSE
│   │   │   ├── NOTICE
│   │   │   ├── weather.avro
│   │   │   ├── weather-deflate.avro
│   │   │   ├── weather-snappy.avro
│   │   │   ├── weather-zstd.avro
│   │   │   ├── weather-sorted.avro
│   │   │   ├── interop.avsc
│   │   │   └── weather.avsc
│   │   └── fastavro/              # MIT licensed
│   │       ├── LICENSE
│   │       ├── no-fields.avro
│   │       ├── null.avro
│   │       ├── recursive.avro
│   │       └── java-generated-uuid.avro
│   ├── interop_tests.rs           # Existing
│   ├── property_tests.rs          # Existing
│   ├── schema_tests.rs            # Existing
│   ├── spec_compliance_tests.rs   # Existing
│   └── e2e_real_files_tests.rs    # NEW - to be created
├── python/
│   └── tests/
│       ├── test_python_api.py     # Existing
│       └── test_e2e_files.py      # NEW - to be created
└── e2e-test-plan.md               # This document
```

### Additional Files to Download

The following should be downloaded from Apache Avro repository when needed:

**Priority 1** - Essential for basic compatibility:
- ✅ `weather.avro` - Already downloaded
- ✅ `weather-deflate.avro` - Already downloaded
- ✅ `weather-snappy.avro` - Already downloaded
- ✅ `weather-zstd.avro` - Already downloaded

**Priority 2** - For comprehensive codec testing:
- [ ] Generate interop files using `interop.avsc` (with null, deflate, snappy, zstd codecs)
- [ ] `share/test/data/syncInMeta.avro` (22K) - Sync marker edge cases

**Priority 3** - For edge case testing:
- [ ] goavro malformed files (if we want to test error handling thoroughly)
- [ ] Additional fastavro compression level variants (deflate 0,1,3,6,9)

## E2E Test Implementation Plan

### Phase 1: Basic Interoperability Tests (PRIORITY 1)

**Goal**: Verify Jetliner can read standard Avro files from other implementations

**Test File**: `tests/e2e_real_files_tests.rs`

#### Test 1.1: Read Apache Avro Weather Files

```rust
#[test]
fn test_read_weather_uncompressed() {
    // Read weather.avro
    // Validate schema matches weather.avsc
    // Validate we can read all records
    // Compare output with expected values
}

#[test]
fn test_read_weather_deflate() {
    // Read weather-deflate.avro
    // Verify same data as uncompressed
    // Validate deflate codec is used
}

#[test]
fn test_read_weather_snappy() {
    // Read weather-snappy.avro
    // Verify same data as uncompressed
    // Validate snappy codec with CRC32
}

#[test]
fn test_read_weather_zstd() {
    // Read weather-zstd.avro
    // Verify same data as uncompressed
    // Validate zstd codec is used
}
```

**Expected Weather Data**:
The weather files contain simple records with schema:
```json
{
  "type": "record",
  "name": "Weather",
  "fields": [
    {"name": "station", "type": "string"},
    {"name": "time", "type": "long"},
    {"name": "temp", "type": "int"}
  ]
}
```

Should contain records like:
```
{station: "011990-99999", time: 1234567890, temp: 20}
{station: "011990-99999", time: 1234567895, temp: 22}
...
```

#### Test 1.2: Read fastavro Edge Case Files

```rust
#[test]
fn test_read_no_fields_record() {
    // Read no-fields.avro
    // Schema: record with no fields
    // Validates empty record handling
}

#[test]
fn test_read_null_type() {
    // Read null.avro
    // Validates null type handling
}

#[test]
fn test_read_recursive_record() {
    // Read recursive.avro
    // Validates self-referential record types
}

#[test]
fn test_read_java_uuid() {
    // Read java-generated-uuid.avro
    // Validates UUID logical type
    // Tests reading Java-generated files
}
```

#### Test 1.3: Codec Comparison

```rust
#[test]
fn test_all_codecs_produce_same_data() {
    // Read weather.avro, weather-deflate.avro, weather-snappy.avro, weather-zstd.avro
    // Verify all produce identical records
    // Validates codec implementations are correct
}
```

**Success Criteria**:
- All files read without errors
- Data matches expected schemas
- Compressed files produce same data as uncompressed
- Logical types (UUID) decoded correctly

### Phase 2: Comprehensive Type Coverage (PRIORITY 2)

**Goal**: Generate and test files with all Avro types using interop.avsc

**Approach**:
1. Use Python `avro` library to generate test files with `interop.avsc`
2. Create files with various data patterns (nulls, defaults, edge values)
3. Test reading with Jetliner

#### Test 2.1: Generate Interop Test Files

Create Python script `tests/generate_interop_files.py`:

```python
import avro.datafile
import avro.io
import avro.schema
import random

# Load interop.avsc
with open('tests/data/apache-avro/interop.avsc') as f:
    schema = avro.schema.parse(f.read())

# Generate test data with all field types populated
def generate_interop_record():
    return {
        "intField": random.randint(-2147483648, 2147483647),
        "longField": random.randint(-9223372036854775808, 9223372036854775807),
        "stringField": "test_string_" + str(random.randint(0, 1000)),
        "boolField": random.choice([True, False]),
        "floatField": random.uniform(-1e6, 1e6),
        "doubleField": random.uniform(-1e12, 1e12),
        "bytesField": bytes([random.randint(0, 255) for _ in range(16)]),
        "nullField": None,
        "arrayField": [random.uniform(0, 100) for _ in range(5)],
        "mapField": {"key1": {"label": "label1"}, "key2": {"label": "label2"}},
        "unionField": random.choice([
            True,  # boolean branch
            3.14,  # double branch
            [b"bytes1", b"bytes2"]  # array branch
        ]),
        "enumField": random.choice(["A", "B", "C"]),
        "fixedField": bytes([random.randint(0, 255) for _ in range(16)]),
        "recordField": {
            "label": "root",
            "children": [
                {"label": "child1", "children": []},
                {"label": "child2", "children": []}
            ]
        }
    }

# Generate files with different codecs
for codec in ["null", "deflate", "snappy", "zstd"]:
    filename = f"tests/data/generated/interop-{codec}.avro"
    with open(filename, 'wb') as f:
        writer = avro.datafile.DataFileWriter(f, avro.io.DatumWriter(), schema, codec=codec)
        for i in range(100):  # 100 records
            writer.append(generate_interop_record())
        writer.close()
```

#### Test 2.2: Read and Validate Interop Files

```rust
#[test]
fn test_read_interop_all_types() {
    // Read interop-null.avro
    // Validate all 14 fields present
    // Validate field types correct
    // Validate 100 records
}

#[test]
fn test_interop_primitive_types() {
    // Focus on: int, long, string, bool, float, double, bytes, null
    // Validate value ranges
    // Validate type conversions to Arrow/Polars
}

#[test]
fn test_interop_complex_types() {
    // Focus on: array, map, union, enum, fixed, record
    // Validate nested structures
    // Validate recursive records (Node with children)
}

#[test]
fn test_interop_all_codecs() {
    // Read interop-{null,deflate,snappy,zstd}.avro
    // Verify all produce identical data
    // Validates compression correctness
}
```

**Success Criteria**:
- All primitive types decode correctly
- Complex types (arrays, maps, unions) work
- Recursive types handled properly
- All codecs produce identical output

### Phase 3: Block Handling & Streaming (PRIORITY 2)

**Goal**: Test multi-block files, sync markers, streaming behavior

#### Test 3.1: Multi-Block Files

Generate large files with multiple blocks:

```python
# Generate file with 10,000+ records (will span multiple blocks)
def generate_large_file(num_records=10000):
    schema = avro.schema.parse('{"type": "record", "name": "Test", "fields": [{"name": "id", "type": "long"}, {"name": "data", "type": "string"}]}')

    with open('tests/data/generated/large-multiblock.avro', 'wb') as f:
        writer = avro.datafile.DataFileWriter(f, avro.io.DatumWriter(), schema)
        for i in range(num_records):
            writer.append({"id": i, "data": f"record_{i}"})
        writer.close()
```

```rust
#[test]
fn test_read_multiblock_file() {
    // Read large-multiblock.avro
    // Verify 10,000 records read
    // Verify block boundaries handled correctly
    // Track block count
}

#[test]
fn test_sync_marker_validation() {
    // Read file, validate sync markers between blocks
    // Test our sync marker validation works
}

#[test]
fn test_streaming_memory_efficiency() {
    // Read large file in batches
    // Verify memory doesn't grow unbounded
    // Test backpressure in prefetch buffer
}
```

#### Test 3.2: Block Seeking (If Implemented)

```rust
#[test]
#[ignore] // Only if seeking is implemented
fn test_seek_to_sync_marker() {
    // Seek to middle of file
    // Find next sync marker
    // Resume reading from there
    // Validates parallel processing capability
}
```

### Phase 4: Error Handling & Resilience (PRIORITY 3)

**Goal**: Test error recovery, skip mode, corrupted files

#### Test 4.1: Corrupted File Handling

Create corrupted test files:

```python
def create_corrupted_files():
    # 1. Invalid magic bytes
    with open('tests/data/generated/invalid-magic.avro', 'wb') as f:
        f.write(b'INVALID!')  # Should be "Obj\x01"

    # 2. Truncated file (cut off mid-block)
    # Read valid file, write only first 50%

    # 3. Invalid sync marker
    # Read valid file, corrupt sync marker bytes

    # 4. Block with invalid record count
    # Manually craft file with count=0 or negative
```

```rust
#[test]
fn test_invalid_magic_bytes() {
    // Try to read invalid-magic.avro
    // Should error with ParseError
    // Error should mention magic bytes
}

#[test]
fn test_truncated_file_strict_mode() {
    // Read truncated file in strict mode
    // Should error when hitting EOF
}

#[test]
fn test_truncated_file_skip_mode() {
    // Read truncated file in skip mode
    // Should read valid blocks, skip truncated one
    // Should accumulate error
}

#[test]
fn test_corrupted_sync_marker_recovery() {
    // File with bad sync marker
    // Skip mode should recover by scanning for next valid sync
    // Should continue reading after recovery
}
```

#### Test 4.2: Missing Codec Libraries

```rust
#[test]
#[cfg(not(feature = "bzip2"))]
fn test_missing_codec_library() {
    // Try to read bzip2-compressed file without bzip2 feature
    // Should error with CodecError
    // Error should mention missing codec
}
```

### Phase 5: Schema Evolution (PRIORITY 3)

**Goal**: Test reader/writer schema compatibility

#### Test 5.1: Schema Evolution Scenarios

Generate files with different schemas, read with evolved schema:

```python
# Writer schema v1
writer_schema_v1 = {
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "id", "type": "long"},
        {"name": "name", "type": "string"}
    ]
}

# Reader schema v2 (added field with default)
reader_schema_v2 = {
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "id", "type": "long"},
        {"name": "name", "type": "string"},
        {"name": "email", "type": "string", "default": "unknown@example.com"}
    ]
}

# Write with v1, read with v2
```

```rust
#[test]
fn test_schema_evolution_add_field_with_default() {
    // Write file with schema v1
    // Read with schema v2 (added field)
    // Verify default value populated
}

#[test]
fn test_schema_evolution_remove_field() {
    // Write with schema v2
    // Read with schema v1 (removed field)
    // Verify removed field ignored
}

#[test]
fn test_schema_evolution_type_promotion_int_to_long() {
    // Write int field
    // Read as long
    // Verify promotion works
}

#[test]
fn test_schema_evolution_type_promotion_int_to_double() {
    // Write int field
    // Read as double
    // Verify promotion works
}
```

### Phase 6: Python API E2E Tests (PRIORITY 2)

**Test File**: `python/tests/test_e2e_files.py`

```python
import jetliner
import polars as pl
import pytest

def test_read_apache_avro_weather_files():
    """Test reading official Apache Avro test files"""
    # Test uncompressed
    with jetliner.open("tests/data/apache-avro/weather.avro") as reader:
        dfs = list(reader)
        assert len(dfs) > 0
        df = pl.concat(dfs)
        assert "station" in df.columns
        assert "time" in df.columns
        assert "temp" in df.columns

    # Test compressed variants
    for codec in ["deflate", "snappy", "zstd"]:
        with jetliner.open(f"tests/data/apache-avro/weather-{codec}.avro") as reader:
            dfs = list(reader)
            assert len(dfs) > 0

def test_read_fastavro_files():
    """Test reading fastavro test files"""
    # Test reading Java-generated UUID file
    with jetliner.open("tests/data/fastavro/java-generated-uuid.avro") as reader:
        dfs = list(reader)
        df = pl.concat(dfs)
        # Validate UUID field present and formatted correctly
        assert "uuid_field" in df.columns

def test_scan_with_projection():
    """Test LazyFrame scan with projection pushdown"""
    lf = jetliner.scan("tests/data/apache-avro/weather.avro")
    result = lf.select(["station", "temp"]).collect()

    # Should only have projected columns
    assert result.columns == ["station", "temp"]
    assert "time" not in result.columns

def test_scan_with_filter():
    """Test LazyFrame scan with predicate pushdown"""
    lf = jetliner.scan("tests/data/apache-avro/weather.avro")
    result = lf.filter(pl.col("temp") > 15).collect()

    # All temps should be > 15
    assert (result["temp"] > 15).all()

def test_interop_all_types():
    """Test reading interop schema with all Avro types"""
    lf = jetliner.scan("tests/data/generated/interop-null.avro")
    df = lf.collect()

    # Validate all 14 fields present
    expected_fields = [
        "intField", "longField", "stringField", "boolField",
        "floatField", "doubleField", "bytesField", "nullField",
        "arrayField", "mapField", "unionField", "enumField",
        "fixedField", "recordField"
    ]
    for field in expected_fields:
        assert field in df.columns

@pytest.mark.parametrize("codec", ["null", "deflate", "snappy", "zstd"])
def test_interop_all_codecs(codec):
    """Test reading interop files with different codecs"""
    lf = jetliner.scan(f"tests/data/generated/interop-{codec}.avro")
    df = lf.collect()
    assert len(df) == 100  # Generated with 100 records

def test_error_accumulation_skip_mode():
    """Test error tracking in skip mode"""
    # Read file with some corrupted blocks
    with jetliner.open("tests/data/generated/partially-corrupted.avro", strict=False) as reader:
        dfs = list(reader)

        # Should have read some data
        assert len(dfs) > 0

        # Should have accumulated errors
        assert reader.error_count > 0

        # Errors should be accessible
        for error in reader.errors:
            assert error.kind is not None
            assert error.message is not None

def test_large_file_streaming():
    """Test memory efficiency with large file"""
    # Read 10,000 record file
    total_records = 0
    with jetliner.open("tests/data/generated/large-multiblock.avro", batch_size=1000) as reader:
        for df in reader:
            total_records += len(df)
            # Verify batch size respected (roughly)
            assert len(df) <= 1100  # Some tolerance

    assert total_records == 10000
```

### Phase 7: Performance Benchmarks (PRIORITY 4)

Create benchmarks comparing against other libraries:

**File**: `benches/interop_benchmarks.rs`

```rust
use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId};

fn benchmark_read_weather_files(c: &mut Criterion) {
    let mut group = c.benchmark_group("read_weather");

    for codec in ["null", "deflate", "snappy", "zstd"] {
        let path = format!("tests/data/apache-avro/weather-{}.avro", codec);

        group.bench_with_input(
            BenchmarkId::from_parameter(codec),
            &path,
            |b, path| {
                b.iter(|| {
                    // Read file, count records
                })
            }
        );
    }

    group.finish();
}

fn benchmark_read_interop(c: &mut Criterion) {
    // Benchmark reading interop file with all types
    // Compare performance across codecs
}

criterion_group!(benches, benchmark_read_weather_files, benchmark_read_interop);
criterion_main!(benches);
```

Compare against:
- Python fastavro (via pytest-benchmark)
- Python apache-avro
- Rust apache-avro crate

## Test Success Metrics

### Coverage Targets

**Type Coverage**: 100%
- ✅ All 8 primitive types
- ✅ All complex types (record, array, map, union, enum, fixed)
- ✅ All logical types (decimal, date, time, timestamp, uuid, duration)

**Codec Coverage**: 100%
- ✅ null (no compression)
- ✅ deflate
- ✅ snappy (with CRC32 validation)
- ✅ zstd
- ✅ bzip2, xz (if implemented)

**Interoperability**: 100%
- ✅ Can read files from Apache Avro (Java, Python)
- ✅ Can read files from fastavro
- ✅ Can read files from other implementations

**Edge Cases**: High coverage
- ✅ Empty files, empty blocks
- ✅ Single record, large files (10,000+ records)
- ✅ Recursive types, deeply nested structures
- ✅ Records with no fields
- ✅ All null values
- ✅ Special float values (NaN, Infinity)
- ✅ Max/min int/long values
- ✅ Empty strings/bytes/arrays/maps
- ✅ Unicode strings

**Error Handling**:
- ✅ Invalid magic bytes
- ✅ Truncated files (EOF handling)
- ✅ Corrupted sync markers
- ✅ Invalid block counts
- ✅ Missing codecs
- ✅ Schema incompatibilities

**Performance**:
- ⭐ Should be competitive with fastavro (within 2x)
- ⭐ Should be faster than apache-avro Python (target 5-10x)
- ⭐ Memory usage should scale with batch_size, not file size

## Implementation Instructions

### Step 1: Set Up License Files

```bash
# Create license files for test data
cat > tests/data/apache-avro/LICENSE << 'EOF'
Test data files are from the Apache Avro project.
Copyright 2010-2025 The Apache Software Foundation
Licensed under the Apache License, Version 2.0
See: https://github.com/apache/avro
EOF

cat > tests/data/apache-avro/NOTICE << 'EOF'
Apache Avro
Copyright 2009-2025 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
EOF

cat > tests/data/fastavro/LICENSE << 'EOF'
Test data files are from the fastavro project.
Copyright (c) 2011 Miki Tebeka
Licensed under the MIT License
See: https://github.com/fastavro/fastavro
EOF
```

### Step 2: Create Test Data Generation Script

Create `scripts/generate_test_data.py`:

```python
#!/usr/bin/env python3
"""
Generate additional test data files for Jetliner e2e tests.

This script uses the Apache Avro Python library to generate test files
that Jetliner will read to validate interoperability.
"""

import avro.datafile
import avro.io
import avro.schema
import random
import os

def ensure_dir(path):
    os.makedirs(path, exist_ok=True)

def generate_interop_files():
    """Generate interop test files with all Avro types."""
    print("Generating interop test files...")

    # Load schema
    with open('tests/data/apache-avro/interop.avsc') as f:
        schema = avro.schema.parse(f.read())

    # Generate data
    def make_record():
        return {
            "intField": random.randint(-2147483648, 2147483647),
            "longField": random.randint(-9223372036854775808, 9223372036854775807),
            "stringField": f"string_{random.randint(0, 1000)}",
            "boolField": random.choice([True, False]),
            "floatField": random.uniform(-1e6, 1e6),
            "doubleField": random.uniform(-1e12, 1e12),
            "bytesField": bytes([random.randint(0, 255) for _ in range(16)]),
            "nullField": None,
            "arrayField": [random.uniform(0, 100) for _ in range(5)],
            "mapField": {"k1": {"label": "l1"}, "k2": {"label": "l2"}},
            "unionField": random.choice([True, 3.14, [b"b1", b"b2"]]),
            "enumField": random.choice(["A", "B", "C"]),
            "fixedField": bytes([random.randint(0, 255) for _ in range(16)]),
            "recordField": {
                "label": "root",
                "children": [
                    {"label": "c1", "children": []},
                    {"label": "c2", "children": []}
                ]
            }
        }

    # Generate with different codecs
    ensure_dir('tests/data/generated')
    for codec in ["null", "deflate", "snappy"]:  # Skip zstd if not available
        filename = f"tests/data/generated/interop-{codec}.avro"
        print(f"  Writing {filename}")
        with open(filename, 'wb') as f:
            writer = avro.datafile.DataFileWriter(
                f, avro.io.DatumWriter(), schema, codec=codec
            )
            for i in range(100):
                writer.append(make_record())
            writer.close()

def generate_large_multiblock_file():
    """Generate large file with multiple blocks."""
    print("Generating large multiblock file...")

    schema = avro.schema.parse('''{
        "type": "record",
        "name": "Test",
        "fields": [
            {"name": "id", "type": "long"},
            {"name": "data", "type": "string"}
        ]
    }''')

    filename = "tests/data/generated/large-multiblock.avro"
    print(f"  Writing {filename} (10,000 records)")
    with open(filename, 'wb') as f:
        writer = avro.datafile.DataFileWriter(
            f, avro.io.DatumWriter(), schema
        )
        for i in range(10000):
            writer.append({"id": i, "data": f"record_{i}"})
        writer.close()

def generate_edge_case_files():
    """Generate edge case test files."""
    print("Generating edge case files...")

    # Empty file (just header, no blocks)
    schema = avro.schema.parse('{"type": "record", "name": "Empty", "fields": []}')
    filename = "tests/data/generated/empty-records.avro"
    print(f"  Writing {filename}")
    with open(filename, 'wb') as f:
        writer = avro.datafile.DataFileWriter(
            f, avro.io.DatumWriter(), schema
        )
        writer.close()

    # Single record
    schema = avro.schema.parse('{"type": "string"}')
    filename = "tests/data/generated/single-record.avro"
    print(f"  Writing {filename}")
    with open(filename, 'wb') as f:
        writer = avro.datafile.DataFileWriter(
            f, avro.io.DatumWriter(), schema
        )
        writer.append("single record")
        writer.close()

if __name__ == "__main__":
    print("Generating test data files for Jetliner...")
    print()
    generate_interop_files()
    generate_large_multiblock_file()
    generate_edge_case_files()
    print()
    print("Done! Test files created in tests/data/generated/")
```

Make executable and run:
```bash
chmod +x scripts/generate_test_data.py
python3 scripts/generate_test_data.py
```

### Step 3: Create Rust E2E Test File

Create `tests/e2e_real_files_tests.rs`:

```rust
//! End-to-end tests using real Avro files from other implementations
//!
//! These tests validate Jetliner's ability to read files created by:
//! - Apache Avro (Java, Python, C++)
//! - fastavro (Python)
//! - Other Avro implementations
//!
//! Test data sources:
//! - tests/data/apache-avro/: Apache License 2.0
//! - tests/data/fastavro/: MIT License
//! - tests/data/generated/: Generated by scripts/generate_test_data.py

use jetliner::{LocalSource, AvroStreamReader, ReaderConfig, ErrorMode};
use std::path::PathBuf;

fn test_data_path(path: &str) -> PathBuf {
    PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("tests")
        .join("data")
        .join(path)
}

// Phase 1: Basic Interoperability Tests

#[tokio::test]
async fn test_read_weather_uncompressed() {
    let path = test_data_path("apache-avro/weather.avro");
    let source = LocalSource::new(path).await.unwrap();

    let config = ReaderConfig {
        error_mode: ErrorMode::Strict,
        batch_size: 1000,
        ..Default::default()
    };

    let mut reader = AvroStreamReader::new(source, config).await.unwrap();

    let mut record_count = 0;
    while let Some(batch) = reader.next_batch().await.unwrap() {
        record_count += batch.height();

        // Validate schema
        assert!(batch.column("station").is_ok());
        assert!(batch.column("time").is_ok());
        assert!(batch.column("temp").is_ok());
    }

    assert!(record_count > 0, "Should read at least one record");
}

#[tokio::test]
async fn test_read_weather_deflate() {
    let path = test_data_path("apache-avro/weather-deflate.avro");
    let source = LocalSource::new(path).await.unwrap();
    let config = ReaderConfig::default();

    let mut reader = AvroStreamReader::new(source, config).await.unwrap();

    let mut record_count = 0;
    while let Some(batch) = reader.next_batch().await.unwrap() {
        record_count += batch.height();
    }

    assert!(record_count > 0);
}

#[tokio::test]
async fn test_read_weather_snappy() {
    let path = test_data_path("apache-avro/weather-snappy.avro");
    let source = LocalSource::new(path).await.unwrap();
    let config = ReaderConfig::default();

    let mut reader = AvroStreamReader::new(source, config).await.unwrap();

    let mut record_count = 0;
    while let Some(batch) = reader.next_batch().await.unwrap() {
        record_count += batch.height();
    }

    assert!(record_count > 0);
}

#[tokio::test]
async fn test_read_weather_zstd() {
    let path = test_data_path("apache-avro/weather-zstd.avro");
    let source = LocalSource::new(path).await.unwrap();
    let config = ReaderConfig::default();

    let mut reader = AvroStreamReader::new(source, config).await.unwrap();

    let mut record_count = 0;
    while let Some(batch) = reader.next_batch().await.unwrap() {
        record_count += batch.height();
    }

    assert!(record_count > 0);
}

#[tokio::test]
async fn test_all_codecs_produce_same_data() {
    // Read all weather variants
    let variants = vec![
        "weather.avro",
        "weather-deflate.avro",
        "weather-snappy.avro",
        "weather-zstd.avro"
    ];

    let mut all_counts = vec![];

    for variant in variants {
        let path = test_data_path(&format!("apache-avro/{}", variant));
        let source = LocalSource::new(path).await.unwrap();
        let mut reader = AvroStreamReader::new(source, ReaderConfig::default())
            .await
            .unwrap();

        let mut count = 0;
        while let Some(batch) = reader.next_batch().await.unwrap() {
            count += batch.height();
        }
        all_counts.push(count);
    }

    // All should have same record count
    assert!(all_counts.iter().all(|&c| c == all_counts[0]));
}

// More tests following the plan structure...
// - Phase 2: Comprehensive type coverage
// - Phase 3: Block handling
// - Phase 4: Error handling
// - Phase 5: Schema evolution
```

### Step 4: Create Python E2E Tests

Create `python/tests/test_e2e_files.py` following the Phase 6 structure above.

### Step 5: Add Test Runner Commands

Add to `pyproject.toml`:

```toml
[tool.poe.tasks]
test-e2e-rust = "cargo test --test e2e_real_files_tests --all-features"
test-e2e-python = "pytest python/tests/test_e2e_files.py -v"
test-e2e-all = ["test-e2e-rust", "test-e2e-python"]
generate-test-data = "python3 scripts/generate_test_data.py"
```

Usage:
```bash
# Generate test data first
poe generate-test-data

# Run Rust e2e tests
poe test-e2e-rust

# Run Python e2e tests
poe test-e2e-python

# Run all e2e tests
poe test-e2e-all
```

### Step 6: CI Integration

Add to GitHub Actions workflow:

```yaml
- name: Generate test data
  run: |
    pip install avro-python3
    python scripts/generate_test_data.py

- name: Run E2E tests
  run: |
    cargo test --test e2e_real_files_tests --all-features
    pytest python/tests/test_e2e_files.py
```

## Expected Outcomes

After implementing this test plan, Jetliner will have:

1. **Proven Interoperability**: Validated ability to read files from all major Avro implementations
2. **Comprehensive Type Coverage**: All Avro types tested with real-world files
3. **Codec Validation**: All compression codecs tested against reference implementations
4. **Edge Case Handling**: Robust handling of edge cases discovered by other implementations
5. **Performance Baseline**: Benchmarks comparing against fastavro and apache-avro
6. **Regression Prevention**: Real test files prevent spec compliance regressions
7. **User Confidence**: Users can trust Jetliner will read any valid Avro file

## Maintenance Plan

### Keeping Tests Up to Date

**Quarterly Review**:
- Check for new test files in Apache Avro repository
- Review Avro specification updates
- Update interop schema if needed
- Check for new codecs or logical types

**When Issues Arise**:
- If a user reports a file that doesn't read correctly, add it to test suite
- If new edge cases are discovered, generate test files
- If other libraries add new tests, evaluate adding similar tests

### Test Data Management

**Size Considerations**:
- Current test files: ~100KB total
- Generated files: ~500KB-1MB
- Keep test files small and focused
- For large file tests, generate programmatically rather than committing

**Version Control**:
- Commit small binary test files (<100KB) directly to repo
- Generate large files in CI/tests
- Document source and purpose of each test file in README

## Next Steps

1. ✅ **Complete this research** (DONE)
2. ⏭️ **Create license attribution files** for test data
3. ⏭️ **Implement test data generation script**
4. ⏭️ **Create Rust e2e test file** (start with Phase 1)
5. ⏭️ **Create Python e2e test file** (start with basic tests)
6. ⏭️ **Run tests and fix any issues found**
7. ⏭️ **Expand to Phase 2-5** (iterate based on findings)
8. ⏭️ **Add performance benchmarks** (Phase 7)
9. ⏭️ **Document results** in main README

## Conclusion

This e2e test plan provides a comprehensive roadmap for validating Jetliner's compatibility with the Avro ecosystem. By testing against real files from multiple implementations, we ensure that Jetliner can reliably read any valid Avro file users encounter in production.

The phased approach allows incremental implementation, starting with basic interoperability and expanding to comprehensive coverage. All test data is properly licensed and attributed, and the test suite will provide confidence for both developers and users.

---

**Research completed**: 2026-01-08
**Test files location**: `tests/data/`
**Implementation priority**: High (Task 17 in tasks.md)
