# Task 25.9: S3 Read Reduction Benchmark

## Summary

Added benchmark `bench_read_buffer_chunk_size` to demonstrate I/O efficiency improvements from read buffering, particularly important for S3 sources.

## Key Findings

### Benchmark Results (weather-large.avro, 2.55 MB, 10K records)

| Chunk Size           | Time   | Throughput | I/O Operations (approx) |
| -------------------- | ------ | ---------- | ----------------------- |
| 16KB                 | 5.1ms  | 497 MiB/s  | ~160 reads              |
| 64KB (local default) | 2.3ms  | 1.08 GiB/s | ~40 reads               |
| 1MB                  | 1.26ms | 1.97 GiB/s | ~3 reads                |
| 4MB (S3 default)     | 1.22ms | 2.03 GiB/s | ~1 read                 |

### S3 Latency Impact

For S3 sources, each `read_range` call is an HTTP GET request with typical latency of 50-100ms.

**Example: File with 1000 small blocks (1KB each = 1MB total)**

| Configuration   | HTTP Requests | Latency Overhead |
| --------------- | ------------- | ---------------- |
| No buffering    | ~1000         | 50-100 seconds   |
| 4MB buffer      | ~1            | 50-100 ms        |
| **Improvement** | **1000x**     | **1000x**        |

### Design Decisions

1. **Local default (64KB)**: Balances memory usage with I/O efficiency for local files where latency is negligible.

2. **S3 default (4MB)**: Optimized for network latency - larger chunks amortize HTTP request overhead across many blocks.

3. **Prefetch threshold (0.5 for S3)**: Triggers prefetch when buffer is 50% consumed, hiding network latency by overlapping I/O with processing.

## Implementation Notes

- `ReadBufferConfig::LOCAL_DEFAULT`: 64KB chunk, 0.0 prefetch threshold
- `ReadBufferConfig::S3_DEFAULT`: 4MB chunk, 0.5 prefetch threshold
- Auto-detection in Python API: `read_chunk_size=None` selects appropriate default based on source type (S3 URI vs local path)

## References

- Task 25: BlockReader read buffering optimization
- Requirements 3.8-3.13: Read buffering requirements
- Property 16: BlockReader I/O Efficiency (tests/property_tests.rs)
