# v0.9.10 Pre-Stat Optimization for Benchmarking

**Date**: October 19, 2025  
**Status**: Planning  
**Priority**: HIGH - Critical for sai3-bench performance

## Problem Statement

### Current Workflow (Sequential Stat Overhead)

```
For each object to download:
  1. stat(uri) to get size         ← LATENCY (5-50ms per object)
  2. if size >= threshold:
       download_with_ranges(uri, size)
  3. else:
       simple_download(uri)
```

**Performance Impact**:
- **Sequential bottleneck**: Must stat before download
- **Latency multiplication**: 1000 objects × 20ms stat = **20 seconds wasted**
- **Underutilized bandwidth**: Network idle during stat operations
- **Cascading delays**: Can't start optimized download until stat completes

### Benchmarking Workload Characteristics

**sai3-bench typical usage**:
- Download **1000-10000 objects** in a benchmark run
- Objects often have similar sizes (training data, prepared datasets)
- Object URIs known upfront (from manifest/list)
- Need maximum throughput (minimize overhead)

**Current overhead**:
```
1000 objects × 20ms stat/object = 20 seconds of pure overhead
At 5 GB/s download speed, that's 100 GB of lost throughput time!
```

## Proposed Solution: Async Pre-Stat with Size Cache

### Architecture Overview

```
┌────────────────────────────────────────────────────────────────┐
│  Phase 1: PRE-STAT (Async, Concurrent)                         │
├────────────────────────────────────────────────────────────────┤
│  sai3-bench calls: pre_stat_objects(uri_list)                  │
│                                                                 │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐          │
│  │ stat #1 │  │ stat #2 │  │ stat #3 │  │ stat #N │          │
│  │ 20ms    │  │ 20ms    │  │ 20ms    │  │ 20ms    │          │
│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘          │
│       └────────────┴────────────┴────────────┘                │
│                        ↓                                        │
│           ObjectSizeCache (HashMap)                            │
│           uri → (size, timestamp)                              │
└────────────────────────────────────────────────────────────────┘
                         ↓ cached sizes available
┌────────────────────────────────────────────────────────────────┐
│  Phase 2: DOWNLOAD (Using Cached Sizes)                        │
├────────────────────────────────────────────────────────────────┤
│  For each object:                                               │
│    if cache.has(uri):                                          │
│      size = cache.get(uri)  ← NO STAT LATENCY!                │
│      download_with_ranges(uri, size)                           │
│    else:                                                        │
│      fallback: stat + download (original behavior)             │
└────────────────────────────────────────────────────────────────┘
```

### Key Design Principles

1. **Non-blocking**: Pre-stat runs concurrently, doesn't block downloads
2. **Optional**: If cache miss, fallback to original stat-then-download
3. **Expiration**: Cached sizes expire after configurable TTL (60s default)
4. **Concurrency control**: Limit concurrent stats (default 100)
5. **Thread-safe**: Multiple download threads can query cache simultaneously

## Implementation Design

### Component 1: ObjectSizeCache

**File**: `src/object_size_cache.rs` (NEW)

```rust
use std::collections::HashMap;
use std::sync::Arc;
use tokio::sync::RwLock;
use std::time::{Duration, Instant};

/// Cached object size with timestamp
#[derive(Debug, Clone)]
struct CachedSize {
    size: u64,
    cached_at: Instant,
}

/// Thread-safe cache of object sizes from stat/head operations
/// 
/// Enables pre-statting objects before download to eliminate
/// per-object stat latency in high-throughput workloads.
#[derive(Debug)]
pub struct ObjectSizeCache {
    cache: Arc<RwLock<HashMap<String, CachedSize>>>,
    ttl: Duration,
}

impl ObjectSizeCache {
    /// Create a new size cache with the given TTL
    pub fn new(ttl: Duration) -> Self {
        Self {
            cache: Arc::new(RwLock::new(HashMap::new())),
            ttl,
        }
    }

    /// Get cached size for URI (returns None if not cached or expired)
    pub async fn get(&self, uri: &str) -> Option<u64> {
        let cache = self.cache.read().await;
        if let Some(entry) = cache.get(uri) {
            if entry.cached_at.elapsed() < self.ttl {
                return Some(entry.size);
            }
        }
        None
    }

    /// Store size for URI
    pub async fn put(&self, uri: String, size: u64) {
        let mut cache = self.cache.write().await;
        cache.insert(uri, CachedSize {
            size,
            cached_at: Instant::now(),
        });
    }

    /// Clear expired entries
    pub async fn clear_expired(&self) {
        let mut cache = self.cache.write().await;
        cache.retain(|_, entry| entry.cached_at.elapsed() < self.ttl);
    }

    /// Get cache statistics
    pub async fn stats(&self) -> CacheStats {
        let cache = self.cache.read().await;
        CacheStats {
            total_entries: cache.len(),
            expired_entries: cache.values()
                .filter(|e| e.cached_at.elapsed() >= self.ttl)
                .count(),
        }
    }
}

#[derive(Debug)]
pub struct CacheStats {
    pub total_entries: usize,
    pub expired_entries: usize,
}
```

### Component 2: Pre-Stat API

**File**: `src/object_store.rs` (ADD TO TRAIT)

```rust
#[async_trait]
pub trait ObjectStore: Send + Sync {
    // ... existing methods ...

    /// Pre-stat multiple objects concurrently to populate size cache
    /// 
    /// This is a performance optimization for workloads where object URIs
    /// are known upfront (e.g., benchmark tools, batch processing).
    /// 
    /// # Arguments
    /// * `uris` - List of object URIs to stat
    /// * `max_concurrent` - Maximum concurrent stat operations (default: 100)
    /// 
    /// # Returns
    /// Map of URI → size for successfully statted objects
    /// 
    /// # Example
    /// 
    /// ```no_run
    /// // In sai3-bench: Pre-stat all benchmark objects
    /// let uris = vec![
    ///     "s3://bucket/object1.dat",
    ///     "s3://bucket/object2.dat",
    ///     // ... 1000 more objects
    /// ];
    /// 
    /// let size_map = store.pre_stat_objects(&uris, 100).await?;
    /// 
    /// // Now download with no stat overhead
    /// for uri in uris {
    ///     let bytes = store.get(uri).await?;  // Uses cached size!
    /// }
    /// ```
    async fn pre_stat_objects(
        &self,
        uris: &[String],
        max_concurrent: usize,
    ) -> Result<HashMap<String, u64>> {
        use futures::stream::{self, StreamExt};
        
        let results: Vec<_> = stream::iter(uris)
            .map(|uri| async move {
                match self.stat(uri).await {
                    Ok(metadata) => Some((uri.clone(), metadata.size)),
                    Err(e) => {
                        tracing::warn!("Pre-stat failed for {}: {}", uri, e);
                        None
                    }
                }
            })
            .buffer_unordered(max_concurrent)
            .collect()
            .await;
        
        Ok(results.into_iter().flatten().collect())
    }

    /// Pre-stat objects and populate internal size cache
    /// 
    /// After calling this, subsequent get() calls will use cached sizes
    /// and skip the per-object stat operation.
    async fn pre_stat_and_cache(&self, uris: &[String], max_concurrent: usize) -> Result<usize>;
}
```

### Component 3: Integration with ObjectStore Backends

**File**: `src/object_store.rs` (MODIFY IMPLEMENTATIONS)

```rust
pub struct S3ObjectStore {
    // ... existing fields ...
    
    /// Size cache for pre-statted objects
    size_cache: Arc<ObjectSizeCache>,
}

impl S3ObjectStore {
    pub fn new(config: ObjectStoreConfig) -> Self {
        // ... existing initialization ...
        
        let cache_ttl = config.size_cache_ttl
            .unwrap_or_else(|| Duration::from_secs(60));
        
        Self {
            // ... existing fields ...
            size_cache: Arc::new(ObjectSizeCache::new(cache_ttl)),
        }
    }
}

#[async_trait]
impl ObjectStore for S3ObjectStore {
    async fn get(&self, uri: &str) -> Result<Bytes> {
        // OPTIMIZATION: Check size cache first
        let object_size = if let Some(cached_size) = self.size_cache.get(uri).await {
            tracing::debug!("Using cached size for {}: {} bytes", uri, cached_size);
            cached_size
        } else {
            // Fallback: stat to get size (original behavior)
            let metadata = self.stat(uri).await?;
            let size = metadata.size;
            
            // Cache for future use
            self.size_cache.put(uri.to_string(), size).await;
            size
        };
        
        // Use RangeEngine if enabled and size >= threshold
        if self.config.enable_range_engine && object_size >= self.config.range_engine.min_split_size {
            return self.get_with_range_engine(uri, object_size).await;
        }
        
        // Simple download for small objects
        self.simple_get(uri).await
    }

    async fn pre_stat_and_cache(&self, uris: &[String], max_concurrent: usize) -> Result<usize> {
        let size_map = self.pre_stat_objects(uris, max_concurrent).await?;
        
        // Populate cache
        for (uri, size) in size_map.iter() {
            self.size_cache.put(uri.clone(), *size).await;
        }
        
        Ok(size_map.len())
    }
}

// Same pattern for AzureObjectStore and GcsObjectStore
```

### Component 4: Configuration

**File**: `src/config.rs` (ADD FIELDS)

```rust
#[derive(Debug, Clone, Deserialize)]
pub struct ObjectStoreConfig {
    // ... existing fields ...
    
    /// Enable size caching for pre-statted objects
    pub enable_size_cache: bool,
    
    /// TTL for cached sizes (seconds)
    pub size_cache_ttl: Option<u64>,
    
    /// Max concurrent pre-stat operations
    pub pre_stat_concurrency: usize,
}

impl Default for ObjectStoreConfig {
    fn default() -> Self {
        Self {
            // ... existing defaults ...
            enable_size_cache: true,
            size_cache_ttl: Some(60),  // 60 seconds
            pre_stat_concurrency: 100,
        }
    }
}
```

## Usage Pattern for sai3-bench

### Before (Current - Sequential Stat Overhead)

```rust
// sai3-bench current code
for uri in object_list {
    let bytes = store.get(uri).await?;  // Stats THEN downloads (sequential)
    // ... process bytes ...
}
```

**Performance**: 1000 objects × 20ms stat = **20 seconds wasted**

### After (With Pre-Stat Optimization)

```rust
// sai3-bench OPTIMIZED code
let store = store_for_uri("s3://...")?;

// PHASE 1: Pre-stat ALL objects concurrently (runs in background)
tracing::info!("Pre-statting {} objects...", object_list.len());
let pre_stat_start = Instant::now();
let cached_count = store.pre_stat_and_cache(&object_list, 100).await?;
tracing::info!(
    "Pre-statted {} objects in {:.2}s (avg {:.2}ms/object)",
    cached_count,
    pre_stat_start.elapsed().as_secs_f64(),
    pre_stat_start.elapsed().as_millis() as f64 / cached_count as f64
);

// PHASE 2: Download with NO stat overhead
for uri in object_list {
    let bytes = store.get(uri).await?;  // Uses cached size, no stat!
    // ... process bytes ...
}
```

**Performance**: 1000 objects pre-statted in ~200ms (100 concurrent), then downloads with **zero stat overhead**

**Time saved**: 20 seconds - 0.2 seconds = **19.8 seconds saved** (99% reduction!)

## Performance Impact Analysis

### Scenario: sai3-bench with 1000 × 64MB Objects

#### Before (Current)

```
Phase: Sequential stat-then-download
- Stat overhead: 1000 × 20ms = 20 seconds
- Download time: (1000 × 64MB) / 5GB/s = 12.8 seconds
- Total time: 32.8 seconds
- Effective throughput: 64GB / 32.8s = 1.95 GB/s

Efficiency: 39% (12.8s useful / 32.8s total)
```

#### After (With Pre-Stat)

```
Phase 1: Pre-stat (concurrent)
- Pre-stat time: 1000 stats / 100 concurrent = 10 batches × 20ms = 0.2 seconds

Phase 2: Download (no stat overhead)
- Download time: (1000 × 64MB) / 5GB/s = 12.8 seconds
- Total time: 13.0 seconds
- Effective throughput: 64GB / 13.0s = 4.92 GB/s

Efficiency: 98% (12.8s useful / 13.0s total)

IMPROVEMENT: 2.5x faster (32.8s → 13.0s), 2.5x higher throughput (1.95 → 4.92 GB/s)
```

### Impact Breakdown

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Total time | 32.8s | 13.0s | **2.5x faster** |
| Stat overhead | 20.0s | 0.2s | **99% reduction** |
| Throughput | 1.95 GB/s | 4.92 GB/s | **2.5x higher** |
| Efficiency | 39% | 98% | **59% improvement** |

## Testing Strategy

### Unit Tests

```rust
#[tokio::test]
async fn test_size_cache_basic() {
    let cache = ObjectSizeCache::new(Duration::from_secs(60));
    
    // Store and retrieve
    cache.put("s3://bucket/object".to_string(), 12345).await;
    assert_eq!(cache.get("s3://bucket/object").await, Some(12345));
    
    // Cache miss
    assert_eq!(cache.get("s3://bucket/other").await, None);
}

#[tokio::test]
async fn test_size_cache_expiration() {
    let cache = ObjectSizeCache::new(Duration::from_millis(100));
    
    cache.put("s3://bucket/object".to_string(), 12345).await;
    assert_eq!(cache.get("s3://bucket/object").await, Some(12345));
    
    tokio::time::sleep(Duration::from_millis(150)).await;
    assert_eq!(cache.get("s3://bucket/object").await, None);  // Expired
}

#[tokio::test]
async fn test_pre_stat_concurrent() {
    let store = S3ObjectStore::new(ObjectStoreConfig::default());
    
    let uris: Vec<String> = (0..1000)
        .map(|i| format!("s3://test-bucket/object-{}.dat", i))
        .collect();
    
    let start = Instant::now();
    let count = store.pre_stat_and_cache(&uris, 100).await.unwrap();
    let elapsed = start.elapsed();
    
    assert_eq!(count, 1000);
    assert!(elapsed < Duration::from_secs(5));  // Should be much faster than serial
}
```

### Integration Tests (with sai3-bench)

```bash
# Test 1: Baseline (no pre-stat)
sai3-bench --no-pre-stat --objects 1000 --size 64M

# Test 2: With pre-stat
sai3-bench --pre-stat --objects 1000 --size 64M

# Compare throughput (expect 2-3x improvement)
```

## Implementation Checklist

- [ ] Create `src/object_size_cache.rs` with ObjectSizeCache
- [ ] Add `pre_stat_objects()` to ObjectStore trait
- [ ] Add `pre_stat_and_cache()` to ObjectStore trait
- [ ] Integrate size cache into S3ObjectStore
- [ ] Integrate size cache into AzureObjectStore
- [ ] Integrate size cache into GcsObjectStore
- [ ] Add configuration options (enable_size_cache, ttl, concurrency)
- [ ] Add unit tests for ObjectSizeCache
- [ ] Add integration tests for pre-stat
- [ ] Add benchmarks comparing with/without pre-stat
- [ ] Update sai3-bench to use pre-stat API
- [ ] Document API in README and docs

## Risks & Mitigation

### Risk 1: Stale Size Data

**Risk**: Object size changes between pre-stat and download

**Mitigation**:
- Short TTL (60s default)
- If download fails due to size mismatch, invalidate cache and retry
- Acceptable for benchmarking (objects don't change during benchmark)

### Risk 2: Memory Usage

**Risk**: Caching 10,000 URIs uses too much memory

**Mitigation**:
- Each entry: ~100 bytes (URI string + size + timestamp)
- 10,000 entries: ~1 MB (negligible)
- Auto-expire after TTL
- Optional: Add max cache size limit

### Risk 3: Pre-Stat Overhead

**Risk**: Pre-statting 10,000 objects takes too long

**Mitigation**:
- Controlled concurrency (100 default)
- Optional: Make pre-stat async (don't wait for completion)
- Can start downloads before all stats complete (progressive cache population)

## Future Enhancements

### Enhancement 1: Progressive Pre-Stat

```rust
// Start pre-stat, don't wait for completion
tokio::spawn(async move {
    store.pre_stat_and_cache(&uris, 100).await
});

// Start downloads immediately (cache populates in background)
for uri in uris {
    store.get(uri).await?;  // Uses cache if available, else stats
}
```

### Enhancement 2: Persistent Cache

```rust
// Save cache to disk between runs (for repeated benchmarks)
cache.save_to_file("~/.s3dlio/size_cache.db").await?;
cache.load_from_file("~/.s3dlio/size_cache.db").await?;
```

### Enhancement 3: Smart Cache Warming

```rust
// Automatically pre-stat when listing objects
let objects = store.list("s3://bucket/prefix/", true).await?;
// Implicitly: cache.warm_from_list_metadata(objects);
```

## Conclusion

Pre-stat optimization provides **massive** benefits for benchmarking workloads:

✅ **2.5x faster** benchmark runs  
✅ **99% reduction** in stat overhead  
✅ **2.5x higher** effective throughput  
✅ **Simple API** for sai3-bench integration  
✅ **Zero risk** (fallback to original behavior if cache miss)  

Combined with buffer pool optimization, v0.9.10 will deliver **real, measurable** performance improvements for object storage workloads.
