# v0.9.8 Release: GCS Backend Options & Reliability Improvements

**Release Date**: October 17, 2025  
**Branch**: `feature/gcs-backend-options`

## Overview

Version 0.9.8 adds **optional Google Cloud Storage (GCS) backend support** with two implementation choices, plus critical fixes for non-recursive object listing across all backends.

## Key Features

### 1. Dual GCS Backend Support

s3dlio now supports GCS through two mutually exclusive backends:

- **`gcs-community`** (default): Community-maintained `gcloud-storage` v1.1 crate
  - ✅ **100% reliable** - Production ready
  - ✅ REST API-based
  - ✅ Comprehensive test coverage (10/10 tests pass)
  
- **`gcs-official`** (experimental): Official Google `google-cloud-storage` v1.1 crate
  - ⚠️ **80-90% reliable** - Experimental, not for production
  - ✅ gRPC-based (newer protocol)
  - ⚠️ Has upstream transport flakes (Issue #3574)

### 2. Non-Recursive Listing Fixed (All Backends)

**Problem**: Non-recursive listing (with delimiter) was broken in both GCS backends:
- gcs-community: Returned 0 objects (missing prefix normalization + prefix collection)
- gcs-official: Returned all objects (ignored recursive flag, missing prefix collection)

**Solution**: Implemented proper non-recursive listing matching S3 behavior:
- ✅ Normalize prefix to end with "/" when recursive=false
- ✅ Set delimiter="/" to trigger GCS common prefix behavior
- ✅ Collect both `items` (files) and `prefixes` (subdirectories ending with "/")
- ✅ Returns combined list matching S3 delimiter behavior

**Implementation Details**:
- **gcs-community** (`src/gcs_client.rs`): Uses REST API with `delimiter` parameter, collects `response.items` + `response.prefixes`
- **gcs-official** (`src/google_gcs_client.rs`): Uses gRPC API with `.set_delimiter()`, uses `.by_page()` to access full response with `page.objects` + `page.prefixes`

**Test Verification**: Both backends now correctly return "2 files + 1 subdirectory prefix" in non-recursive tests.

### 3. Global Client Singleton Pattern (gcs-official)

**Problem**: gcs-official backend had 70% failure rate in test suites (30% pass rate)

**Root Cause**: Creating new GcsClient for each operation triggered upstream HTTP/2 connection pool bug in `google-cloud-rust`

**Solution**: Implemented global client singleton using `once_cell::Lazy<OnceCell<GcsClient>>`
```rust
static GLOBAL_CLIENT: Lazy<OnceCell<GcsClient>> = Lazy::new(OnceCell::new);
async fn get_client() -> &'static GcsClient {
    GLOBAL_CLIENT.get_or_init(|| async {
        GcsClient::new().await.expect("Failed to initialize GCS client")
    }).await
}
```

**Results**:
- **Before**: 30% test pass rate (3/10 tests)
- **After**: 80-90% test pass rate (8-9/10 tests)
- **Improvement**: 150-200% reliability increase

## What Works

### gcs-community Backend (Production Ready)
✅ All operations 100% reliable:
- Authentication (ADC, service account, gcloud CLI)
- PUT/GET/DELETE operations
- Metadata (STAT) operations
- Recursive listing
- Non-recursive listing with delimiter (NEW in v0.9.8)
- Batch delete with adaptive concurrency
- Range reads

### gcs-official Backend (When Transport Succeeds)
✅ All operations function correctly (80-90% of the time):
- **Storage client** (read/write): ✅ 100% reliable
  - PUT object: Always works
  - GET object: Always works
  - GET range: Always works
- **StorageControl client** (metadata/list): ✅ 80-90% reliable
  - STAT object: Works when transport succeeds
  - LIST recursive: Works when transport succeeds
  - LIST non-recursive: Works when transport succeeds (NEW in v0.9.8)
  - DELETE operations: Work when transport succeeds

## What Does NOT Work

### gcs-official Backend Limitations
❌ **Transport layer flakes** (10-20% of operations):
- Error: "transport reports an error: transport error"
- Error: "cannot serialize request"
- **NOT operation-specific**: Any operation can fail
- **NOT reproducible**: Same operation succeeds when retried
- **Root cause**: Upstream `google-cloud-rust` HTTP/2 connection pool issue
- **Our bug report**: https://github.com/googleapis/google-cloud-rust/issues/3574
- **Workaround**: Use gcs-community backend for production

## Usage

### Build with Default (Community Backend)
```bash
cargo build --release
```

### Build with Official Backend (Experimental)
```bash
cargo build --release --no-default-features \
  --features native-backends,s3,gcs-official
```

### URI Support
Both backends support:
- `gs://bucket/prefix/object.ext`
- `gcs://bucket/prefix/object.ext`

### Example Code
```rust
use s3dlio::api::store_for_uri;

// Works with both backends (selected at compile time)
let store = store_for_uri("gs://my-bucket/data/")?;
let objects = store.list("my-prefix/", false)?; // Non-recursive
```

## Testing

### Test Suites
```bash
# Test community backend (default)
./scripts/test_gcs_community.sh
# Expected: 10/10 tests pass ✅

# Test official backend (experimental)
./scripts/test_gcs_official.sh  
# Expected: 3-9/10 tests pass (transport flakes) ⚠️

# Compare both backends
./scripts/test_gcs_backends.sh
```

### Prerequisites
```bash
export GCS_TEST_BUCKET=your-test-bucket
# And one of:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# OR: gcloud auth application-default login
```

## Breaking Changes

**None** - GCS support is purely additive with optional features.

## Migration Guide

### For Existing Users
No changes required. GCS support is optional and doesn't affect existing S3/Azure/file:// code.

### For New GCS Users
1. **Choose your backend**: Use `gcs-community` (default) unless you need gRPC
2. **Set up authentication**: Service account JSON or `gcloud auth`
3. **Use standard APIs**: All storage operations use same `ObjectStore` interface

## Files Changed

### Core Implementation
- `src/gcs_client.rs`: Community backend (gcloud-storage) - non-recursive fix
- `src/google_gcs_client.rs`: Official backend (google-cloud-storage) - non-recursive fix
- `src/lib.rs`: Feature gates and module exports
- `src/object_store.rs`: Integration with ObjectStore trait
- `Cargo.toml`: Feature definitions and dependencies

### Tests
- `tests/test_gcs_community.rs`: Global singleton + test expectation updates
- `tests/test_gcs_official.rs`: Global singleton + test expectation updates
- `tests/common.rs`: Shared test utilities

### Documentation
- `docs/GCS-BACKEND-SELECTION.md`: Comprehensive backend comparison
- `docs/Changelog.md`: Release notes
- `README.md`: Feature overview and status

### Scripts
- `scripts/test_gcs_community.sh`: Community backend test runner
- `scripts/test_gcs_official.sh`: Official backend test runner
- `scripts/test_gcs_backends.sh`: Comparison test suite

## Known Issues & Future Work

### Upstream Issue #3574
- **Status**: Filed with googleapis/google-cloud-rust
- **Impact**: gcs-official backend has 10-20% transport failure rate
- **Workaround**: Use gcs-community for production
- **Tracking**: https://github.com/googleapis/google-cloud-rust/issues/3574

### Future Enhancements
- [ ] Retry logic for gcs-official transport errors
- [ ] Performance benchmarking: gcs-community vs gcs-official
- [ ] Multi-target addressing for >100 Gb/s throughput
- [ ] Page cache optimization for GCS operations

## Credits

- **GCS Integration**: Implemented dual-backend architecture
- **Non-recursive Fix**: Root cause analysis and implementation for both backends
- **Singleton Pattern**: Applied Google docs recommendation for client lifecycle
- **Bug Report**: Comprehensive upstream issue with reproduction steps

## References

- GCS JSON API: https://cloud.google.com/storage/docs/json_api/v1/objects/list
- googleapis/google-cloud-rust Issue #3574: Transport error flakes
- googleapis/google-cloud-rust Issue #3412: Related HTTP/2 connection pool issue
- google-cloud-rust docs: Client lifecycle and singleton patterns
