# Verification Architecture

## Introduction

This document describes the verification workflow and CI pipeline for the Pinecone Python SDK, including testing strategies, continuous integration, and quality gates.

## Verification Philosophy

### Core Principles

1. **Test at all layers** - Public API, protocols, and implementations
2. **Fast feedback** - Catch issues early in development
3. **Comprehensive coverage** - Unit, integration, and end-to-end tests
4. **Automated verification** - CI pipeline enforces quality gates
5. **Type safety** - Static type checking with mypy

### Quality Pyramid

```mermaid
graph TB
    subgraph TestPyramid["Testing Pyramid"]
        E2E[End-to-End Tests<br/>Few, slow, high confidence]
        INT[Integration Tests<br/>Some, medium speed]
        UNIT[Unit Tests<br/>Many, fast, focused]
    end

    E2E --> INT --> UNIT

    style E2E fill:#ffebee
    style INT fill:#fff3e0
    style UNIT fill:#e8f5e9
```

## Testing Architecture

### Test Organization

```
tests/
├── unit/                       # Fast, isolated tests
│   ├── test_client.py
│   ├── test_index.py
│   ├── test_data.py
│   ├── test_models.py
│   └── test_utils.py
├── integration/                # Tests with external dependencies
│   ├── test_api_integration.py
│   ├── test_auth.py
│   └── test_error_handling.py
├── e2e/                        # Full workflow tests
│   ├── test_complete_workflow.py
│   └── test_real_api.py
├── protocols/                  # Protocol compliance tests
│   ├── test_client_protocol.py
│   ├── test_index_protocol.py
│   └── test_data_protocol.py
└── conftest.py                # Shared fixtures
```

### Test Layer Architecture

```mermaid
graph TB
    subgraph Tests["Test Layers"]
        PROT[Protocol Tests]
        UNIT[Unit Tests]
        INT[Integration Tests]
        E2E[End-to-End Tests]
    end

    subgraph SDK["SDK Layers"]
        PUBLIC[Public API]
        PROTOCOL[Protocols]
        IMPL[Implementation]
    end

    subgraph External["External"]
        MOCK[Mock API]
        REAL[Real API]
    end

    PROT --> PROTOCOL
    UNIT --> IMPL
    INT --> PUBLIC
    INT --> MOCK
    E2E --> PUBLIC
    E2E --> REAL

    style Tests fill:#e3f2fd
    style SDK fill:#f3e5f5
    style External fill:#fff3e0
```

## Test Types

### Unit Tests

**Purpose:** Test individual components in isolation

**Characteristics:**
- Fast (< 10ms each)
- No external dependencies
- Use mocks for dependencies
- High code coverage

**Example:**

```python
import pytest
from pinecone._internal.client import PineconeClient
from pinecone._internal.http.client import HTTPClient

def test_list_indexes_success(mocker):
    """Test list_indexes returns correct data."""
    # Mock HTTP client
    mock_http = mocker.Mock(spec=HTTPClient)
    mock_http.get.return_value = {
        "indexes": [
            {"name": "index-1"},
            {"name": "index-2"},
        ]
    }

    # Create client with mock
    client = PineconeClient(
        api_key="test-key",
        http_client=mock_http,
    )

    # Test
    indexes = client.list_indexes()

    # Verify
    assert indexes == ["index-1", "index-2"]
    mock_http.get.assert_called_once_with("/indexes")

def test_create_index_validation():
    """Test create_index validates parameters."""
    client = PineconeClient(api_key="test-key")

    with pytest.raises(ValidationError, match="dimension must be positive"):
        client.create_index(
            name="test-index",
            dimension=-1,  # Invalid
        )
```

### Protocol Tests

**Purpose:** Verify implementations satisfy protocol contracts

**Characteristics:**
- Test protocol compliance
- Verify runtime checkability
- Check method signatures
- Test type compatibility

**Example:**

```python
import pytest
from pinecone import IPineconeClient, IIndexClient
from pinecone._internal.client import PineconeClient
from pinecone._internal.index import IndexClient

def test_pinecone_client_implements_protocol():
    """Verify PineconeClient satisfies IPineconeClient."""
    client = PineconeClient(api_key="test-key")

    # Runtime check
    assert isinstance(client, IPineconeClient)

    # Verify methods exist
    assert callable(getattr(client, "list_indexes", None))
    assert callable(getattr(client, "create_index", None))
    assert callable(getattr(client, "index", None))

def test_index_client_implements_protocol():
    """Verify IndexClient satisfies IIndexClient."""
    client = IndexClient(
        name="test-index",
        host="https://test.pinecone.io",
        api_key="test-key",
    )

    assert isinstance(client, IIndexClient)
    assert callable(getattr(client, "upsert", None))
    assert callable(getattr(client, "query", None))
    assert callable(getattr(client, "fetch", None))
    assert callable(getattr(client, "delete", None))

def test_protocol_type_compatibility():
    """Test protocol types work with type checkers."""
    def accepts_client(client: IPineconeClient) -> list[str]:
        return client.list_indexes()

    # Should work with real implementation
    client = PineconeClient(api_key="test-key")
    # This test verifies mypy is happy
    result = accepts_client(client)
```

### Integration Tests

**Purpose:** Test interaction with external systems

**Characteristics:**
- Use mock HTTP servers
- Test error scenarios
- Verify serialization
- Test retry logic

**Example:**

```python
import pytest
import responses
from pinecone import PineconeClient

@responses.activate
def test_list_indexes_with_mock_api():
    """Test list_indexes with mocked HTTP responses."""
    # Mock API response
    responses.add(
        responses.GET,
        "https://api.pinecone.io/indexes",
        json={"indexes": [{"name": "test-index"}]},
        status=200,
    )

    # Create client
    client = PineconeClient(api_key="test-key")

    # Call API
    indexes = client.list_indexes()

    # Verify
    assert indexes == ["test-index"]
    assert len(responses.calls) == 1

@responses.activate
def test_error_handling():
    """Test error handling with API errors."""
    # Mock error response
    responses.add(
        responses.POST,
        "https://api.pinecone.io/indexes",
        json={"error": "Invalid dimension"},
        status=400,
    )

    client = PineconeClient(api_key="test-key")

    with pytest.raises(PineconeApiException, match="Invalid dimension"):
        client.create_index(name="test", dimension=-1)
```

### End-to-End Tests

**Purpose:** Test complete workflows against real API

**Characteristics:**
- Slow (seconds to minutes)
- Use real Pinecone API
- Test full user workflows
- Run in CI but not locally by default

**Example:**

```python
import pytest
from pinecone import PineconeClient

@pytest.mark.e2e
@pytest.mark.skip(reason="Requires real API key")
def test_complete_workflow():
    """Test complete index lifecycle."""
    client = PineconeClient(api_key=os.getenv("PINECONE_API_KEY"))

    # Create index
    index_name = f"test-{uuid.uuid4()}"
    client.create_index(
        name=index_name,
        dimension=128,
        metric="cosine",
    )

    try:
        # Wait for index to be ready
        wait_for_index_ready(client, index_name)

        # Get index client
        index = client.index(index_name)

        # Upsert vectors
        vectors = [
            Vector(id="vec1", values=[0.1] * 128),
            Vector(id="vec2", values=[0.2] * 128),
        ]
        response = index.upsert(vectors=vectors)
        assert response.upserted_count == 2

        # Query
        results = index.query(
            vector=[0.1] * 128,
            top_k=10,
        )
        assert len(results.matches) > 0

        # Fetch
        fetched = index.fetch(ids=["vec1", "vec2"])
        assert len(fetched.vectors) == 2

        # Delete
        index.delete(ids=["vec1"])

    finally:
        # Cleanup
        client.delete_index(index_name)
```

## CI Pipeline Architecture

### Pipeline Overview

```mermaid
graph LR
    subgraph Trigger["Trigger Events"]
        PR[Pull Request]
        PUSH[Push to main]
        SCHEDULE[Scheduled]
    end

    subgraph Pipeline["CI Pipeline"]
        LINT[Lint & Format]
        TYPE[Type Check]
        UNIT[Unit Tests]
        INT[Integration Tests]
        COV[Coverage Check]
        E2E[E2E Tests]
        BUILD[Build Package]
        PUBLISH[Publish]
    end

    PR --> LINT
    PR --> TYPE
    PR --> UNIT
    PR --> INT
    PR --> COV

    PUSH --> E2E
    PUSH --> BUILD

    SCHEDULE --> E2E

    BUILD --> PUBLISH

    style Trigger fill:#e3f2fd
    style Pipeline fill:#e8f5e9
```

### Workflow Stages

```mermaid
graph TB
    START[Git Push / PR]

    subgraph PreChecks["Pre-Checks (Parallel)"]
        LINT[Linting<br/>ruff, black]
        FMT[Format Check<br/>black --check]
        SPELL[Spell Check<br/>codespell]
    end

    subgraph TypeCheck["Type Checking"]
        MYPY[mypy --strict]
        PYRIGHT[pyright]
    end

    subgraph Testing["Testing (Parallel)"]
        U1[Unit Tests<br/>Python 3.9]
        U2[Unit Tests<br/>Python 3.10]
        U3[Unit Tests<br/>Python 3.11]
        U4[Unit Tests<br/>Python 3.12]
        PROT[Protocol Tests]
        INT[Integration Tests]
    end

    subgraph Coverage["Coverage Analysis"]
        COV[Coverage Report]
        THRESH[Threshold Check<br/>>= 90%]
    end

    subgraph E2E["E2E (main only)"]
        E2E_TEST[E2E Tests<br/>Real API]
    end

    subgraph Build["Build & Publish"]
        BUILD[Build Package]
        VERIFY[Verify Package]
        PUB[Publish to PyPI]
    end

    START --> PreChecks
    PreChecks --> TypeCheck
    TypeCheck --> Testing
    Testing --> Coverage
    Coverage --> E2E
    E2E --> Build

    style PreChecks fill:#e3f2fd
    style TypeCheck fill:#fff3e0
    style Testing fill:#e8f5e9
    style Coverage fill:#f3e5f5
    style E2E fill:#fce4ec
    style Build fill:#f3e5f5
```

### GitHub Actions Configuration

```yaml
# .github/workflows/ci.yml
name: CI

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 0 * * *'  # Daily

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install ruff black codespell
      - name: Lint with ruff
        run: ruff check .
      - name: Check formatting
        run: black --check .
      - name: Spell check
        run: codespell

  type-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install mypy pyright
          pip install -e .
      - name: Type check with mypy
        run: mypy pinecone
      - name: Type check with pyright
        run: pyright

  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python-version: ['3.9', '3.10', '3.11', '3.12']
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          pip install -e ".[dev]"
      - name: Run unit tests
        run: pytest tests/unit -v
      - name: Run protocol tests
        run: pytest tests/protocols -v
      - name: Run integration tests
        run: pytest tests/integration -v

  coverage:
    runs-on: ubuntu-latest
    needs: [test]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install -e ".[dev]"
      - name: Run tests with coverage
        run: |
          pytest --cov=pinecone --cov-report=xml --cov-report=html
      - name: Check coverage threshold
        run: |
          coverage report --fail-under=90
      - name: Upload coverage
        uses: codecov/codecov-action@v3

  e2e:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' || github.event_name == 'schedule'
    needs: [lint, type-check, test, coverage]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install -e ".[dev]"
      - name: Run E2E tests
        env:
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
        run: pytest tests/e2e -v --maxfail=1

  build:
    runs-on: ubuntu-latest
    needs: [e2e]
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Build package
        run: |
          pip install build
          python -m build
      - name: Verify package
        run: |
          pip install dist/*.whl
          python -c "import pinecone; print(pinecone.__version__)"
      - name: Upload artifacts
        uses: actions/upload-artifact@v3
        with:
          name: dist
          path: dist/
```

## Quality Gates

### Required Checks

```mermaid
graph TB
    subgraph Gates["Quality Gates"]
        G1[Linting Pass]
        G2[Type Check Pass]
        G3[All Tests Pass]
        G4[Coverage >= 90%]
        G5[No Security Issues]
        G6[Documentation Updated]
    end

    G1 --> MERGE[Merge to main]
    G2 --> MERGE
    G3 --> MERGE
    G4 --> MERGE
    G5 --> MERGE
    G6 --> MERGE

    style Gates fill:#e8f5e9
    style MERGE fill:#4caf50
```

### Coverage Requirements

- **Overall coverage:** >= 90%
- **New code coverage:** >= 95%
- **Protocol implementations:** 100%
- **Public API:** >= 95%
- **Internal utilities:** >= 80%

### Type Checking Requirements

- **mypy:** No errors in strict mode
- **pyright:** No errors in strict mode
- **All public APIs:** Fully typed
- **All protocols:** Complete type annotations

## Local Development Workflow

### Pre-commit Hooks

```yaml
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/psf/black
    rev: 23.3.0
    hooks:
      - id: black
        language_version: python3.11

  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.0.265
    hooks:
      - id: ruff
        args: [--fix, --exit-non-zero-on-fix]

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.3.0
    hooks:
      - id: mypy
        additional_dependencies: [types-all]

  - repo: https://github.com/codespell-project/codespell
    rev: v2.2.4
    hooks:
      - id: codespell
```

### Development Flow

```mermaid
graph TB
    START[Start Development]
    BRANCH[Create Branch]
    CODE[Write Code]
    TEST[Run Tests Locally]
    COMMIT[Commit]
    HOOK[Pre-commit Hooks]
    PUSH[Push to Remote]
    CI[CI Pipeline]
    REVIEW[Code Review]
    MERGE[Merge]

    START --> BRANCH
    BRANCH --> CODE
    CODE --> TEST
    TEST -->|Pass| COMMIT
    TEST -->|Fail| CODE
    COMMIT --> HOOK
    HOOK -->|Pass| PUSH
    HOOK -->|Fail| CODE
    PUSH --> CI
    CI -->|Pass| REVIEW
    CI -->|Fail| CODE
    REVIEW -->|Approved| MERGE
    REVIEW -->|Changes Requested| CODE

    style START fill:#e3f2fd
    style MERGE fill:#4caf50
```

### Running Tests Locally

```bash
# Install development dependencies
pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

# Run all checks
make check  # Runs lint, type-check, test

# Run specific test suites
pytest tests/unit                  # Fast unit tests
pytest tests/protocols             # Protocol compliance
pytest tests/integration           # Integration tests
pytest tests/e2e -m e2e           # E2E tests (requires API key)

# Run with coverage
pytest --cov=pinecone --cov-report=html
open htmlcov/index.html

# Type checking
mypy pinecone
pyright

# Linting
ruff check .
black --check .

# Format code
black .
ruff check --fix .
```

## Performance Testing

### Benchmark Tests

```python
import pytest
from pinecone import PineconeClient

@pytest.mark.benchmark
def test_query_performance(benchmark):
    """Benchmark query performance."""
    client = PineconeClient(api_key="test-key")
    index = client.index("benchmark-index")

    result = benchmark(
        index.query,
        vector=[0.1] * 128,
        top_k=10,
    )

    # Verify performance
    assert benchmark.stats.mean < 0.1  # < 100ms average
```

### Load Testing

```python
import concurrent.futures
from pinecone import PineconeClient

def test_concurrent_queries():
    """Test handling multiple concurrent queries."""
    client = PineconeClient(api_key="test-key")
    index = client.index("load-test-index")

    def query():
        return index.query(vector=[0.1] * 128, top_k=10)

    # Run 100 concurrent queries
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        futures = [executor.submit(query) for _ in range(100)]
        results = [f.result() for f in futures]

    assert len(results) == 100
    assert all(len(r.matches) > 0 for r in results)
```

## Monitoring and Alerting

### CI Metrics

- Build success rate
- Test duration trends
- Coverage trends
- Flaky test detection

### Alerts

- Failed builds on main
- Coverage drops below threshold
- E2E test failures
- Performance regressions

## Related Documentation

- [Overview](./overview.md) - System architecture
- [API Structure](./api-structure.md) - API organization
- [Specifications](./specifications.md) - Protocol specifications
- [Testing Conventions](../conventions/testing.md) - Testing guidelines
- [Performance Conventions](../conventions/performance.md) - Performance testing
