---
title: "Test Implementation: Database Integration Tests with TestContainers"
by: "pydapter-implementer"
created: "2025-05-04"
updated: "2025-05-04"
version: "1.0"
doc_type: TI
output_subdir: tis
description: "Test implementation plan for database integration tests using TestContainers"
---

# Test Implementation Plan: Database Integration Tests with TestContainers

## 1. Overview

### 1.1 Component Under Test

This test implementation plan covers integration tests for the following
database adapters in pydapter:

- PostgreSQL adapter (`PostgresAdapter`)
- MongoDB adapter (`MongoAdapter`)
- Neo4j adapter (`Neo4jAdapter`)
- Qdrant vector database adapter (`QdrantAdapter`)

### 1.2 Test Approach

We will use a combination of:

- Integration tests with real database instances in containers
- TestContainers for database isolation and cleanup
- Pytest fixtures for test setup and teardown
- Parameterized tests for testing multiple adapters with similar patterns

### 1.3 Key Testing Goals

- Verify that adapters can correctly store and retrieve data from real database
  instances
- Test batch operations with multiple records
- Verify error handling for various failure scenarios
- Ensure proper cleanup of test resources

## 2. Test Environment

### 2.1 Test Framework

```
pytest
pytest-asyncio
testcontainers-python
```

### 2.2 Mock Framework

No mocks are needed for these integration tests as we'll be using real database
instances in containers.

### 2.3 Test Database

Each test will use an ephemeral database container:

- PostgreSQL: `postgres:16-alpine`
- MongoDB: `mongo:6.0`
- Neo4j: `neo4j:5.9`
- Qdrant: `qdrant/qdrant:v1.8.1`

## 3. Integration Tests

### 3.1 Test Suite: PostgreSQL Integration

#### 3.1.1 Test Case: Single Record Operations

**Purpose:** Verify that the PostgreSQL adapter can store and retrieve a single
record.

**Setup:**

```python
@pytest.fixture
def postgres_table(pg_url):
    """Create a test table in PostgreSQL."""
    import sqlalchemy as sa
    
    engine = sa.create_engine(pg_url)
    with engine.begin() as conn:
        conn.execute(sa.text("""
            CREATE TABLE IF NOT EXISTS test_table (
                id INTEGER PRIMARY KEY,
                name TEXT,
                value FLOAT
            )
        """))
    
    yield
    
    # Cleanup
    with engine.begin() as conn:
        conn.execute(sa.text("DROP TABLE IF EXISTS test_table"))
```

**Test Implementation:**

```python
def test_postgres_single_record(pg_url, sync_model_factory, postgres_table):
    """Test PostgreSQL adapter with a single record."""
    # Create test instance
    test_model = sync_model_factory(id=42, name="test_postgres", value=12.34)
    
    # Register adapter
    test_model.__class__.register_adapter(PostgresAdapter)
    
    # Store in database
    test_model.adapt_to(
        obj_key="postgres",
        engine_url=pg_url,
        table="test_table"
    )
    
    # Retrieve from database
    retrieved = test_model.__class__.adapt_from(
        {
            "engine_url": pg_url,
            "table": "test_table",
            "selectors": {"id": 42}
        },
        obj_key="postgres",
        many=False
    )
    
    # Verify data integrity
    assert retrieved.id == test_model.id
    assert retrieved.name == test_model.name
    assert retrieved.value == test_model.value
```

#### 3.1.2 Test Case: Batch Operations

**Purpose:** Verify that the PostgreSQL adapter can store and retrieve multiple
records.

**Test Implementation:**

```python
def test_postgres_batch_operations(pg_url, sync_model_factory, postgres_table):
    """Test batch operations with PostgreSQL."""
    model_cls = sync_model_factory(id=1, name="test", value=1.0).__class__
    
    # Register adapter
    model_cls.register_adapter(PostgresAdapter)
    
    # Create multiple test instances
    models = [
        model_cls(id=i, name=f"batch_{i}", value=i * 1.5)
        for i in range(1, 11)
    ]
    
    # Store batch in database
    PostgresAdapter.to_obj(
        models,
        engine_url=pg_url,
        table="test_table",
        many=True
    )
    
    # Retrieve all from database
    retrieved = model_cls.adapt_from(
        {
            "engine_url": pg_url,
            "table": "test_table"
        },
        obj_key="postgres",
        many=True
    )
    
    # Verify all records were stored and retrieved correctly
    assert len(retrieved) == 10
    for i, model in enumerate(sorted(retrieved, key=lambda m: m.id), 1):
        assert model.id == i
        assert model.name == f"batch_{i}"
        assert model.value == i * 1.5
```

#### 3.1.3 Test Case: Error Handling

**Purpose:** Verify that the PostgreSQL adapter handles errors correctly.

**Test Implementation:**

```python
def test_postgres_connection_error(sync_model_factory):
    """Test handling of PostgreSQL connection errors."""
    test_model = sync_model_factory(id=42, name="test_postgres", value=12.34)
    
    # Register adapter
    test_model.__class__.register_adapter(PostgresAdapter)
    
    # Test with invalid connection string
    with pytest.raises(ConnectionError):
        test_model.adapt_to(
            obj_key="postgres",
            engine_url="postgresql://invalid:invalid@localhost:5432/nonexistent",
            table="test_table"
        )
```

### 3.2 Test Suite: MongoDB Integration

#### 3.2.1 Test Case: Single Document Operations

**Purpose:** Verify that the MongoDB adapter can store and retrieve a single
document.

**Test Implementation:**

```python
def test_mongodb_single_document(mongo_url, sync_model_factory):
    """Test MongoDB adapter with a single document."""
    # Create test instance
    test_model = sync_model_factory(id=43, name="test_mongo", value=56.78)
    
    # Register adapter
    test_model.__class__.register_adapter(MongoAdapter)
    
    # Store in database
    test_model.adapt_to(
        obj_key="mongo",
        url=mongo_url,
        db="testdb",
        collection="test_collection"
    )
    
    # Retrieve from database
    retrieved = test_model.__class__.adapt_from(
        {
            "url": mongo_url,
            "db": "testdb",
            "collection": "test_collection",
            "filter": {"id": 43}
        },
        obj_key="mongo",
        many=False
    )
    
    # Verify data integrity
    assert retrieved.id == test_model.id
    assert retrieved.name == test_model.name
    assert retrieved.value == test_model.value
```

#### 3.2.2 Test Case: Batch Operations

**Purpose:** Verify that the MongoDB adapter can store and retrieve multiple
documents.

**Test Implementation:**

```python
def test_mongodb_batch_operations(mongo_url, sync_model_factory):
    """Test batch operations with MongoDB."""
    model_cls = sync_model_factory(id=1, name="test", value=1.0).__class__
    
    # Register adapter
    model_cls.register_adapter(MongoAdapter)
    
    # Create multiple test instances
    models = [
        model_cls(id=i, name=f"batch_{i}", value=i * 1.5)
        for i in range(1, 11)
    ]
    
    # Store batch in database
    MongoAdapter.to_obj(
        models,
        url=mongo_url,
        db="testdb",
        collection="batch_collection",
        many=True
    )
    
    # Retrieve all from database
    retrieved = model_cls.adapt_from(
        {
            "url": mongo_url,
            "db": "testdb",
            "collection": "batch_collection"
        },
        obj_key="mongo",
        many=True
    )
    
    # Verify all records were stored and retrieved correctly
    assert len(retrieved) == 10
    for i, model in enumerate(sorted(retrieved, key=lambda m: m.id), 1):
        assert model.id == i
        assert model.name == f"batch_{i}"
        assert model.value == i * 1.5
```

#### 3.2.3 Test Case: Error Handling

**Purpose:** Verify that the MongoDB adapter handles errors correctly.

**Test Implementation:**

```python
def test_mongodb_connection_error(sync_model_factory):
    """Test handling of MongoDB connection errors."""
    test_model = sync_model_factory(id=43, name="test_mongo", value=56.78)
    
    # Register adapter
    test_model.__class__.register_adapter(MongoAdapter)
    
    # Test with invalid connection string
    with pytest.raises(ConnectionError):
        test_model.adapt_to(
            obj_key="mongo",
            url="mongodb://invalid:invalid@localhost:27017",
            db="testdb",
            collection="test_collection"
        )
```

### 3.3 Test Suite: Neo4j Integration

#### 3.3.1 Test Case: Single Node Operations

**Purpose:** Verify that the Neo4j adapter can store and retrieve a single node.

**Test Implementation:**

```python
def test_neo4j_single_node(neo4j_url, sync_model_factory):
    """Test Neo4j adapter with a single node."""
    # Create test instance
    test_model = sync_model_factory(id=44, name="test_neo4j", value=90.12)
    
    # Register adapter
    test_model.__class__.register_adapter(Neo4jAdapter)
    
    # Store in database
    test_model.adapt_to(
        obj_key="neo4j",
        url=neo4j_url,
        label="TestModel",
        merge_on="id"
    )
    
    # Retrieve from database
    retrieved = test_model.__class__.adapt_from(
        {
            "url": neo4j_url,
            "label": "TestModel",
            "where": "n.id = 44"
        },
        obj_key="neo4j",
        many=False
    )
    
    # Verify data integrity
    assert retrieved.id == test_model.id
    assert retrieved.name == test_model.name
    assert retrieved.value == test_model.value
```

#### 3.3.2 Test Case: Batch Operations

**Purpose:** Verify that the Neo4j adapter can store and retrieve multiple
nodes.

**Test Implementation:**

```python
def test_neo4j_batch_operations(neo4j_url, sync_model_factory):
    """Test batch operations with Neo4j."""
    model_cls = sync_model_factory(id=1, name="test", value=1.0).__class__
    
    # Register adapter
    model_cls.register_adapter(Neo4jAdapter)
    
    # Create multiple test instances
    models = [
        model_cls(id=i, name=f"batch_{i}", value=i * 1.5)
        for i in range(1, 11)
    ]
    
    # Store batch in database
    Neo4jAdapter.to_obj(
        models,
        url=neo4j_url,
        label="TestModel",
        merge_on="id",
        many=True
    )
    
    # Retrieve all from database
    retrieved = model_cls.adapt_from(
        {
            "url": neo4j_url,
            "label": "TestModel"
        },
        obj_key="neo4j",
        many=True
    )
    
    # Verify all records were stored and retrieved correctly
    assert len(retrieved) == 10
    for i, model in enumerate(sorted(retrieved, key=lambda m: m.id), 1):
        assert model.id == i
        assert model.name == f"batch_{i}"
        assert model.value == i * 1.5
```

#### 3.3.3 Test Case: Error Handling

**Purpose:** Verify that the Neo4j adapter handles errors correctly.

**Test Implementation:**

```python
def test_neo4j_connection_error(sync_model_factory):
    """Test handling of Neo4j connection errors."""
    test_model = sync_model_factory(id=44, name="test_neo4j", value=90.12)
    
    # Register adapter
    test_model.__class__.register_adapter(Neo4jAdapter)
    
    # Test with invalid connection string
    with pytest.raises(ConnectionError):
        test_model.adapt_to(
            obj_key="neo4j",
            url="neo4j://invalid:invalid@localhost:7687",
            label="TestModel",
            merge_on="id"
        )
```

### 3.4 Test Suite: Qdrant Integration

#### 3.4.1 Test Case: Vector Storage and Retrieval

**Purpose:** Verify that the Qdrant adapter can store and retrieve vectors.

**Test Implementation:**

```python
def test_qdrant_vector_storage(qdrant_url, sync_vector_model_factory):
    """Test Qdrant adapter with vector storage and retrieval."""
    # Create test instance with embedding vector
    test_model = sync_vector_model_factory(
        id=45, 
        name="test_qdrant", 
        value=100.0,
        embedding=[0.1, 0.2, 0.3, 0.4, 0.5]
    )
    
    # Register adapter
    test_model.__class__.register_adapter(QdrantAdapter)
    
    # Store in database
    test_model.adapt_to(
        obj_key="qdrant",
        url=qdrant_url,
        collection="test_collection",
        vector_field="embedding"
    )
    
    # Retrieve from database using vector similarity search
    retrieved = test_model.__class__.adapt_from(
        {
            "url": qdrant_url,
            "collection": "test_collection",
            "query_vector": test_model.embedding,
            "top_k": 1
        },
        obj_key="qdrant",
        many=False
    )
    
    # Verify data integrity
    assert retrieved.id == test_model.id
    assert retrieved.name == test_model.name
    assert retrieved.value == test_model.value
```

#### 3.4.2 Test Case: Vector Similarity Search

**Purpose:** Verify that the Qdrant adapter can perform vector similarity
search.

**Test Implementation:**

```python
def test_qdrant_similarity_search(qdrant_url, sync_vector_model_factory):
    """Test Qdrant adapter with vector similarity search."""
    model_cls = sync_vector_model_factory(id=1, name="test", value=1.0, embedding=[0.1, 0.2, 0.3, 0.4, 0.5]).__class__
    
    # Register adapter
    model_cls.register_adapter(QdrantAdapter)
    
    # Create multiple test instances with different embeddings
    models = [
        model_cls(
            id=i, 
            name=f"vector_{i}", 
            value=i * 1.5,
            embedding=[i/10, (i+1)/10, (i+2)/10, (i+3)/10, (i+4)/10]
        )
        for i in range(1, 11)
    ]
    
    # Store batch in database
    QdrantAdapter.to_obj(
        models,
        url=qdrant_url,
        collection="similarity_test",
        vector_field="embedding",
        many=True
    )
    
    # Search for vectors similar to the first model
    query_vector = models[0].embedding
    results = model_cls.adapt_from(
        {
            "url": qdrant_url,
            "collection": "similarity_test",
            "query_vector": query_vector,
            "top_k": 3
        },
        obj_key="qdrant",
        many=True
    )
    
    # Verify search results
    assert len(results) == 3
    # The first result should be the exact match (models[0])
    assert results[0].id == models[0].id
```

#### 3.4.3 Test Case: Error Handling

**Purpose:** Verify that the Qdrant adapter handles errors correctly.

**Test Implementation:**

```python
def test_qdrant_connection_error(sync_vector_model_factory):
    """Test handling of Qdrant connection errors."""
    test_model = sync_vector_model_factory(
        id=45, 
        name="test_qdrant", 
        value=100.0,
        embedding=[0.1, 0.2, 0.3, 0.4, 0.5]
    )
    
    # Register adapter
    test_model.__class__.register_adapter(QdrantAdapter)
    
    # Test with invalid connection string
    with pytest.raises(ConnectionError):
        test_model.adapt_to(
            obj_key="qdrant",
            url="http://invalid:6333",
            collection="test_collection",
            vector_field="embedding"
        )
```

## 4. Fixtures

### 4.1 Database Container Fixtures

```python
@pytest.fixture(scope="session")
def pg_url():
    """PostgreSQL container fixture for tests."""
    from testcontainers.postgres import PostgresContainer
    
    with PostgresContainer("postgres:16-alpine") as pg:
        url = pg.get_connection_url()
        yield url


@pytest.fixture(scope="session")
def mongo_url():
    """MongoDB container fixture for tests."""
    from testcontainers.mongodb import MongoDbContainer
    
    with MongoDbContainer("mongo:6.0") as mongo:
        yield mongo.get_connection_url()


@pytest.fixture(scope="session")
def neo4j_url():
    """Neo4j container fixture for tests."""
    from testcontainers.neo4j import Neo4jContainer
    
    with Neo4jContainer("neo4j:5.9") as neo4j:
        yield neo4j.get_connection_url()


@pytest.fixture(scope="session")
def qdrant_url():
    """Qdrant container fixture for tests."""
    from testcontainers.qdrant import QdrantContainer
    
    with QdrantContainer("qdrant/qdrant:v1.8.1") as qc:
        yield f"http://{qc.get_container_host_ip()}:{qc.get_exposed_port(6333)}"
```

### 4.2 Model Fixtures

```python
@pytest.fixture
def sync_model_factory():
    """Factory for creating test models with adapters registered."""
    from pydantic import BaseModel
    from pydapter.core import Adaptable
    
    def create_model(**kw):
        class TestModel(Adaptable, BaseModel):
            id: int
            name: str
            value: float
        
        return TestModel(**kw)
    
    return create_model


@pytest.fixture
def sync_vector_model_factory():
    """Factory for creating test models with vector field."""
    from pydantic import BaseModel
    from pydapter.core import Adaptable
    
    def create_model(**kw):
        class VectorModel(Adaptable, BaseModel):
            id: int
            name: str
            value: float
            embedding: list[float]
        
        return VectorModel(**kw)
    
    return create_model
```

## 5. Test Coverage Targets

- **Line Coverage Target:** 90% for database adapter code
- **Branch Coverage Target:** 85% for database adapter code
- **Critical Modules:** 95% coverage for core adapter functionality

## 6. Continuous Integration

The tests should be run in a CI environment with Docker support. If Docker is
not available, the tests should be skipped.

```python
def is_docker_available():
    """Check if Docker is available."""
    import subprocess
    try:
        subprocess.run(["docker", "info"], check=True, capture_output=True)
        return True
    except (subprocess.SubprocessError, FileNotFoundError):
        return False

# Skip tests if Docker is not available
pytestmark = pytest.mark.skipif(
    not is_docker_available(),
    reason="Docker is not available"
)
```

## 7. Notes and Caveats

### 7.1 Known Limitations

- Tests require Docker to be installed and running
- Tests may be slow due to container startup time
- Some tests may be flaky if container initialization fails

### 7.2 Future Improvements

- Add more comprehensive error handling tests
- Add performance tests for batch operations
- Add tests for more complex queries and filters
- Add support for more database types
