# snowflake-snowconvert-testing-orchestration

> `test-runner` -- A testing framework for capturing and validating stored procedure baselines during database migrations to Snowflake.

Part of the [SnowConvert](https://www.snowflake.com/en/data-cloud/snowconvert/) migration toolchain, this tool ensures that migrated objects produce the same results as the source system by capturing baseline execution results and comparing them against the target system.

## Overview

`test-runner` supports:

- Capturing stored procedure execution results from source databases (SQL Server, Oracle)
- Deploying and executing procedures on Snowflake
- Comparing results between source and target systems
- Detailed validation reports including schema, metrics, and row-level comparisons

## Installation

### From PyPI

```bash
pip install snowflake-snowconvert-testing-orchestration
```

### From source (development)

```bash
cd test-runner
pip install -e ".[dev]"
```

### Prerequisites

- Python >= 3.10
- pip

### Dependencies

- `snowflake-data-validation[all]` -- Snowflake integration and data comparison
- `snowflake-code-unit-registry` -- Code Unit Registry access (Rust native extension, installed from PyPI)
- `dependency-injector` -- Dependency injection framework
- `pyyaml` -- YAML configuration parsing

## Usage

The `test-runner` CLI provides two main commands:

### 1. Capture Baselines

Execute stored procedures on the source database and save the results as baseline files.

```bash
test-runner capture --project-root <path>
```

**Options:**

- `--project-root`: Path to the project directory (default: current directory)
- `--source-connection`: Source database connection string
- `--baseline-dir`: Directory to store baseline files (default: `.scai/baselines/<date>`)
- `--baseline-stage`: Snowflake stage to upload baselines
- `-c, --connection`: Snowflake connection name for stage upload
- `--where`: SQL-like WHERE clause to filter code units (same syntax as `scai code deploy --where`)
- `--include-dependencies`: When used with `--where`, also include dependencies of matched code units
- `-w, --watch`: Watch mode for continuous execution
- `-q, --quiet`: Suppress verbose output

### 2. Validate Against Baselines

Execute procedures on Snowflake and compare results against captured baselines.

```bash
test-runner validate --project-root <path>
```

**Options:**

- `--project-root`: Path to the project directory (default: current directory)
- `-c, --connection`: Snowflake connection name
- `--baseline-dir`: Directory containing baseline files
- `--baseline-stage`: Snowflake stage containing baselines
- `--where`: SQL-like WHERE clause to filter code units (same syntax as `scai code deploy --where`)
- `--include-dependencies`: When used with `--where`, also include dependencies of matched code units
- `--create-schema`: Automatically create schema if missing
- `--output-dir`: Directory for validation reports
- `-w, --watch`: Watch mode for continuous validation
- `-q, --quiet`: Suppress verbose output

## Project Structure

A test-runner project requires the following structure:

```
project-root/
├── settings/
│   └── test_config.yaml          # Configuration file
├── artifacts/
│   └── <database>/
│       └── <schema>/
│           └── procedure/
│               └── <ProcedureName>/
│                   └── test/
│                       └── <procedure>.yml  # Test definition
├── .scai/
│   └── baselines/
│       └── <date>/
│           └── <database>_<schema>_<ProcedureName>/
│               └── case_*.json   # Captured baselines
└── reports/                      # Validation reports (auto-generated)
```

## Configuration

### test_config.yaml

Create a `test_config.yaml` file in the `settings/` directory:

```yaml
source_platform: sqlserver
target_platform: snowflake
output_directory_path: /tmp/test_runner_reports

validation_configuration:
  schema_validation: true
  metrics_validation: true
  row_validation: true
  max_failed_rows_number: 100
```

Connection details are resolved at runtime from TOML files or CLI flags:

- **Source**: `~/.snowflake/snowct/<dialect>.toml` (`default_connection_name`), or `--source-connection <name>`
- **Target** (Snowflake): `~/.snowflake/config.toml` (`default_connection_name`), or `--connection <name>`

### Test Definition Files

Create a `.yml` file for each stored procedure in `artifacts/<database>/<schema>/procedure/<ProcedureName>/test/`:

```yaml
validation:
  source:
    steps:
      run: |-
        EXECUTE testdb.dbo.GetAllProducts
  target:
    steps:
      run: |-
        CALL TESTDB.DBO.GETALLPRODUCTS()
  test_cases:
    - []
```

For procedures with parameters:

```yaml
validation:
  source:
    steps:
      run: |-
        EXECUTE testdb.dbo.GetProductsByCategory @Category = ?
  target:
    steps:
      run: |-
        CALL TESTDB.DBO.GETPRODUCTSBYCATEGORY(?)
  test_cases:
    - ["Electronics"]
    - ["Hardware"]
    - ["Accessories"]
```

## Example: e2e_project

The `e2e_project` directory contains a complete example:

### 1. Capture baselines from SQL Server

```bash
cd e2e_project
test-runner capture --project-root .
```

This will:

- Connect to SQL Server using credentials from `settings/test_config.yaml`
- Execute all procedures defined in `artifacts/testdb/dbo/procedure/*/test/*.yml`
- Save baseline results to `.scai/baselines/<date>/`

Example baseline file (`.scai/baselines/20260217/testdb_dbo_GetAllProducts/case_000_99914b93.json`):

```json
{
  "procedure": "testdb.dbo.GetAllProducts",
  "parameters": {},
  "captured_at": "2026-02-17T03:13:17.882140+00:00",
  "result_sets": [
    [
      {
        "Id": 1,
        "Name": "Widget",
        "Price": "9.99",
        "Category": "Hardware"
      }
    ]
  ]
}
```

### 2. Validate on Snowflake

```bash
test-runner validate --project-root . -c your_snowflake_connection
```

This will:

- Deploy procedures to Snowflake
- Execute them with the same parameters
- Compare results against baselines
- Generate validation reports in `reports/`

### 3. Review validation results

Reports are saved in the `reports/` directory and include:

- Schema validation (column names, types, order)
- Metrics validation (row count, null counts)
- Row-level validation (data differences)

## Library Usage

The test-runner can also be used as a Python library for integration into CI/CD pipelines or other tools:

```python
from test_runner import create_container, run_capture, run_validate

container = create_container("my_project")
config = container.config()
registry = container.factory_registry()

capture_stats = run_capture(
    config=config,
    factory_registry=registry,
    project_root="my_project",
)
print(f"Captured {capture_stats.total} baselines")

validate_stats = run_validate(
    config=config,
    factory_registry=registry,
    project_root="my_project",
)
print(f"Passed: {validate_stats.passed}, Failed: {validate_stats.failed}")
```

Both functions require the same `settings/test_config.yaml` and `artifacts/**/test/*.yml` structure described above.

## Development

### Prerequisites

- Python >= 3.10

### Setup

```bash
git clone https://github.com/snowflake-eng/migrations-testing-orchestration.git

# Install test-runner with dev dependencies
cd migrations-testing-orchestration/test-runner
pip install -e ".[dev]"
```

> **Note:** To test against a local CUR build (e.g. an unreleased version), a [Rust toolchain](https://rustup.rs/) is required. Clone CUR and install it into the same environment after the step above:
> ```bash
> git clone https://github.com/snowflake-eng/migrations-code-unit-registry.git
> pip install -e ../migrations-code-unit-registry/crates/scai-state-python
> ```

### Running Tests

```bash
cd test-runner
pytest --tb=short -v
```

### Running Tests with Coverage

```bash
pytest --cov=test_runner --cov-report=html
```

### Build

This package uses [Hatch](https://hatch.pypa.io/) as the build system.

```bash
pip install hatch
hatch build
```

## CI/CD

This package is built and published through GitHub Actions workflows:

| Pipeline | Registry | Trigger |
|----------|----------|---------|
| CI | [TestPyPI](https://test.pypi.org/project/snowflake-snowconvert-testing-orchestration/) | Pull requests |
| CD | Azure DevOps (snowmountain-python) | Merge to main |
| Release | [PyPI](https://pypi.org/project/snowflake-snowconvert-testing-orchestration/) | GitHub Release tag |

Release packages are signed with [sigstore](https://www.sigstore.dev/) for supply-chain verification. Signature files (`.sig`, `.crt`, `.sigstore`) are included in each GitHub Release.

## Exit Codes

- `0`: All tests passed
- `> 0`: Number of failed tests or errors

## Support

For issues or questions, please open an issue at [github.com/snowflake-eng/migrations-testing-orchestration](https://github.com/snowflake-eng/migrations-testing-orchestration/issues).
