---
name: registry-mirror-legacy-e2e-test
description: Test the registry mirror → compile (with legacy migration) → compare pipeline. Mirrors prod data to a dev bucket, runs compile with --with-legacy-migration=v1 to clean up disabled connector artifacts, and compares the result against prod. Use when validating that the legacy migration correctly handles strict-encrypt and other disabled connectors.
devin-enabled: true
icon: flask-conical
---

# Registry Mirror + Legacy Migration E2E Test

Test the full mirror → compile → compare pipeline with legacy migration cleanup. Validates that mirroring prod data and running `--with-legacy-migration=v1` correctly removes disabled connector artifacts (e.g. strict-encrypt variants with `registryOverrides.{registry}.enabled: false`) and produces a clean registry.

## Prerequisites

- **Repo:** `~/repos/airbyte-ops-mcp`
- **Credentials:** `GCS_CREDENTIALS` or `GCP_GSM_CREDENTIALS` environment variable with a service account JSON key that has access to:
  - Read: `prod-airbyte-cloud-connector-metadata-service` (project: `prod-ab-cloud-proj`)
  - Read/Write: `dev-airbyte-cloud-connector-metadata-service-2` (project: `dev-ab-cloud-proj`)
- If `GCS_CREDENTIALS` is not set but `GCP_GSM_CREDENTIALS` is, export it: `export GCS_CREDENTIALS="$GCP_GSM_CREDENTIALS"`
- **CLI tools:** `gcloud` and `gsutil` (installed via `brew install --cask google-cloud-sdk`). If unavailable, Python/gcsfs can be used as a fallback for file verification.

## Devin Secrets Needed

- `GCP_GSM_CREDENTIALS` — Service account JSON key with GCS bucket access (read prod, read/write dev)

## Step 1: Create a Dev Store

Create a datestamped store ID for the test:

```bash
STORE_ID="$(date +%Y%m%d)-mirror-legacy-e2e-$(openssl rand -hex 3)"
echo "Store ID: ${STORE_ID}"
```

## Step 2: Mirror Prod to Dev

Mirror all connector metadata from prod to the dev bucket under the test prefix:

```bash
uv run airbyte-ops registry store mirror \
  --source-store coral:prod \
  --gcs-bucket dev-airbyte-cloud-connector-metadata-service-2 \
  --output-path-root "${STORE_ID}"
```

This copies all connector metadata from `prod-airbyte-cloud-connector-metadata-service` to `dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/`.

**Failure criteria:** If the mirror fails, the test fails. Record the error and stop.

**Expected output:** A count of connectors and files copied. This may take several minutes for a full mirror.

To mirror only specific connectors (faster, for targeted testing):

```bash
uv run airbyte-ops registry store mirror \
  --source-store coral:prod \
  --gcs-bucket dev-airbyte-cloud-connector-metadata-service-2 \
  --output-path-root "${STORE_ID}" \
  --connector-name destination-postgres \
  --connector-name destination-postgres-strict-encrypt \
  --connector-name source-faker
```

## Step 3: Compile with Legacy Migration

Run compile with `--with-legacy-migration v1`. This is **required** (not optional) until the full migration to the new pipeline is complete.

```bash
uv run airbyte-ops registry store compile \
  --store "coral:dev/${STORE_ID}" \
  --with-legacy-migration v1
```

### What this does

1. **Steps 1-5:** Standard compile pipeline — scan versions, compute latest GA, sync `latest/` directories
2. **Step 5m (migration):** Reads each connector's `latest/metadata.yaml`, checks `registryOverrides.{cloud,oss}.enabled`. For connectors where `enabled=false`, deletes all `cloud.json` and/or `oss.json` files from every version directory and `latest/`.
3. **Step 6:** Builds global registry index (`cloud_registry.json`, `oss_registry.json`) and per-connector version indexes. Definition ID deduplication ensures no duplicate `definitionId` entries (e.g. strict-encrypt + base connector sharing the same ID).

### Pass criteria

- Compile exits 0
- Output includes `Step 5m: Legacy migration v1` with a count of deleted files
- Output includes `Migration v1: deleted N files across M connectors` where N > 0 (assuming disabled connectors exist in prod)
- No errors reported

### Fail criteria

- Compile fails (non-zero exit code)
- Migration reports 0 files deleted (unexpected if prod has strict-encrypt connectors)
- Any error in the output

## Step 4: Verify Migration Results

After compile, verify that disabled connector artifacts were cleaned up.

### 4a: Check that disabled connector artifacts are gone

For known disabled connectors (e.g. `destination-postgres-strict-encrypt`), verify their `cloud.json` and `oss.json` files no longer exist:

```bash
# Should return empty / "not found" for disabled connectors
gsutil ls "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/destination-postgres-strict-encrypt/latest/cloud.json" 2>&1 || echo "GONE (expected)"
gsutil ls "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/destination-postgres-strict-encrypt/latest/oss.json" 2>&1 || echo "GONE (expected)"
```

### 4b: Check that metadata.yaml is preserved

The migration should only delete `cloud.json`/`oss.json`, not `metadata.yaml` or other artifacts:

```bash
# Should still exist
gsutil ls "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/destination-postgres-strict-encrypt/latest/metadata.yaml"
```

### 4c: Check that enabled connectors are unaffected

```bash
# Should exist
gsutil ls "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/destination-postgres/latest/cloud.json"
gsutil ls "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/source-faker/latest/cloud.json"
```

### 4d: Verify global index excludes disabled connectors

```bash
# Download and inspect the cloud registry
gsutil cp "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/registries/v0/cloud_registry.json" /tmp/cloud_registry.json

# Check that strict-encrypt is NOT in the index
python3 -c "
import json
with open('/tmp/cloud_registry.json') as f:
    reg = json.load(f)
all_repos = []
for section in ('destinations', 'sources'):
    all_repos.extend(e.get('dockerRepository', '') for e in reg.get(section, []))
se_repos = [r for r in all_repos if 'strict-encrypt' in r]
print(f'Total connectors: {len(all_repos)}')
print(f'Strict-encrypt in index: {se_repos if se_repos else \"NONE (expected)\"}')
"
```

### Fallback: Python/gcsfs verification

If `gsutil` is not available, use Python:

```python
import gcsfs, json, os
fs = gcsfs.GCSFileSystem(token=json.loads(os.environ['GCS_CREDENTIALS']))
store_id = '<STORE_ID>'
base = f'dev-airbyte-cloud-connector-metadata-service-2/{store_id}'

# Check disabled connector artifacts are gone
print('SE cloud.json exists:', fs.exists(f'{base}/metadata/airbyte/destination-postgres-strict-encrypt/latest/cloud.json'))
# Check enabled connector artifacts still exist
print('DP cloud.json exists:', fs.exists(f'{base}/metadata/airbyte/destination-postgres/latest/cloud.json'))
# Check metadata.yaml preserved
print('SE metadata.yaml exists:', fs.exists(f'{base}/metadata/airbyte/destination-postgres-strict-encrypt/latest/metadata.yaml'))
```

## Step 5: Compare Against Prod

Run the compare command to verify the dev store matches prod (minus the expected cleanup):

```bash
uv run airbyte-ops registry store compare \
  --store "coral:dev/${STORE_ID}" \
  --reference-store coral:prod
```

### Expected differences

- **Index diffs:** The dev store's global index should have *fewer* entries than prod if prod still has disabled connector entries that haven't been cleaned up yet. After prod is migrated, indexes should match.
- **Artifact diffs:** Disabled connectors will be missing `cloud.json`/`oss.json` in the dev store (expected — that's the point of the migration).
- **Tolerated diffs:** Timestamp fields (`registry_entry_generated_at`, `metadata_etag`, `metadata_last_modified`) are expected.

### Pass criteria

- Compare completes without errors
- All diffs are either tolerated (timestamps) or expected (disabled connector artifact removal)
- No unexpected missing or extra connectors in the index

## Step 6: Cleanup

Delete the test data from the dev bucket:

```bash
gsutil -m rm -r "gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/"
```

Or with Python/gcsfs:

```python
fs.rm(f'dev-airbyte-cloud-connector-metadata-service-2/{store_id}', recursive=True)
```

## Step 7: Report Results

Send the user a summary including:

1. **Overall result:** PASS or FAIL
2. **Mirror:** Number of connectors/files copied
3. **Migration:** Number of files deleted, which connectors were affected
4. **Verification:** Whether disabled artifacts were removed and enabled artifacts preserved
5. **Compare:** Summary of diffs (expected vs unexpected)
6. **Store location:** `gs://dev-airbyte-cloud-connector-metadata-service-2/<STORE_ID>/` (if not cleaned up yet)

## Reference

- **Dev bucket:** `dev-airbyte-cloud-connector-metadata-service-2` (project: `dev-ab-cloud-proj`)
- **Prod bucket:** `prod-airbyte-cloud-connector-metadata-service` (project: `prod-ab-cloud-proj`)
- **Related PR:** https://github.com/airbytehq/airbyte-ops-mcp/pull/546 (definitionId dedup + legacy migration)
- **Related issue:** https://github.com/airbytehq/airbyte-ops-mcp/issues/544
