# Dependency Verification Guide

Diagnosing and resolving package version mismatches between local and Ray worker environments.

---

## Why Dependency Mismatches Matter

When running Geneva UDFs on Ray, your code is serialized locally and executed on remote workers. If the worker environment differs from your local environment, you may encounter:

| Symptom | Likely Cause |
|---------|--------------|
| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch |
| `TypeError: Can't instantiate abstract class` | Package structure differences |
| `ArrowInvalid: cannot cast` / serialization errors | NumPy 1.x vs 2.x mismatch |
| `ModuleNotFoundError` on workers | Package only installed locally |
| Model loading failures | PyTorch version mismatch |
| Unexpected behavior, wrong results | Silent version incompatibilities |

These issues are notoriously difficult to debug because the error messages often don't indicate the root cause.

---

## The `compare_ray_environments` Tool

Geneva provides a diagnostic tool to compare your local environment against Ray workers:

```
src/geneva/runners/ray/compare_env.py
```

### Programmatic Usage

```python
from geneva.runners.ray.compare_env import compare_ray_environments

# Compare and print (requires Ray to be initialized, e.g., in Geneva context)
result = compare_ray_environments()

# Data-only mode (no printing)
from geneva.runners.ray.compare_env import get_comparison_result
result = get_comparison_result()

# Auto-init and connect to existing cluster
result = compare_ray_environments(auto_init=True)

# Auto-init a new local cluster for testing
result = compare_ray_environments(auto_init=True, ray_address=None)

# Filter environment variables by prefix
result = compare_ray_environments(env_prefix="PY")
```

### CLI Usage

```bash
# Connect to existing Ray cluster
python -m geneva.runners.ray.compare_env

# Start new local Ray cluster
python -m geneva.runners.ray.compare_env --address local

# Filter env vars by prefix
python -m geneva.runners.ray.compare_env --env-prefix RAY

# Show full JSON snapshots
python -m geneva.runners.ray.compare_env --show-all

# Skip sys.path comparison
python -m geneva.runners.ray.compare_env --no-sys-path
```

---

## Understanding the Output

The tool outputs several sections:

### PYTHON / PLATFORM

Shows Python version and OS information for both environments:

```
=== PYTHON / PLATFORM ===
Local:
  Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
  Impl  : CPython
  Exec  : /home/user/.venv/bin/python
  OS    : Linux 5.15.0-generic (x86_64)

Remote:
  Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
  Impl  : CPython
  Exec  : /home/ray/anaconda3/bin/python
  OS    : Linux 5.4.0-aws (x86_64)
```

**Red flags**: Different Python versions, different OS (macOS local vs Linux remote).

### ENV VARS: keys only in LOCAL/REMOTE

Environment variables present in only one environment:

```
=== ENV VARS: keys only in LOCAL ===
  + CONDA_PREFIX
  + VIRTUAL_ENV

=== ENV VARS: keys only in REMOTE ===
  + RAY_ADDRESS
  + KUBERNETES_SERVICE_HOST
```

**Red flags**: Missing `AWS_*` or `GOOGLE_APPLICATION_CREDENTIALS` (storage auth).

### ENV VARS: differing values

Variables present in both but with different values:

```
=== ENV VARS: differing values ===
  * PATH
    local : /home/user/.venv/bin:/usr/local/bin:/usr/bin
    remote: /home/ray/anaconda3/bin:/usr/local/bin:/usr/bin
  * HOME
    local : /home/user
    remote: /home/ray
```

### PACKAGES: only in LOCAL/REMOTE

Packages installed in only one environment:

```
=== PACKAGES: only in LOCAL ===
  + my-custom-package
  + dev-tools

=== PACKAGES: only in REMOTE ===
  + kuberay-client
```

**Red flags**: UDF dependencies missing from workers.

### PACKAGES: version mismatches

Packages with different versions:

```
=== PACKAGES: version mismatches ===
  * numpy: local=1.26.4  remote=2.2.6
  * torch: local=2.0.1  remote=2.8.0+cpu
  * attrs: local=23.2.0  remote=24.2.0
  * pyarrow: local=14.0.1  remote=17.0.0
```

**Red flags**: Major version differences (NumPy 1.x vs 2.x), PyTorch versions.

### sys.path differences

Shows differences in Python module search paths:

```
=== sys.path differences ===
  Only in LOCAL (3):
    + /home/user/project/src
    + /home/user/.venv/lib/python3.11/site-packages
  Only in REMOTE (2):
    + /home/ray/anaconda3/lib/python3.11/site-packages
  Shared entries: 5 (local=8, remote=7)
```

---

## Common Issues and Solutions

### NumPy 1.x vs 2.x Mismatch

NumPy 2.0 introduced breaking changes in array serialization:

```
=== PACKAGES: version mismatches ===
  * numpy: local=1.26.4  remote=2.2.6
```

**Symptoms**: `ArrowInvalid`, `ValueError: cannot convert`, serialization failures.

**Solution**: Pin NumPy version using a manifest:

```python
from geneva.manifest.builder import PipManifestBuilder

manifest = (
    PipManifestBuilder.create("my-manifest")
    .pip(["numpy==1.26.4"])
    .build()
)
```

### PyTorch Version Mismatch

```
=== PACKAGES: version mismatches ===
  * torch: local=2.0.1  remote=2.8.0+cpu
```

**Symptoms**: Model loading failures, CUDA errors, unexpected inference results.

**Solution**: Use matching versions via a manifest:

```python
from geneva.manifest.builder import PipManifestBuilder

manifest = (
    PipManifestBuilder.create("my-manifest")
    .pip(["torch==2.0.1"])
    .build()
)
```

### attrs Version Mismatch

```
=== PACKAGES: version mismatches ===
  * attrs: local=23.2.0  remote=24.2.0
```

**Symptoms**: `TypeError: Enum.__new__() missing 1 required positional argument`.

**Solution**: Sync attrs version via a manifest:

```python
from geneva.manifest.builder import PipManifestBuilder

manifest = (
    PipManifestBuilder.create("my-manifest")
    .pip(["attrs==23.2.0"])
    .build()
)
```

### Missing Package on Workers

```
=== PACKAGES: only in LOCAL ===
  + my-udf-dependency
```

**Symptoms**: `ModuleNotFoundError: No module named 'my_udf_dependency'`.

**Solution**: Add to a manifest:

```python
from geneva.manifest.builder import PipManifestBuilder

manifest = (
    PipManifestBuilder.create("my-manifest")
    .pip(["my-udf-dependency==1.0.0"])
    .build()
)
```

### Architecture Mismatch (macOS to Linux)

```
=== PYTHON / PLATFORM ===
Local:
  OS    : Darwin 23.0.0 (arm64)
Remote:
  OS    : Linux 5.4.0-aws (x86_64)
```

**Symptoms**: `ModuleNotFoundError` for compiled extensions, segfaults.

**Solution**: Run Geneva from the same OS/architecture as your cluster (Linux x86_64).

---

## Fixing Mismatches

### Option 1: Manifest pip Dependencies (Quick Fix)

Specify packages in a Geneva manifest:

```python
from geneva.manifest.builder import PipManifestBuilder

manifest = (
    PipManifestBuilder.create("my-manifest")
    .pip([
        "numpy==1.26.4",
        "torch==2.0.1",
        "attrs==23.2.0",
    ])
    .build()
)

# Then use with db.context()
conn = geneva.connect("s3://my-bucket/my-db")
conn.define_manifest("my-manifest", manifest)
with conn.context(cluster="my-cluster", manifest="my-manifest"):
    conn.open_table("my-table").backfill("my-column")
```

**Pros**: Quick, reusable across sessions, stored with your database.
**Cons**: Slower startup (downloads packages), may not work for complex dependencies.

### Option 2: Custom Ray Worker Image

For KubeRay deployments, build a custom worker image:

```dockerfile
# Dockerfile.ray-worker
FROM rayproject/ray:2.54.0-py311

# Install exact versions
RUN pip install \
    numpy==1.26.4 \
    torch==2.10.0 \
    attrs==23.2.0 \
    geneva==0.8.0

# Copy any custom packages
COPY ./my_udfs /app/my_udfs
```

Then reference in RayCluster spec:

```yaml
spec:
  workerGroupSpecs:
    - template:
        spec:
          containers:
            - image: your-registry/ray-worker:latest
```

**Pros**: Fastest startup, reproducible.
**Cons**: Requires image build/push workflow.

### Option 3: Conda Environment

Use a conda environment on workers via the cluster builder:

```python
from geneva.cluster.builder import KubeRayClusterBuilder

cluster = (
    KubeRayClusterBuilder.create("my-cluster")
    .ray_init_kwargs({
        "runtime_env": {"conda": "environment.yml"}
    })
    .build()
)
```

Or specify conda channels and dependencies inline:

```python
cluster = (
    KubeRayClusterBuilder.create("my-cluster")
    .ray_init_kwargs({
        "runtime_env": {
            "conda": {
                "channels": ["conda-forge"],
                "dependencies": [
                    "python=3.10",
                    "ffmpeg<8",
                    "torchvision=0.22.1"
                ]
            },
            "config": {"eager_install": True}
        }
    })
    .build()
)
```

---

## Diagnostic Workflow

When encountering serialization or "abstract class" errors:

1. **Run the diagnostic tool**:
   ```bash
   python -m geneva.runners.ray.compare_env
   ```

2. **Check PACKAGES: version mismatches** section first.

3. **Identify critical packages**: numpy, torch, pyarrow, attrs, pydantic.

4. **Fix with manifest** for quick testing:
   ```python
   from geneva.manifest.builder import PipManifestBuilder
   manifest = PipManifestBuilder.create("fix").pip(["numpy==1.26.4"]).build()
   ```

5. **Build custom image** for production (if using KubeRay).

---

## API Reference

### `compare_ray_environments()`

Main function that compares and prints differences.

```python
def compare_ray_environments(
    env_prefix: str | None = None,
    show_all: bool = False,
    include_sys_path: bool = True,
    auto_init: bool = False,
    ray_address: str | None = "auto",
) -> ComparisonResult:
```

**Parameters**:
- `env_prefix`: Filter environment variables by prefix (e.g., `"PY"`, `"RAY"`).
- `show_all`: Print full JSON snapshots.
- `include_sys_path`: Include sys.path comparison.
- `auto_init`: If `True`, initialize Ray automatically. If `False` (default), require Ray to already be initialized.
- `ray_address`: Ray address (only used when `auto_init=True`). `"auto"` connects to existing cluster, `None` starts new local cluster.

**Returns**: `ComparisonResult` TypedDict with:
- `local`: Local `EnvSnapshot`
- `remote`: Remote `EnvSnapshot`
- `env_diffs`: Environment variable differences
- `pkg_diffs`: Package differences
- `sys_path_diffs`: sys.path differences

**Raises**: `RuntimeError` if Ray is not initialized and `auto_init=False`.

### `get_comparison_result()`

Data-only version (no printing).

```python
def get_comparison_result(
    env_prefix: str | None = None,
    auto_init: bool = False,
    ray_address: str | None = "auto",
) -> ComparisonResult:
```

### `snapshot_env()`

Collect environment snapshot (can be used locally without Ray).

```python
def snapshot_env() -> EnvSnapshot:
```

---

## Related Documentation

- [operations.md](operations.md) - General troubleshooting
- [performance.md](performance.md) - Resource tuning
- [storage.md](storage.md) - S3/GCS configuration
