# Package Management Audit: Findings & Proposal

## Executive Summary

This audit examines the current package management system in atopile, identifies gaps in version validation and freshness checking, and proposes improvements inspired by the Parts Lifecycle Manager pattern.

**Key Findings:**
1. `ato sync` only checks if *a* version is installed, not if the *correct* version is installed
2. No mechanism exists to detect locally modified package files
3. No lock file for transitive dependencies
4. The Parts Lifecycle Manager provides a proven pattern we can adopt

---

## Part 1: Current Architecture

### 1.1 CLI Commands

| Command | File | Description |
|---------|------|-------------|
| `ato add <pkg>` | `src/atopile/cli/install.py` | Add dependency to project |
| `ato remove <pkg>` | `src/atopile/cli/install.py` | Remove dependency from project |
| `ato sync` | `src/atopile/cli/install.py` | Sync dependencies with manifest |

### 1.2 Core Files

```
src/faebryk/libs/project/dependencies.py  # Core resolution & installation logic
src/atopile/config.py                      # Dependency specs & manifest I/O
src/faebryk/libs/backend/packages/api.py   # Registry API client
src/faebryk/libs/package/dist.py           # Distribution loading & validation
src/atopile/version.py                     # Semver matching logic
```

### 1.3 Dependency Storage

Dependencies are stored in `ato.yaml`:
```yaml
dependencies:
  - type: registry
    identifier: atopile/buttons
    release: 0.3.1
  - type: git
    repo_url: https://github.com/example/pkg.git
    ref: main
```

Installed packages live in: `.ato/modules/{owner}/{package-name}/`

### 1.4 Current Sync Behavior

From `dependencies.py`, the `sync_versions()` method:

```python
def sync_versions(self) -> None:
    """Ensure installed version matches manifested version."""
    for dep in self.dag.values():
        if dep.spec.type != "registry":
            continue

        installed_version = dep.project_config.package.version
        manifest_version = dep.spec.release

        if installed_version != manifest_version:
            # Re-install the correct version
            dep.install()
```

**Issue:** This only runs if you explicitly call `ProjectDependencies(sync_versions=True)`. The current `ato sync` command does trigger this, but:
1. It only compares version strings - no integrity validation
2. It doesn't detect corrupted or manually edited files
3. It doesn't handle the case where a package directory exists but is from a different source

---

## Part 2: Identified Gaps

### Gap 1: No Integrity Verification

**Problem:** When syncing, we only check if the version string in the installed package's `ato.yaml` matches the manifest. We don't verify:
- The files haven't been corrupted
- The files haven't been manually modified
- The files are actually from the expected source (registry vs git vs local)

**Risk:** A user could modify files in `.ato/modules/` and those changes would silently persist through `ato sync`.

### Gap 2: No Lock File for Transitive Dependencies

**Problem:** Only direct dependencies have pinned versions in `ato.yaml`. Transitive dependencies are resolved fresh on each sync.

**Risk:** Running `ato sync` on Monday vs Friday could yield different transitive dependency versions if new releases were published.

### Gap 3: File Dependencies Have No Version Tracking

**Problem:** `file://` dependencies point to local paths with no version or hash tracking.

```yaml
- type: file
  path: ../local-package
```

**Risk:** The local package could change arbitrarily, and we'd never detect it.

### Gap 4: Git Dependencies Don't Pin Commits

**Problem:** Git dependencies can specify a branch/tag but not necessarily a specific commit hash.

```yaml
- type: git
  repo_url: https://github.com/example/pkg.git
  ref: main  # This is a moving target!
```

**Risk:** `main` can change between syncs, causing inconsistent builds.

### Gap 5: No Manual Modification Detection

**Problem:** Users might edit files in `.ato/modules/` for debugging or prototyping. Currently:
- We don't detect this
- We'd silently overwrite their changes on sync
- No warning is provided

### Gap 6: No Freshness Checking for Cloned Repos

**Problem:** When a user clones a project with existing dependencies, we don't automatically verify:
- Dependencies are actually installed
- Installed versions match the manifest
- Dependencies aren't corrupted

---

## Part 3: Parts Lifecycle Manager Pattern

The Parts Lifecycle Manager (`src/faebryk/libs/part_lifecycle.py`) provides an excellent pattern for resource management.

### 3.1 Key Patterns

#### Pattern 1: Checksum-Based Integrity
```python
class AutoGenerated:
    system: str = "ato_part"     # Source system
    date: datetime               # Generation time
    source: str                  # e.g., "easyeda:C123456"
    checksum: str                # SHA256 of normalized content
```

The checksum is embedded in the file itself and verified on load.

#### Pattern 2: Manual Modification Detection
```python
@staticmethod
def verify(stated_checksum: str, file_contents: str):
    with_placeholder = file_contents.replace(
        stated_checksum, CHECKSUM_PLACEHOLDER
    )
    Checksum.verify(stated_checksum, with_placeholder)
    # Raises _FileManuallyModified on mismatch
```

If a user modifies the file, the checksum won't match and we detect it.

#### Pattern 3: Source Tracking
Every managed resource records:
- What system generated it
- When it was generated
- Where the source data came from

#### Pattern 4: Graceful Handling of Manual Modifications
```python
try:
    existing = AtoPart.load(part.path)
except _FileManuallyModified as ex:
    raise UserValueError(
        f"Part {part.path} has been manually modified. "
        "Please revert or remove the `is_auto_generated` trait."
    )
```

Users are warned, not silently overwritten.

---

## Part 4: Proposed Solution

### 4.1 New Package Metadata File

Create a `.ato/modules/{pkg}/.package_meta.json`:

```json
{
  "schema_version": 1,
  "identifier": "atopile/buttons",
  "version": "0.3.1",
  "source": {
    "type": "registry",
    "url": "https://packages.atopileapi.com",
    "release_sha256": "abc123..."
  },
  "installed_at": "2024-01-15T10:30:00Z",
  "installed_by": "ato@0.14.0",
  "content_checksum": "def456...",
  "files": {
    "ato.yaml": "sha256:...",
    "buttons.ato": "sha256:...",
    "parts/button.ato": "sha256:..."
  }
}
```

### 4.2 Enhanced Sync Operation

```python
class PackageState(Enum):
    NOT_INSTALLED = "not_installed"           # Directory doesn't exist
    INSTALLED_FRESH = "installed_fresh"       # Matches manifest exactly
    INSTALLED_WRONG_VERSION = "wrong_version" # Different version installed
    INSTALLED_MODIFIED = "modified"           # Files changed locally
    INSTALLED_CORRUPTED = "corrupted"         # Checksum mismatch (unintentional)
    INSTALLED_NO_META = "no_meta"             # Legacy install without metadata

def get_package_state(dep: ProjectDependency) -> PackageState:
    if not dep.target_path.exists():
        return PackageState.NOT_INSTALLED

    meta_path = dep.target_path / ".package_meta.json"
    if not meta_path.exists():
        return PackageState.INSTALLED_NO_META

    meta = PackageMeta.load(meta_path)

    # Check version
    if meta.version != dep.spec.release:
        return PackageState.INSTALLED_WRONG_VERSION

    # Check file integrity
    for file, expected_hash in meta.files.items():
        actual_hash = compute_file_hash(dep.target_path / file)
        if actual_hash != expected_hash:
            return PackageState.INSTALLED_MODIFIED

    return PackageState.INSTALLED_FRESH
```

### 4.3 Smart Sync Behavior

```python
def sync_dependency(dep: ProjectDependency) -> SyncResult:
    state = get_package_state(dep)

    match state:
        case PackageState.NOT_INSTALLED:
            dep.install()
            return SyncResult.INSTALLED

        case PackageState.INSTALLED_FRESH:
            return SyncResult.ALREADY_UP_TO_DATE

        case PackageState.INSTALLED_WRONG_VERSION:
            dep.install()  # Re-install correct version
            return SyncResult.UPDATED

        case PackageState.INSTALLED_MODIFIED:
            # DON'T silently overwrite!
            raise PackageModifiedError(
                f"Package {dep.identifier} has local modifications.\n"
                f"Options:\n"
                f"  1. Run 'ato sync --force' to overwrite changes\n"
                f"  2. Run 'ato sync --keep-local' to keep your changes\n"
                f"  3. Manually backup and restore after sync"
            )

        case PackageState.INSTALLED_NO_META:
            # Legacy package - warn and upgrade
            logger.warning(f"Upgrading legacy package: {dep.identifier}")
            dep.install()
            return SyncResult.UPGRADED
```

### 4.4 Lock File for Reproducible Builds

Create `ato.lock` alongside `ato.yaml`:

```yaml
# Auto-generated by ato sync. Do not edit manually.
# This file ensures reproducible builds across machines.

lockfile_version: 1
generated_at: 2024-01-15T10:30:00Z
atopile_version: 0.14.0

packages:
  atopile/buttons:
    version: "0.3.1"
    source: registry
    integrity: sha256-abc123...
    requires_atopile: "^0.14.0"

  atopile/resistors:  # Transitive dependency
    version: "0.2.0"
    source: registry
    integrity: sha256-def456...
    required_by:
      - atopile/buttons@0.3.1

  local/my-lib:
    version: "0.0.0"
    source: file
    path: ../my-lib
    content_hash: sha256-ghi789...  # Hash of directory contents

  github/some-pkg:
    version: "1.0.0"
    source: git
    repo: https://github.com/org/repo.git
    commit: abc123def456...  # Pinned to exact commit
    path_within_repo: packages/some-pkg
```

### 4.5 UI Integration for Fresh Clone Scenario

When a user opens a project with unsynced dependencies:

```typescript
// In extension activation
async function checkDependencyState(projectRoot: string): Promise<DependencyCheckResult> {
  const result = await sendActionWithResponse('checkDependencies', { projectRoot });
  return result;
}

// Result might be:
{
  status: 'needs_sync',
  summary: {
    missing: ['atopile/buttons@0.3.1', 'atopile/resistors@0.2.0'],
    outdated: [],
    modified: [],
    total: 2
  },
  suggestedAction: 'sync'
}
```

Show a notification:
```
📦 This project has 2 missing dependencies.
[Sync Dependencies] [View Details] [Dismiss]
```

### 4.6 New CLI Flags

```bash
# Standard sync - errors on modified packages
ato sync

# Force overwrite local modifications
ato sync --force

# Keep local modifications (skip those packages)
ato sync --keep-local

# Show what would happen without doing it
ato sync --dry-run

# Verify integrity without installing
ato sync --check

# Update lock file without installing
ato sync --lock-only
```

---

## Part 5: Implementation Phases

### Phase 1: Metadata Foundation (Low Risk)
1. Create `PackageMeta` dataclass
2. Write metadata file on package install
3. Add `get_package_state()` function
4. Update `ato sync` to use state checking

### Phase 2: User Warnings (Medium Risk)
1. Detect modified packages
2. Add `--force` and `--keep-local` flags
3. Show warnings in CLI
4. Add UI notifications for sync needed

### Phase 3: Lock File (Medium Risk)
1. Design lock file schema
2. Generate lock file on install/sync
3. Read lock file during sync for reproducibility
4. Add `--lock-only` flag

### Phase 4: UI Polish (Low Risk)
1. "Sync Needed" badge in packages panel
2. Detailed sync preview dialog
3. Per-package sync/skip controls
4. Progress tracking for large syncs

---

## Part 6: Migration Strategy

### Existing Projects
When encountering a package without `.package_meta.json`:
1. Mark as `INSTALLED_NO_META`
2. On next sync, generate metadata from current state
3. Log: "Upgraded package {name} to tracked format"

### Lock File Adoption
1. If no `ato.lock` exists, sync creates one
2. If `ato.lock` exists but is outdated, warn and update
3. CI/CD can use `ato sync --check` to fail on drift

---

## Appendix: Comparison Table

| Feature | Current | Proposed | Parts Lifecycle |
|---------|---------|----------|-----------------|
| Version tracking | ✓ | ✓ | ✓ |
| Integrity checksums | ✗ | ✓ | ✓ |
| Modification detection | ✗ | ✓ | ✓ |
| Source provenance | Partial | ✓ | ✓ |
| Transitive pinning | ✗ | ✓ (lock file) | N/A |
| Graceful degradation | ✗ | ✓ | ✓ |
| User warnings | Minimal | ✓ | ✓ |
| Atomic updates | ✓ | ✓ | ✓ |

---

## Questions for Discussion

1. **Lock file format**: YAML (readable) vs JSON (simpler) vs TOML (trendy)? - We should use TOML
2. **Checksum scope**: Hash entire package or individual files?
3. **Modified package handling**: Default to error or warning?
4. **Git dependencies**: Always pin to commit hash on install?
5. **File dependencies**: Calculate content hash or skip integrity checks?

---

## Implementation Status

### Completed (Phase 1 + Partial Phase 2)

#### Backend - Package Metadata System
- [x] **`src/faebryk/libs/package/meta.py`** (new file)
  - `PackageMeta` dataclass with schema version, identifier, version, source, timestamps
  - `PackageSource` dataclass tracking registry/git/file origin
  - `PackageState` enum: `NOT_INSTALLED`, `INSTALLED_FRESH`, `INSTALLED_WRONG_VERSION`, `INSTALLED_MODIFIED`, `INSTALLED_NO_META`
  - `get_package_state()` function for checking package integrity
  - `PackageModifiedError` with helpful escape-hatch message
  - SHA256 checksums computed for all package files

- [x] **`src/faebryk/libs/package/dist.py`**
  - `Dist.install()` now accepts `PackageSource` parameter
  - Writes `.package_meta.json` after extraction with file checksums

- [x] **`src/faebryk/libs/project/dependencies.py`**
  - Added `get_package_source()` method to `ProjectDependency`
  - Updated all install calls to pass source info
  - Rewrote `sync_versions(force=False)` to use state-based checking
  - Added `force_sync` parameter to `ProjectDependencies.__init__`

- [x] **`src/atopile/cli/install.py`**
  - Added `--force` / `-f` flag to `ato sync` command
  - Catches `PackageModifiedError` and displays user-friendly message

- [x] **`src/atopile/dataclasses.py`**
  - Added `status` field to `DependencyInfo`

- [x] **`src/atopile/server/domains/projects.py`**
  - Added `_get_dependency_status()` helper
  - Dependencies API now returns `status` field

#### Frontend - Status Badges
- [x] **`src/ui-server/src/types/build.ts`**
  - Added `PackageStatus` type
  - Added `status` field to `ProjectDependency` interface

- [x] **`src/ui-server/src/components/PackagesPanel.tsx`**
  - Added `PackageStatusBadge` component with colored icons:
    - ✓ Green (`installed_fresh`): Up to date
    - ⚠ Yellow (`modified`): Locally modified
    - ⚠ Peach (`no_meta`): Untracked - needs sync
    - ✗ Red (`wrong_version`, `not_installed`): Version mismatch

- [x] **`src/ui-server/src/components/PackagesPanel.css`**
  - Added status badge styles

#### User Experience
- [x] Modified package error message suggests copying to project directory
- [x] `--force` flag provides escape hatch for intentional overwrites
- [x] Tooltips explain each status badge

---

### Outstanding Tasks

#### Phase 2 Completion
- [ ] **`--keep-local` flag**: Skip modified packages instead of erroring
- [ ] **`--dry-run` flag**: Show what would happen without making changes
- [ ] **`--check` flag**: Verify integrity without installing (for CI)
- [ ] **UI notification on clone**: Prompt user to sync when opening project with missing deps

#### Phase 3: Lock File
- [ ] Design `ato.lock` schema (YAML recommended for readability)
- [ ] Generate lock file on `ato sync` with transitive dependency versions
- [ ] Read lock file during sync for reproducibility
- [ ] Pin git dependencies to exact commit hash
- [ ] `--lock-only` flag to update lock without installing

#### Phase 4: UI Polish
- [ ] Aggregate status indicator (e.g., "2 packages need attention")
- [ ] Sync button in packages panel header
- [ ] Per-package actions (sync individual, view diff, copy to project)
- [ ] Progress tracking for large syncs

#### Future Considerations
- [ ] Hash verification for downloaded distributions (API returns SHA256)
- [ ] Content hash for `file://` dependencies
- [ ] Automatic sync prompt on project open
- [ ] VS Code problem matcher integration for modified package warnings

---

## CLI Flag Summary

| Flag | Status | Description |
|------|--------|-------------|
| `--upgrade` / `-U` | ✅ Existing | Update to latest compatible versions |
| `--pin` / `-p` | ✅ Existing | Pin unpinned versions to current |
| `--force` / `-f` | ✅ New | Overwrite locally modified packages |
| `--keep-local` | ❌ TODO | Skip modified packages (don't error) |
| `--dry-run` | ❌ TODO | Preview changes without applying |
| `--check` | ❌ TODO | Verify integrity only (for CI) |
| `--lock-only` | ❌ TODO | Update lock file without installing |
