# Gap Analysis: gpa-parallel-njobs

## 1. Current State Investigation

### Existing Assets

| Asset | Location | Relevance |
|-------|----------|-----------|
| `GeneralizedProcrustesAnalysis` class | `ktch/landmark/_procrustes_analysis.py:30-550` | **Primary target** — `n_jobs` parameter declared (line 113) and stored (line 124), but never wired to `Parallel`/`delayed` |
| `EllipticFourierAnalysis` with `n_jobs` | `ktch/harmonic/_elliptic_fourier_analysis.py` | **Reference pattern** — working `Parallel`/`delayed` integration |
| `SphericalHarmonicAnalysis` with `n_jobs` | `ktch/harmonic/_spherical_harmonic_analysis.py` | **Reference pattern** — same approach |
| Existing test suite | `ktch/landmark/tests/test_procrustes_analysis.py` | **Extend** — comprehensive tests exist but no `n_jobs` parametrization |
| EFA `n_jobs` test | `ktch/harmonic/tests/test_elliptic_fourier_analysis.py:88-106` | **Test pattern reference** — `@pytest.mark.parametrize("n_jobs", [None, 1, 3])` |

### Established Parallelization Pattern

From EFA and SPHARM:
```python
from sklearn.utils.parallel import Parallel, delayed

# In transform methods:
results = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(
    delayed(self._process_single)(X_[i], ...) for i in range(len(X_))
)
```

Key conventions:
- Import from `sklearn.utils.parallel` (not `joblib` directly)
- Pass `n_jobs=self.n_jobs` and `verbose=self.verbose`
- Use `delayed` wrapper on a per-specimen processing function
- Collect results with `np.stack()` or `np.array()`

### GPA-Specific Constraints

The GPA algorithm is **iterative** (while loop until convergence). Parallelization targets are the per-specimen operations **within each iteration**, not across iterations (which are sequential by nature due to mean shape dependency).

**`_transform_shape` (shape mode, no semilandmarks)**:
- Line 277: `results = [sp.spatial.procrustes(mu, x) for x in X_]` — **primary target**, embarrassingly parallel per-specimen Procrustes alignment

**`_transform_shape` (shape mode, with semilandmarks)**:
- Lines 259-263: per-specimen `orthogonal_procrustes` + rotation — parallelizable
- Lines 268-274: per-specimen center + scale — parallelizable
- `_slide_semilandmarks` (line 266): calls per-specimen `_slide_procrustes`/`_slide_bending_energy` + `_reproject_onto_curves` — parallelizable

**`_transform_size_and_shape` (size-and-shape mode)**:
- Line 303: per-specimen `orthogonal_procrustes` — parallelizable
- **Note**: This method has a bug: references `self.n_specimen_` and `self.n_landmarks_` which are never set. This must be addressed.

### Identified Bug

`_transform_size_and_shape` at line 315 references `self.n_specimen_` and `self.n_landmarks_` but these attributes are never assigned anywhere in the class. This method will currently raise `AttributeError` at runtime. This must be fixed as part of the implementation (or at minimum, documented as out-of-scope).

## 2. Requirements Feasibility Analysis

### Requirement-to-Asset Map

| Requirement | Existing Asset | Gap |
|-------------|---------------|-----|
| **Req 1**: Parallel shape alignment (no semilandmarks) | Loop at line 277 | **Missing**: `Parallel`/`delayed` wrapping, need to extract single-specimen function |
| **Req 2**: Parallel size-and-shape alignment | Loop at line 303 | **Missing**: Same as Req 1 + **Bug**: broken attribute references (`self.n_specimen_`, `self.n_landmarks_`) |
| **Req 3**: Parallel semilandmark sliding | Loop at lines 259-263, 268-274, and `_slide_semilandmarks` line 540-542 | **Missing**: `Parallel`/`delayed` wrapping for 3 separate loops |
| **Req 4**: API consistency | `self.n_jobs` stored but no import of `Parallel`/`delayed` | **Missing**: import statement, `verbose` parameter (EFA/SPHARM use `verbose`) |
| **Req 5**: Testing | Existing tests at `test_procrustes_analysis.py` | **Missing**: `n_jobs` parametrized test variants |

### Notable Gaps

1. **No `verbose` parameter**: EFA and SPHARM both pass `verbose=self.verbose` to `Parallel`. GPA does not have a `verbose` parameter (only `debug`). Decision needed: add `verbose` for consistency or reuse `debug`.

2. **`_transform_size_and_shape` is broken**: The reshape at line 315 uses `self.n_specimen_` and `self.n_landmarks_` which don't exist. This is orthogonal to parallelization but blocks Req 2 testing.

3. **Iteration-internal parallelization**: Unlike EFA/SPHARM (which parallelize the entire `transform` call across specimens), GPA parallelizes **within** each iteration of a convergence loop. The overhead of `Parallel` per iteration could degrade performance for small datasets. **Research Needed**: Whether `Parallel` overhead is acceptable per-iteration, or if a single-dispatch pattern should be used.

4. **Multiple loops in semilandmark path**: The semilandmark branch has 3 separate per-specimen loops (rotation, sliding, center+scale). These could be consolidated into a single per-specimen function to minimize `Parallel` dispatch overhead.

### Complexity Signals

- **Algorithmic logic**: Wrapping existing for-loops with `Parallel`/`delayed`
- **Pattern reuse**: Direct pattern from EFA/SPHARM
- **No external integrations**: Pure internal refactoring

## 3. Implementation Approach Options

### Option A: Minimal Wiring (Extend Existing)

**Strategy**: Add `Parallel`/`delayed` wrapping directly in existing methods with minimal structural change.

- Add `from sklearn.utils.parallel import Parallel, delayed`
- Wrap the list comprehension at line 277 with `Parallel(n_jobs=self.n_jobs)`
- Wrap the for-loops in semilandmark path similarly
- Fix `_transform_size_and_shape` attribute bug
- Add `n_jobs` parametrized tests

**Trade-offs**:
- ✅ Minimal code change, low risk
- ✅ Follows existing codebase patterns exactly
- ❌ Multiple `Parallel` dispatches per iteration in semilandmark path (3 loops → 3 dispatches per iteration)
- ❌ Potential performance overhead for small specimen counts

### Option B: Consolidated Per-Specimen Functions (Hybrid)

**Strategy**: Extract per-specimen work into dedicated `_align_single` / `_align_single_with_semilandmarks` functions (analogous to EFA's `_transform_single_2d`), then parallelize the single function call.

- Create `_align_single_shape(x, mu)` → returns aligned specimen
- Create `_align_single_semilandmark(x, mu, curves, x_curve_geom)` → returns aligned + slid specimen
- Wrap each with a single `Parallel` dispatch per iteration
- Fix `_transform_size_and_shape` attribute bug

**Trade-offs**:
- ✅ Single `Parallel` dispatch per iteration (less overhead)
- ✅ Cleaner separation of per-specimen logic
- ✅ Closer to EFA/SPHARM pattern (`_transform_single_*`)
- ❌ More structural refactoring
- ❌ Semilandmark path has interleaved state (curve_geom updates), making consolidation non-trivial

### Option C: Conditional Parallelization

**Strategy**: Same as Option A or B, but add a minimum specimen threshold below which `Parallel` is skipped to avoid overhead.

- Only invoke `Parallel` when `n_specimens > threshold` (e.g., 10)
- Fall back to sequential loop otherwise

**Trade-offs**:
- ✅ Avoids overhead for small datasets
- ❌ Extra complexity and branching
- ❌ sklearn convention is to let `Parallel(n_jobs=None)` handle this automatically (it runs sequentially)
- ❌ Non-standard pattern for this project

## 4. Effort and Risk Assessment

**Effort: S (1–3 days)**
- Well-established pattern to follow (EFA/SPHARM reference)
- No new dependencies
- Limited to a single file + test file

**Risk: Low**
- Familiar technology (`sklearn.utils.parallel`)
- Existing test infrastructure
- Clear reference implementations in the same codebase
- One minor risk: `_transform_size_and_shape` bug may slightly expand scope

## 5. Recommendations for Design Phase

### Preferred Approach
**Option A (Minimal Wiring)** is recommended as the primary approach. The `Parallel` overhead concern (Option C) is handled by sklearn's built-in behavior (`n_jobs=None` → sequential). If profiling later shows overhead issues with the semilandmark path, consolidation (Option B) can be done as a follow-up.

### Key Decisions for Design
1. **`verbose` parameter**: Should GPA add a `verbose` parameter for API consistency with EFA/SPHARM? Or pass `verbose=0` to `Parallel`?
2. **`_transform_size_and_shape` bug**: Fix as part of this spec (small scope expansion) or separate issue?
3. **Semilandmark loop consolidation**: Wrap each of the 3 loops independently, or consolidate into a single per-specimen function?

### Research Items
- None — all patterns are well-established in the codebase and no external research is needed.
