---
name: duplicates
source: botcore
description: >
  Find code duplication using hybrid analysis (algorithmic + semantic).
  Use JSCPD for Type-1/2/3 clones, AI analysis for Type-4 semantic clones.
  Integrates with MCP discovery for cross-repo analysis.
version: "1.0.0"
triggers:
  - duplicates
  - duplication
  - copy paste
  - code clone
  - DRY
  - refactor
  - shared utility
---

# Duplicates

Find and eliminate code duplication using hybrid analysis (2025-2026 best practices).

## The Duplication Problem in AI Codebases

> AI agents create flat, duplicative codebases because:
> - They don't know existing utilities unless in context
> - They implement first, don't search for existing code
> - No refactoring instinct when patterns emerge

**Result:** Code bloat, inconsistency, harder maintenance.

---

## Detection Strategy: Three-Tier Approach

| Tier | Type | Detection Method | Speed |
|------|------|-----------------|-------|
| **1. Syntactic** | Type-1/2/3 | JSCPD (token-based) | Fast (~3s) |
| **2. Semantic** | Type-4 | Vector similarity (AST embeddings) | Slow (~30s) |
| **3. Agentic** | Prevention | MCP discovery before generation | Real-time |

### Clone Types

| Type | Description | Detection |
|------|-------------|-----------|
| **Type-1** | Exact copy | JSCPD |
| **Type-2** | Same logic, different variable names | JSCPD |
| **Type-3** | Similar structure, minor changes | JSCPD |
| **Type-4** | Same behavior, different implementation | AI/Vector DB |

---

## Tool: JSCPD (Algorithmic Detection)

For Type-1/2/3 clones, use JSCPD via `lush discover duplicates`:

```bash
# Scan current directory
lush discover duplicates

# Scan specific path
lush discover duplicates --path src/

# With threshold
lush discover duplicates --threshold 3
```

### Configuration (`.jscpd.json`)

```json
{
  "threshold": 5,
  "reporters": ["json", "console"],
  "ignore": [
    "**/node_modules/**",
    "**/.git/**",
    "**/dist/**",
    "**/__pycache__/**"
  ],
  "mode": "mild"
}
```

**Modes:**
- `strict` — Exact matching only
- `mild` — Ignores whitespace/formatting
- `weak` — Ignores comments and variable names (finds Type-2)

---

## AI-Based Detection (Semantic)

For Type-4 clones (same behavior, different code), use the `duplicates` subagent:

```bash
# Run via subagent (AI analysis)
lush-subagent run role='duplicates'
```

This uses LLM reasoning to find:
- Functions that accomplish the same goal differently
- Reimplementations of existing utilities
- Cross-repo duplication

---

## MCP Discovery Layer (Prevention)

The best duplication is the one that never happens. Use MCP-based discovery:

```python
# Before writing new code, search for existing
@mcp.tool()
def search_utilities(query: str, language: str = "python") -> str:
    """
    Find existing utilities before creating new ones.
    Use this BEFORE implementing common functionality.
    """
    # Vector search against codebase embeddings
    results = vector_db.query(query, lang=language)
    return format_results(results)
```

**Agent instruction:**
> Before creating utility functions, call `search_utilities` to check 
> if similar functionality exists. Reuse existing code when found.

---

## Analysis Workflow

### 1. Run Algorithmic Detection

```bash
lush discover duplicates --path src/
```

Output identifies exact and near-miss clones with locations.

### 2. Run Semantic Analysis (Optional)

For deeper analysis including Type-4 clones:

```bash
lush-subagent run role='duplicates'
```

### 3. Prioritize by Value

| Priority | Criteria | Action |
|----------|----------|--------|
| 🔴 High | 3+ occurrences, >20 lines each | Extract to shared utility |
| 🟡 Medium | 2 occurrences, >10 lines each | Consider extraction |
| 🟢 Low | Small blocks, edge cases | Document, defer |

---

## Safe Refactoring: Red-Green-Refactor

When consolidating duplicates, follow the TDD-inspired workflow:

### 1. Red (Write Tests First)

```python
# Create tests that cover BOTH duplicate implementations
def test_format_date_legacy():
    """Test existing formatDate in utils/date.ts"""
    assert format_date_legacy("2026-01-11") == "Jan 11, 2026"

def test_format_date_new():
    """Test duplicate in components/calendar.ts"""
    assert format_date_new("2026-01-11") == "Jan 11, 2026"
```

### 2. Green (Verify Tests Pass)

Run tests to confirm they pass against current implementations.

### 3. Refactor (Create Unified Utility)

```python
# Create single shared utility
def format_date(date_str: str) -> str:
    """Unified date formatter. Replaces duplicates in:
    - utils/date.ts
    - components/calendar.ts
    """
    ...
```

### 4. Update Call Sites

Replace all duplicate usages with the new shared utility.

### 5. Verify (Tests Still Pass)

The same tests from step 1 must pass against the unified implementation.

---

## Output Format

```markdown
# Duplicate Analysis: [Project Name]

## 🔴 High-Value Refactoring (3+ occurrences)
1. `formatDate` (3 occurrences, 95% similar)
   - `src/utils/date.ts:12-28`
   - `src/components/calendar.ts:45-61`
   - `src/api/events.ts:89-105`
   → **Suggestion:** Extract to `shared/date-utils.ts`

## 🟡 Moderate Duplication (2 occurrences)
1. `validateEmail` (2 occurrences, 90% similar)
   - `src/auth/signup.ts:34-42`
   - `src/settings/profile.ts:67-75`
   → **Suggestion:** Consider extraction if reused again

## 🟢 Potential Shared Utilities
- Error handling pattern appears in 5+ files
- API response parsing could be standardized

## Summary
Found 12 duplicate patterns totaling ~340 lines. Extracting high-value 
duplicates could reduce codebase by ~180 lines (5% reduction).
```

---

## Cross-Repo Detection

For multi-repo ecosystems (AFD, fast-af, lushly):

### Current: Single-Repo Focus
Phase 1 focuses on within-repo detection.

### Future: MCP Multi-Repo Mounting
```python
# Mount multiple repos for unified indexing
mcp_server.mount("/afd", path="/d/Github/lushly-dev/AFD")
mcp_server.mount("/fast-af", path="/d/Github/lushly-dev/fast-af")
mcp_server.mount("/lushly", path="/d/Github/lushly-dev/lushly")

# Cross-repo search
results = search_utilities("format currency", repos=["afd", "fast-af"])
```

---

## Detailed References

- `docs/research/duplicates-research-request.md` — Full 2026 duplication research
- `.agent/roles/duplicates.yaml` — AI-based semantic detection role
- `docs/features/proposed/discover-duplicates.spec.md` — JSCPD implementation spec
