# Fuzzy Matching Engine — Setup Guide

## Prerequisites

- [ ] The `[matching]` extra installed: `uv add entr-adapter-core[matching]`
- [ ] Entity reference data available (list of dicts from database, API, CSV, etc.)
- [ ] Field mapping defined: which field is the unique ID, which is the description to match against
- [ ] Matching thresholds agreed with the client (or use defaults: 90.0 high, 75.0 low)

## Configuration

The matching engine uses a `MatchConfig` Pydantic model — no env vars needed. Configure in Python:

```python
from entr_adapter_core.utils.matching import MatchConfig

match_config = MatchConfig(
    id_field="ItemCode",              # Unique ID field in entity dicts
    description_field="Description",  # Text field to match against
    search_fields=["SearchCode"],     # Additional fields for candidate search (optional)
    score_threshold=90.0,             # High confidence threshold
    high_confidence_threshold=95.0,   # Very high confidence threshold
    low_confidence_threshold=75.0,    # Low confidence threshold
    min_tokens_to_match=2,            # Minimum query tokens before matching
    synonym_map={                     # Tenant-specific synonyms (optional)
        "bubble": "bubbel",
        "stretch": "wikkelfolie",
        "cardboard": "karton",
    },
    stopwords={                       # Words to ignore during matching (optional)
        "de", "het", "een", "van",
        "the", "and", "for", "with",
    },
)
```

### Configuration Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `id_field` | str | `"id"` | Unique identifier field name in entity dicts |
| `description_field` | str | `"description"` | Primary text field to match against |
| `search_fields` | list[str] | `[]` | Additional fields included in token-based candidate search |
| `score_threshold` | float | `90.0` | Score at or above = "high confidence" |
| `high_confidence_threshold` | float | `95.0` | Score at or above = "very high confidence" |
| `low_confidence_threshold` | float | `75.0` | Score below = "low confidence" |
| `min_tokens_to_match` | int | `2` | Minimum tokens in query before matching is attempted |
| `synonym_map` | dict | `{}` | Maps variant spellings to canonical forms |
| `stopwords` | set | `{}` | Tokens to remove during normalization |
| `scorer_weights` | dict | See below | Weights for the four rapidfuzz scorers |
| `max_tokens` | int | `9` | Maximum query tokens (longest preferred) |
| `max_combinations_per_level` | int | `5000` | Cap on token combinations per level |

**Default scorer weights:**

```python
{"wratio": 0.30, "token_sort": 0.25, "token_set": 0.35, "partial": 0.10}
```

## Registration

Use the matching engine in your adapter's handler:

```python
from entr_adapter_core.handlers import BaseHandler
from entr_adapter_core.schemas import AdapterResponse, DocumentPayload, StatusEnum
from entr_adapter_core.utils.matching import MatchingEngine, MatchConfig

class MyHandler(BaseHandler):
    def __init__(self, settings, **kwargs):
        super().__init__(settings, **kwargs)

        self.match_config = MatchConfig(
            id_field="ItemCode",
            description_field="Description",
            score_threshold=90.0,
        )
        self.matcher = MatchingEngine(self.match_config)

        # Load reference data (from database, API, etc.)
        self.entities = []  # Populated at startup or on first use

    async def process(self, payload: DocumentPayload) -> AdapterResponse:
        # Load entities if not cached
        if not self.entities:
            self.entities = await self._load_entities()

        # Match each line item from the document
        for item in payload.data.get("line_items", []):
            description = item.get("description", "")
            result = self.matcher.match(description, self.entities)

            if result and result.confidence in ("very_high", "high"):
                item["matched_code"] = result.entity_id
                item["match_score"] = result.score
            elif result:
                item["matched_code"] = result.entity_id
                item["match_score"] = result.score
                item["needs_review"] = True

        return AdapterResponse(status=StatusEnum.SUCCESS, message="Processed")
```

## Verification

**1. Test matching in a Python shell:**

```python
from entr_adapter_core.utils.matching import MatchingEngine, MatchConfig

config = MatchConfig(id_field="code", description_field="name")
engine = MatchingEngine(config)

entities = [
    {"code": "A001", "name": "Blue bubble wrap 50x60"},
    {"code": "A002", "name": "Brown cardboard box 40x30"},
]

result = engine.match("bubble wrap blue", entities)
print(f"Match: {result.entity_id}, Score: {result.score}, Confidence: {result.confidence}")
```

**2. Expected output:**

```
Match: A001, Score: 92.5, Confidence: high
```

**3. Expected log output:**

```
INFO  MATCH: 'Blue bubble wrap 50x60' (id=A001) score=92.5 (high)
```

**4. Common failure modes:**

| Error | Cause | Fix |
|-------|-------|-----|
| `ImportError: ...rapidfuzz...` | Missing extra | Run `uv add entr-adapter-core[matching]` |
| Returns `None` | Query has fewer tokens than `min_tokens_to_match` | Lower `min_tokens_to_match` or use longer queries |
| All results are "low" confidence | Thresholds too high or synonyms not configured | Lower thresholds or add domain-specific synonyms |
| Slow matching on large catalogs | Too many entities or tokens | Reduce `max_tokens` or `max_combinations_per_level` |

## Common Pitfalls

1. **Threshold tuning**: Start with defaults (90/75) and adjust based on real data. Too strict = missed matches, too loose = false positives. Log scores for the first batch to calibrate.
2. **Synonym maps matter**: The engine normalizes text before scoring. Without synonyms, "bubble wrap" won't match "bubbel folie". Add domain-specific synonyms for each tenant.
3. **Stopwords**: Remove filler words (articles, prepositions, units) that add noise. Dutch: `de, het, een, van`. English: `the, and, for, with`. Units: `mm, cm, st, pcs`.
4. **Entity data freshness**: The engine matches against whatever entities you pass in. Stale data = wrong matches. Refresh entity data regularly (daily or per-run).
5. **Field name mismatches**: If `id_field` or `description_field` don't match the actual keys in your entity dicts, the engine returns empty strings for IDs and descriptions. Double-check field names.
6. **Single-token queries**: By default, queries with only 1 token are still matched. Set `min_tokens_to_match=2` to skip very short queries that produce unreliable matches.
7. **Accents and diacritics**: The normalizer strips accents (`cafe` == `cafe`). No special configuration needed.

## Client Onboarding Template

Send this checklist to the client's IT team before integration:

- [ ] **Reference data export**: provide a full export of entities to match against (e.g., product catalog, customer list)
- [ ] **Export format**: CSV, JSON, or API endpoint that returns entity data
- [ ] **Field mapping**: which field is the unique ID? Which is the description/name?
- [ ] **Update frequency**: how often does the reference data change? (determines cache strategy)
- [ ] **Matching scope**: which document fields should be matched? (e.g., product descriptions, vendor names)
- [ ] **Confidence thresholds**: what score is acceptable for auto-matching vs manual review?
- [ ] **Synonym list**: are there known abbreviations or alternative names in the domain? (e.g., "BB" = "Bubble Wrap")
- [ ] **Handling of unmatched items**: should unmatched items be flagged for review, rejected, or assigned a default?
- [ ] **Multiple match handling**: if multiple entities score above threshold, should the best be auto-selected or all shown for review?
