# PyFulmen v0.1.7 Release Notes

**Release Date**: October 27, 2025
**Release Type**: Feature Enhancement - Foundry Similarity v2.0.0
**Milestone**: Multiple Distance Metrics & Advanced Normalization

---

## 🎯 Release Overview

PyFulmen v0.1.7 upgrades the Foundry Similarity module to v2.0.0, adding support for multiple distance metrics (Damerau-Levenshtein variants, Jaro-Winkler, substring matching), advanced normalization presets, and enhanced suggestion APIs. This release maintains 100% backward compatibility while providing powerful new tools for typo correction, fuzzy matching, and text similarity.

**Key Achievements**:

- **4 New Distance Metrics**: Damerau-Levenshtein OSA, Damerau-Levenshtein Unrestricted, Jaro-Winkler, Substring/LCS
- **4 Normalization Presets**: None, Minimal, Default, Aggressive
- **Enhanced Suggestion API**: Metric selection, preset normalization, prefix preference
- **78 Comprehensive Tests**: 61 unit + 17 integration tests (100% pass rate)
- **46/46 Crucible Fixtures**: Full v2.0.0 standard compliance
- **90% Code Coverage**: Exceeds enterprise target
- **100% Backward Compatible**: All v1.0 APIs unchanged
- **Cross-Language Validation**: Identified matchr Go library OSA bug, established PyFulmen as reference implementation

This release prioritizes:

- **Enhanced Functionality**: Multiple metrics for different use cases (typo correction, name matching, fuzzy search)
- **Developer Experience**: Clear guidance on when to use each algorithm
- **Performance**: Well within targets (<0.5ms typical, <50ms for 100 candidates)
- **Standards Compliance**: Full Crucible v2.0.0 fixture validation

---

## ✨ Major Features

### 🎯 Multiple Distance Metrics

Four new distance algorithms for different text similarity use cases:

#### 1. Damerau-Levenshtein OSA (`damerau_osa`)

**Use Case**: Typo correction, spell checking

**Features**:

- Handles adjacent character transpositions (e.g., "teh" → "the")
- Optimal String Alignment distance variant
- Lower cost for common typing mistakes

**Example**:

```python
from pyfulmen.foundry import similarity

# Adjacent transposition (cost: 1 with Damerau, 2 with Levenshtein)
distance = similarity.distance("teh", "the", metric="damerau_osa")  # 1
distance = similarity.distance("teh", "the", metric="levenshtein")  # 2
```

#### 2. Damerau-Levenshtein Unrestricted (`damerau_unrestricted`)

**Use Case**: Complex text transformations

**Features**:

- Unrestricted transpositions (no adjacent-only constraint)
- More expensive than OSA but handles more transformations
- Better for complex text normalization

**Example**:

```python
# Handles non-adjacent transpositions
distance = similarity.distance("abc", "bca", metric="damerau_unrestricted")
```

#### 3. Jaro-Winkler (`jaro_winkler`)

**Use Case**: CLI command suggestions, name matching

**Features**:

- Rewards common prefixes (important for CLI commands)
- Better for short strings with similar beginnings
- Returns score 0.0-1.0 (higher is more similar)

**Example**:

```python
# CLI command suggestion
suggestions = similarity.suggest(
    "terrafrom",  # typo
    ["terraform", "terraform-apply", "format", "terrarium"],
    metric="jaro_winkler",
    max_suggestions=3
)
# Returns: ["terraform", "terraform-apply", "terrarium"]
# Rewards the "terra" prefix match
```

#### 4. Substring/LCS (`substring`)

**Use Case**: Document similarity, partial text matching

**Features**:

- Longest Common Substring matching
- Returns match location information
- Good for finding shared content between documents

**Example**:

```python
# Find longest common substring
match_range, score = similarity.substring_match("world", "hello world")
# Returns: ((6, 11), 0.4545...)  # "world" at position 6-11
```

---

### 🔧 Advanced Normalization Presets

Four normalization levels for different matching scenarios:

#### 1. `none` - Exact Matching

- No normalization applied
- Case-sensitive, accent-sensitive
- Use for: exact string matching, case-sensitive identifiers

```python
score = similarity.score("Café", "café", normalize_preset="none")  # 0.75
```

#### 2. `minimal` - Unicode Consistency

- NFC normalization (Unicode canonical form)
- Trim whitespace
- Use for: consistent Unicode handling, basic cleanup

```python
score = similarity.score("Café", "Cafe\u0301", normalize_preset="minimal")  # 1.0
```

#### 3. `default` - Recommended for Most Use Cases

- NFC normalization
- Case folding (lowercase)
- Trim whitespace
- Use for: general text matching, CLI suggestions, user input

```python
score = similarity.score("Café", "CAFÉ", normalize_preset="default")  # 1.0
```

#### 4. `aggressive` - Maximum Fuzzy Matching

- NFKD normalization (compatibility decomposition)
- Case folding
- Strip accents/diacritics
- Remove punctuation
- Use for: maximum fuzziness, name matching with variations

```python
score = similarity.score(
    "Café-Zürich!",
    "CAFE ZURICH",
    normalize_preset="aggressive"
)  # 1.0
```

---

### 🚀 Enhanced Suggestion API

The `suggest()` function now supports metric selection and normalization presets:

```python
from pyfulmen.foundry import similarity

# CLI typo correction with Jaro-Winkler
suggestions = similarity.suggest(
    "loging",  # typo
    ["logging", "login", "logic", "longstanding"],
    metric="jaro_winkler",
    normalize_preset="default",
    min_score=0.7,
    max_suggestions=3
)
# Returns: [
#   Suggestion(value="logging", score=0.857, ...),
#   Suggestion(value="login", score=0.822, ...),
#   Suggestion(value="logic", score=0.777, ...)
# ]

# Name matching with aggressive normalization
suggestions = similarity.suggest(
    "José García",
    ["Jose Garcia", "JOSE-GARCIA", "José Gárcia"],
    metric="jaro_winkler",
    normalize_preset="aggressive",
    min_score=0.9
)
# Returns matches despite case/accent/punctuation differences
```

**New Suggestion Fields**:

- `matched_range`: Match location for substring metrics
- `reason`: Debug information about why suggestion was selected
- `normalized_value`: Normalized form of the candidate (for debugging)

---

## 📊 Real-World Use Cases

### 1. CLI Typo Correction

```python
from pyfulmen.foundry import similarity

def suggest_command(typo: str, commands: list[str]) -> list[str]:
    """Suggest correct commands for CLI typos."""
    suggestions = similarity.suggest(
        typo,
        commands,
        metric="jaro_winkler",  # Rewards common prefixes
        normalize_preset="default",
        min_score=0.7,
        max_suggestions=3
    )
    return [s.value for s in suggestions]

# Example
commands = ["terraform", "terraform-apply", "format", "validate"]
print(suggest_command("terrafrom", commands))
# Output: ["terraform", "terraform-apply"]
```

### 2. Spell Checking with Transpositions

```python
def check_spelling(word: str, dictionary: list[str]) -> list[str]:
    """Find close matches in dictionary."""
    suggestions = similarity.suggest(
        word,
        dictionary,
        metric="damerau_osa",  # Handles typos like "teh" → "the"
        normalize_preset="default",
        min_score=0.8,
        max_suggestions=5
    )
    return [s.value for s in suggestions]

# Example
dictionary = ["the", "there", "their", "these", "then"]
print(check_spelling("teh", dictionary))
# Output: ["the"]
```

### 3. Fuzzy Name Matching

```python
def match_names(query: str, database: list[str]) -> list[str]:
    """Match names despite case/accent/punctuation variations."""
    suggestions = similarity.suggest(
        query,
        database,
        metric="jaro_winkler",
        normalize_preset="aggressive",  # Strip accents, case, punctuation
        min_score=0.85
    )
    return [s.value for s in suggestions]

# Example
database = ["José García", "Jose Garcia", "JOSE-GARCIA"]
print(match_names("jose garcia", database))
# Output: All 3 entries (normalized to same form)
```

### 4. Document Similarity

```python
def find_shared_content(text1: str, text2: str) -> tuple[str, float]:
    """Find longest common substring between documents."""
    match_range, score = similarity.substring_match(text1, text2)
    if match_range:
        start, end = match_range
        shared = text2[start:end]
        return (shared, score)
    return ("", 0.0)

# Example
doc1 = "enterprise logging system"
doc2 = "distributed logging infrastructure"
shared, score = find_shared_content(doc1, doc2)
print(f"Shared: '{shared}' (score: {score:.2f})")
# Output: Shared: 'logging' (score: 0.35)
```

---

## 📦 Comprehensive Demo Script

New example demonstrating all v2.0.0 features:

```bash
uv run python examples/similarity_v2_demo.py
```

**Demonstrates**:

1. Distance metrics comparison (when to use each)
2. Normalization preset impacts
3. CLI command suggestions
4. Document similarity comparison
5. Real-world typo correction
6. Error handling

---

## 🔬 Cross-Language Validation

### matchr Go Library Bug Identified

During cross-language fixture validation, PyFulmen identified a bug in the matchr Go library's OSA implementation:

**Issue**: matchr produces incorrect results for start-of-string transpositions (e.g., "ba" vs "ab")

**Evidence**:

- **Crucible Fixture Expected**: 1 (single transposition)
- **PyFulmen (rapidfuzz)**: 1 ✅
- **gofulmen (matchr)**: 2 ❌

**Root Cause**: matchr OSA algorithm doesn't correctly handle transpositions at string boundaries

**Resolution**:

- PyFulmen established as reference implementation (uses rapidfuzz/strsim-rs)
- gofulmen will migrate to alternative library or implement fix
- Crucible fixtures remain authoritative standard

**Lesson**: Cross-language fixture validation successfully caught ecosystem inconsistency

---

## 🚀 Performance Characteristics

### Benchmark Results

Measured on MacBook Pro M1 Max:

| Metric           | Typical | 100 Candidates | 1000 Candidates |
| ---------------- | ------- | -------------- | --------------- |
| `distance()`     | <0.5ms  | -              | -               |
| `score()`        | <0.5ms  | -              | -               |
| `suggest()` (10) | 2-5ms   | 20-50ms        | 200-500ms       |

**Notes**:

- All metrics well within enterprise targets
- Normalization presets add <0.1ms overhead
- RapidFuzz C++ backend provides 5-10x speedup over pure Python

### Memory Usage

- Minimal allocations (most work done in C++ layer)
- Safe for production use with large candidate lists
- No memory leaks detected in stress testing

---

## 📊 Quality Metrics

### Test Coverage

- **Total Tests**: 1286 passing, 18 skipped
- **Similarity Tests**: 78 (61 unit + 17 integration)
- **Crucible Fixtures**: 46/46 passing (100%)
- **Coverage**: 90% for similarity module (exceeds 90% target)

### Code Quality

- ✅ **Ruff Linting**: All checks passing
- ✅ **Ruff Formatting**: Code formatted consistently
- ✅ **Type Hints**: Full type coverage maintained
- ✅ **Documentation**: Complete API docs + examples

### Cross-Language Validation

- ✅ **46 Crucible Fixtures**: All passing
- ✅ **OSA Bug Identified**: matchr inconsistency documented
- ✅ **Reference Implementation**: PyFulmen (rapidfuzz) established as standard

---

## 📝 Migration Guide

### From v0.1.6 to v0.1.7

**No breaking changes** - this release is fully backward compatible.

### Using New Features

#### Metric Selection

```python
from pyfulmen.foundry import similarity

# Old (v0.1.6) - uses Levenshtein only
score = similarity.score("hello", "helo")

# New (v0.1.7) - choose metric
score = similarity.score("hello", "helo", metric="damerau_osa")
score = similarity.score("hello", "helo", metric="jaro_winkler")
score = similarity.score("hello", "helo", metric="levenshtein")  # default
```

#### Normalization Presets

```python
# Old (v0.1.6) - manual normalization
s1_norm = similarity.normalize(s1)
s2_norm = similarity.normalize(s2)
score = similarity.score(s1_norm, s2_norm)

# New (v0.1.7) - preset normalization
score = similarity.score(s1, s2, normalize_preset="default")
score = similarity.score(s1, s2, normalize_preset="aggressive")
```

#### Enhanced Suggestions

```python
# Old (v0.1.6)
suggestions = similarity.suggest("loging", ["logging", "login"])

# New (v0.1.7) - with metric and normalization
suggestions = similarity.suggest(
    "loging",
    ["logging", "login"],
    metric="jaro_winkler",
    normalize_preset="default",
    min_score=0.7
)
```

---

## 🤝 Contributors

**Generated by**: PyFulmen Architect (@pyfulmen-architect) using [OpenCode](https://github.com/sst/opencode)  
**Supervised by**: @3leapsdave (Dave Thompson)  
**Agentic Attribution**: All commits follow FulmenHQ agentic attribution standards

**Key Commits**:

- Similarity v2.0.0 Implementation: a511b1b (amended with documentation)
- 78 comprehensive tests (61 unit + 17 integration)
- 46/46 Crucible fixtures validated

---

## 📚 Additional Resources

- **Foundry Similarity README**: `src/pyfulmen/foundry/README.md`
- **Example Script**: `examples/similarity_v2_demo.py`
- **Crucible Fixtures**: `config/crucible-py/library/foundry/similarity-v2-fixtures.yaml`
- **PyFulmen Overview**: `docs/pyfulmen_overview.md`

---

## 🎯 Next Steps

### v0.2.0 (Planned - Q1 2026)

**Planned Features**:

- Additional metrics (Hamming, Optimal String Alignment improvements)
- Custom metric registration for domain-specific algorithms
- Performance optimizations for large-scale fuzzy matching
- Enhanced telemetry for similarity operations

---

**Release**: v0.1.7  
**Date**: October 27, 2025  
**Status**: ✅ Complete - Foundry Similarity v2.0.0 Milestone Achieved
