# pygarble

**Detect gibberish, garbled text, and corrupted content with high accuracy using advanced machine learning techniques.**

pygarble is a powerful Python library designed to identify nonsensical, garbled, or corrupted text content that often appears in data processing pipelines, user inputs, or automated systems. Whether you're dealing with random character sequences, encoding errors, keyboard mashing, or corrupted data streams, pygarble provides multiple detection strategies to filter out unwanted content and maintain data quality. The library uses statistical analysis, entropy calculations, pattern matching, and n-gram analysis to distinguish between meaningful text and gibberish with configurable sensitivity levels.

## Features

- **24 Detection Strategies**: Choose from multiple garble detection algorithms including Markov chains, n-gram analysis, mojibake detection, and homoglyph detection
- **Zero Dependencies**: Core library works without any external dependencies
- **Ensemble Detector**: Combine multiple strategies for higher accuracy with voting mechanisms
- **Scikit-learn Interface**: Familiar `predict()` and `predict_proba()` methods
- **Configurable Thresholds**: Adjust sensitivity for each strategy
- **Probability Scores**: Get confidence scores for garble detection
- **Input Validation**: Built-in validation for thresholds and parameters
- **Type Hints**: Full type annotation support throughout the codebase
- **Modular Design**: Easy to extend with new detection strategies
- **Smart Edge Cases**: Automatically detects extremely long strings without whitespace (like base64 data)

## Installation

You can install pygarble using pip:

```bash
# Core library (zero dependencies)
pip install pygarble

# With pyspellchecker for legacy word validation (optional)
pip install pygarble[spellchecker]
```

## Quick Start

```python
from pygarble import GarbleDetector, Strategy, EnsembleDetector

# RECOMMENDED: Default ensemble (99.5% precision, majority voting)
ensemble = EnsembleDetector()
print(ensemble.predict("hello world"))      # False
print(ensemble.predict("asdfghjkl"))        # True
print(ensemble.predict("xkjqzpvmw"))        # True (impossible bigrams)

# Individual strategies for specific use cases
detector = GarbleDetector(Strategy.MARKOV_CHAIN)  # Best overall (92% F1)
print(detector.predict("the quick brown fox"))    # False

detector = GarbleDetector(Strategy.BIGRAM_PROBABILITY)  # 100% precision
print(detector.predict("qxjjxz"))                 # True (impossible bigrams)

# Batch processing
texts = ["Hello world", "asdfghjkl", "qwertyuiop"]
results = ensemble.predict(texts)
print(results)  # [False, True, True]

# Get probability scores
probabilities = ensemble.predict_proba(texts)
print(probabilities)  # [0.0, 0.8, 0.6]
```

## Benchmark Results

Based on **1,644 test cases** (420 internal + 1,224 external validation):

### Top Strategy Performance

| Strategy | Accuracy | Precision | Recall | F1 Score |
|----------|----------|-----------|--------|----------|
| **markov_chain** | 93.19% | 98.80% | 86.39% | **92.18%** |
| **ensemble** | 89.84% | **99.50%** | 78.53% | **87.78%** |
| ngram_frequency | 87.41% | 96.34% | 75.79% | 84.84% |
| pronounceability | 81.08% | 84.47% | 72.64% | 78.11% |
| letter_position | 77.86% | 99.02% | 52.88% | 68.94% |
| bigram_probability | 69.16% | **100%** | 33.64% | 50.34% |

### High-Precision Strategies (v0.5.0)

New strategies designed for **maximum precision** (minimize false positives):

| Strategy | Precision | Target Use Case |
|----------|-----------|-----------------|
| **bigram_probability** | **100%** | Impossible letter pairs (qx, jj, xz) |
| **rare_trigram** | **100%** | Impossible trigrams (jjj, qqq, xqz) |
| **vowel_pattern** | **100%** | Invalid vowel sequences (aaaaaaa) |
| **letter_frequency** | **100%** | Abnormal letter distribution |
| **consonant_sequence** | 99.38% | Impossible consonant runs (6+ consonants) |
| **letter_position** | 99.02% | Letters in impossible positions |

### Default Ensemble Configuration

The default `EnsembleDetector()` uses **majority voting** with high-precision strategies:
- `MARKOV_CHAIN` (95% precision, 61% recall)
- `WORD_LOOKUP` (89% precision, 51% recall)
- `NGRAM_FREQUENCY` (88% precision, 47% recall)
- `BIGRAM_PROBABILITY` (100% precision, 25% recall)
- `LETTER_POSITION` (93% precision, 35% recall)

**Result: 99.5% precision** - only 3 false positives out of 1,644 test cases.

Run the benchmark yourself:
```bash
python regression/benchmark.py
```

## Detection Strategies

Each strategy implements a different approach to detect garbled text. All strategies return probability scores between 0.0 and 1.0, where higher scores indicate more likely garbled text.

### 1. Keyboard Pattern (`KEYBOARD_PATTERN`) ⭐ Best F1 Score

**Implementation Logic**: Detects keyboard row sequences (qwerty, asdf, zxcv) and analyzes trigram patterns. English text has predictable trigram distributions; garbled text doesn't.

**Algorithm**:
1. Extract trigrams from the text
2. Check for keyboard row sequences (forward and reverse)
3. Compare against common English trigrams
4. Detect repeated bigram patterns (ababab)

**Parameters**:
- `keyboard_threshold` (float, default: 0.3): Threshold for keyboard pattern ratio
- `common_trigram_threshold` (float, default: 0.1): Minimum common trigram ratio

```python
detector = GarbleDetector(Strategy.KEYBOARD_PATTERN, threshold=0.5)

# Examples
detector.predict("asdfghjkl")       # True - keyboard row pattern
detector.predict("qwertyuiop")      # True - keyboard row pattern
detector.predict("Hello world")     # False - normal English text
detector.predict("ababababab")      # True - repeated bigram pattern
```

### 2. Vowel Ratio (`VOWEL_RATIO`) ⭐ Best Precision

**Implementation Logic**: Analyzes the ratio of vowels to consonants. Natural English has 35-45% vowels. Also detects consonant clusters that are impossible in English.

**Algorithm**:
1. Calculate vowel ratio in alphabetic characters
2. Detect long consonant clusters (4+ consecutive consonants)
3. Flag text outside normal vowel ratio range (15-65%)

**Parameters**:
- `min_vowel_ratio` (float, default: 0.15): Minimum allowed vowel ratio
- `max_vowel_ratio` (float, default: 0.65): Maximum allowed vowel ratio
- `consonant_cluster_len` (int, default: 4): Max consonant cluster length

```python
detector = GarbleDetector(Strategy.VOWEL_RATIO, threshold=0.5)

# Examples
detector.predict("bcdfghjklmnp")    # True - no vowels
detector.predict("aeiouaeiou")      # True - all vowels
detector.predict("Hello world")     # False - normal vowel ratio (~36%)
detector.predict("rhythm")          # False - valid English word
```

### 3. Entropy Based (`ENTROPY_BASED`)

**Implementation Logic**: Uses Shannon entropy combined with bigram frequency analysis. English text has predictable character and bigram distributions.

**Algorithm**:
1. Calculate Shannon entropy of alphabetic characters
2. Analyze common English bigram frequency (th, he, in, er, etc.)
3. Combine entropy and bigram scores

**Parameters**:
- `entropy_threshold` (float, default: 2.5): Minimum required entropy
- `bigram_threshold` (float, default: 0.15): Minimum common bigram ratio

```python
detector = GarbleDetector(Strategy.ENTROPY_BASED, threshold=0.5)

# Examples
detector.predict("aaaaaaa")         # True - low entropy (repetitive)
detector.predict("xkjqzpv")         # True - no common bigrams
detector.predict("the weather")     # False - high common bigram ratio
```

### 4. Pattern Matching (`PATTERN_MATCHING`)

**Implementation Logic**: Uses regex patterns to detect suspicious sequences including keyboard rows, repeated characters, and consonant clusters.

**Default Patterns**:
- `special_chars`: 3+ special characters
- `repeated_chars`: 4+ repeated characters
- `uppercase_sequence`: 5+ uppercase letters
- `long_numbers`: 8+ consecutive digits
- `keyboard_row_qwerty`: Keyboard row sequences (qwert, asdf, zxcv)
- `keyboard_row_reverse`: Reverse keyboard sequences
- `consonant_cluster`: 5+ consecutive consonants
- `alternating_pattern`: Alternating character patterns (ababab)

```python
detector = GarbleDetector(Strategy.PATTERN_MATCHING, threshold=0.2)

# Examples
detector.predict("asdfghjkl")       # True - keyboard row
detector.predict("AAAAA")           # True - repeated chars
detector.predict("normal text")     # False - no patterns match
```

### 5. Markov Chain (`MARKOV_CHAIN`) ⭐ NEW - Recommended

**Implementation Logic**: Uses a character-level Markov chain trained on English text. Computes the probability of text based on character transition frequencies. Garbled text has unusual character transitions.

**Algorithm**:
1. Train bigram transition probabilities on 300K+ English words
2. Compute average log-probability of character transitions
3. Map to garble score using sigmoid function

**Parameters**:
- `threshold_per_char` (float, default: -3.5): Average log probability threshold

```python
detector = GarbleDetector(Strategy.MARKOV_CHAIN, threshold=0.5)

# Examples
detector.predict("hello world")       # False - common bigrams (he, el, ll, lo, ow, wo, or, rl, ld)
detector.predict("asdfghjkl")         # True - unusual bigrams (sd, df, fg, gh, hj, jk, kl)
detector.predict("xzqkjhf")           # True - rare character transitions
```

### 6. N-gram Frequency (`NGRAM_FREQUENCY`) ⭐ NEW

**Implementation Logic**: Analyzes what proportion of character trigrams appear in common English text. Uses a set of 2000 most common English trigrams.

**Algorithm**:
1. Extract trigrams from words
2. Count how many appear in common trigram set
3. Low ratio = likely garbled

**Parameters**:
- `common_ratio_threshold` (float, default: 0.3): Minimum ratio of common trigrams

```python
detector = GarbleDetector(Strategy.NGRAM_FREQUENCY, threshold=0.5)

# Examples
detector.predict("the quick brown")   # False - trigrams: the, qui, uic, ick, bro, row, own
detector.predict("xzqkjhf")           # True - no common trigrams
```

### 7. Word Lookup (`WORD_LOOKUP`) ⭐ NEW - Zero Dependencies

**Implementation Logic**: Validates words against an embedded dictionary of 50,000 common English words. No external dependencies required.

**Algorithm**:
1. Tokenize text into words
2. Check each word against embedded word set
3. Return ratio of unknown words

**Parameters**:
- `unknown_threshold` (float, default: 0.5): Ratio above which text is garbled

```python
detector = GarbleDetector(Strategy.WORD_LOOKUP, threshold=0.5)

# Examples
detector.predict("hello world")       # False - both words in dictionary
detector.predict("xyzzy plugh")       # True - neither word in dictionary
detector.predict("hello xyzzy")       # True (0.5) - half unknown
```

### 8. Symbol Ratio (`SYMBOL_RATIO`) - NEW

**Implementation Logic**: Detects text with high proportion of special characters, numbers, or non-alphabetic content. Particularly effective for symbol spam, number sequences, and mixed alphanumeric noise.

**Algorithm**:
1. Count non-alphabetic characters (excluding spaces)
2. Calculate ratio to total characters
3. High ratio = likely garbled

**Parameters**:
- `symbol_threshold` (float, default: 0.5): Ratio above which text is garbled
- `min_length` (int, default: 3): Minimum text length to analyze
- `allow_spaces` (bool, default: True): Whether to exclude spaces from ratio

```python
detector = GarbleDetector(Strategy.SYMBOL_RATIO, threshold=0.5)

# Examples
detector.predict("!!!@@@###$$$")     # True - all symbols
detector.predict("abc123def456")     # True - high number ratio
detector.predict("hello world")       # False - mostly alphabetic
```

### 9. Repetition (`REPETITION`) - NEW

**Implementation Logic**: Detects text with excessive character or pattern repetition. Identifies repeated single characters, bigrams, trigrams, and low character diversity.

**Algorithm**:
1. Check for repeated single characters (aaaa)
2. Check for repeated bigrams (ababab)
3. Check for repeated trigrams (abcabcabc)
4. Analyze character diversity

**Parameters**:
- `max_char_repeat` (int, default: 3): Maximum allowed consecutive repeated characters
- `max_pattern_repeat` (int, default: 3): Maximum allowed pattern repetitions
- `diversity_threshold` (float, default: 0.3): Minimum unique character ratio

```python
detector = GarbleDetector(Strategy.REPETITION, threshold=0.5)

# Examples
detector.predict("aaaaaaaaaa")        # True - repeated character
detector.predict("abababababab")      # True - repeated bigram
detector.predict("hello world")       # False - diverse characters
```

### 10. Hex String (`HEX_STRING`) - NEW

**Implementation Logic**: Detects hash strings, UUIDs, base64-like content, and other hexadecimal patterns commonly found in garbled data.

**Algorithm**:
1. Check for pure hash patterns (MD5, SHA256)
2. Detect UUID format (8-4-4-4-12)
3. Identify long hex sequences
4. Check for base64-like patterns

**Parameters**:
- `min_hex_length` (int, default: 16): Minimum hex sequence length to detect
- `hex_ratio_threshold` (float, default: 0.7): Ratio of hex chars above which text is suspicious

```python
detector = GarbleDetector(Strategy.HEX_STRING, threshold=0.5)

# Examples
detector.predict("5d41402abc4b2a76b9719d911017c592")  # True - MD5 hash
detector.predict("550e8400-e29b-41d4-a716-446655440000")  # True - UUID
detector.predict("hello world")                        # False - no hex patterns
```

### 11. Compression Ratio (`COMPRESSION_RATIO`) - NEW v0.4.0

**Implementation Logic**: Uses zlib compression to detect text with unusual entropy patterns. Natural language has patterns and redundancy that compress well, while random text compresses poorly.

**Algorithm**:
1. Compress text using zlib
2. Calculate compression ratio (compressed/original size)
3. Compare against thresholds

**Parameters**:
- `high_ratio_threshold` (float, default: 1.1): Ratio above which text is garbled
- `low_ratio_threshold` (float, default: 0.85): Ratio below which text is valid
- `min_length` (int, default: 100): Minimum text length to analyze

```python
detector = GarbleDetector(Strategy.COMPRESSION_RATIO, threshold=0.5)

# Examples (works best on longer text)
long_random = "xkjhqwerty zxcvbn " * 10
detector.predict(long_random)              # True - random patterns
detector.predict("hello " * 30)            # False - repetitive but valid
```

### 12. Mojibake Detection (`MOJIBAKE`) - NEW v0.4.0 ⭐ 100% Precision

**Implementation Logic**: Detects encoding corruption (mojibake) that occurs when UTF-8 text is incorrectly decoded as Latin-1 or Windows-1252. Identifies patterns like "Ã©" (should be "é") and Unicode replacement characters (�).

**Algorithm**:
1. Search for known mojibake byte patterns
2. Detect Unicode replacement characters (U+FFFD)
3. Check for high density of Latin-1 supplement characters
4. Identify double-encoding signatures

**Parameters**:
- `pattern_threshold` (int, default: 1): Number of patterns to trigger detection
- `ratio_threshold` (float, default: 0.05): High-byte density threshold
- `check_replacement_char` (bool, default: True): Check for � characters

```python
detector = GarbleDetector(Strategy.MOJIBAKE, threshold=0.5)

# Examples
detector.predict("Café au lait")           # False - correct UTF-8
detector.predict("CafÃ© au lait")          # True - mojibake (UTF-8 as Latin-1)
detector.predict("Hello � world")          # True - replacement character
```

### 13. Pronounceability (`PRONOUNCEABILITY`) - NEW v0.4.0

**Implementation Logic**: Analyzes if text follows English phonotactic rules. Detects forbidden consonant clusters, checks vowel distribution, and validates word onset patterns.

**Algorithm**:
1. Extract consonant clusters from words
2. Check against forbidden bigram combinations (e.g., "bk", "zt", "qx")
3. Verify vowel ratio is within pronounceable range
4. Validate word-initial consonant clusters

**Parameters**:
- `forbidden_cluster_threshold` (int, default: 2): Forbidden clusters to trigger
- `min_word_length` (int, default: 3): Minimum word length to analyze
- `vowel_min_ratio` (float, default: 0.1): Minimum vowel ratio

```python
detector = GarbleDetector(Strategy.PRONOUNCEABILITY, threshold=0.5)

# Examples
detector.predict("hello world")            # False - pronounceable
detector.predict("xkcd qwfp zxcv")         # True - unpronounceable clusters
detector.predict("bvnk tspk dkfm")         # True - forbidden consonant pairs
detector.predict("through threshold")       # False - valid English clusters
```

### 14. Unicode Script Mixing (`UNICODE_SCRIPT`) - NEW v0.4.0 ⭐ 100% Precision

**Implementation Logic**: Detects suspicious mixing of Unicode scripts, particularly homoglyph attacks where Cyrillic or Greek characters are disguised as Latin letters. Common in phishing attempts (e.g., "pаypal" with Cyrillic 'а').

**Algorithm**:
1. Check for known homoglyph characters (Cyrillic а, о, е, Greek ο, etc.)
2. Detect words mixing multiple scripts
3. Count total scripts used in text

**Parameters**:
- `homoglyph_threshold` (int, default: 1): Homoglyphs to trigger detection
- `max_scripts` (int, default: 2): Maximum allowed scripts
- `check_homoglyphs` (bool, default: True): Enable homoglyph detection

```python
detector = GarbleDetector(Strategy.UNICODE_SCRIPT, threshold=0.5)

# Examples
detector.predict("paypal")                 # False - all Latin
detector.predict("pаypal")                 # True - Cyrillic 'а' (U+0430)
detector.predict("gооgle")                 # True - Cyrillic 'о' (U+043E)
detector.predict("Hello АБВ World")        # True - mixed Latin/Cyrillic
```

### 15. Bigram Probability (`BIGRAM_PROBABILITY`) - NEW v0.5.0 ⭐ 100% Precision

**Implementation Logic**: Detects impossible letter pair combinations that never occur in English. Uses phonotactic constraints to identify gibberish.

**Algorithm**:
1. Extract all letter bigrams from text
2. Check against set of impossible bigrams (qx, jj, xz, etc.)
3. Calculate ratio of impossible to total bigrams

**Parameters**:
- `impossible_ratio_threshold` (float, default: 0.1): Ratio above which text is garbled

```python
detector = GarbleDetector(Strategy.BIGRAM_PROBABILITY, threshold=0.5)

# Examples
detector.predict("hello world")       # False - valid bigrams
detector.predict("qxjjxz")            # True - impossible bigrams (qx, jj, xz)
detector.predict("bxcxdx")            # True - impossible bigrams
```

### 16. Letter Position (`LETTER_POSITION`) - NEW v0.5.0 ⭐ 99% Precision

**Implementation Logic**: Detects letters appearing in impossible positions within words (start/end constraints).

**Algorithm**:
1. Check for letters that never end words (j, q, v)
2. Check for impossible word-initial letter pairs
3. Calculate violation ratio

```python
detector = GarbleDetector(Strategy.LETTER_POSITION, threshold=0.5)

# Examples
detector.predict("Strange strings")    # False - valid positions
detector.predict("wordj endq")         # True - j and q can't end words
detector.predict("xjword bwtext")      # True - impossible word starts
```

### 17. Consonant Sequence (`CONSONANT_SEQUENCE`) - NEW v0.5.0

**Implementation Logic**: Detects impossibly long consonant sequences. English allows at most 5-6 consecutive consonants (e.g., "strengths").

**Algorithm**:
1. Extract consonant sequences from words
2. Flag sequences of 6+ consonants as impossible
3. Skip all-caps words (likely acronyms)

```python
detector = GarbleDetector(Strategy.CONSONANT_SEQUENCE, threshold=0.5)

# Examples
detector.predict("strengths")         # False - valid (6 consonants max)
detector.predict("bcdfghjklmn")       # True - 11 consonants impossible
detector.predict("HTTP/HTTPS")        # False - acronyms skipped
```

### 18. Vowel Pattern (`VOWEL_PATTERN`) - NEW v0.5.0 ⭐ 100% Precision

**Implementation Logic**: Detects invalid vowel sequences. English has specific vowel patterns; repeated same vowels (5+) are impossible.

**Algorithm**:
1. Extract vowel sequences from text
2. Check for 5+ repeated same vowels
3. Allow valid patterns like "eau", "iou", "ueue"

```python
detector = GarbleDetector(Strategy.VOWEL_PATTERN, threshold=0.5)

# Examples
detector.predict("beautiful")         # False - valid vowel pattern
detector.predict("aaaaaaa")           # True - repeated vowels
detector.predict("queue")             # False - valid "ueue" pattern
```

### 19. Letter Frequency (`LETTER_FREQUENCY`) - NEW v0.5.0 ⭐ 100% Precision

**Implementation Logic**: Detects text dominated by rare letters (j, q, x, z). Uses chi-squared analysis against English letter frequency norms.

**Algorithm**:
1. Calculate letter frequency distribution
2. Compare against expected English frequencies
3. Flag text with excessive rare letters

```python
detector = GarbleDetector(Strategy.LETTER_FREQUENCY, threshold=0.5)

# Examples
detector.predict("The quick brown fox")  # False - normal distribution
detector.predict("jjjj qqqq xxxx zzzz")  # True - dominated by rare letters
detector.predict("xqzjxqzj")             # True - all rare letters
```

### 20. Rare Trigram (`RARE_TRIGRAM`) - NEW v0.5.0 ⭐ 100% Precision

**Implementation Logic**: Detects impossible three-letter combinations that never appear in English.

**Algorithm**:
1. Extract trigrams from text
2. Check against set of impossible trigrams
3. Calculate ratio of impossible trigrams

```python
detector = GarbleDetector(Strategy.RARE_TRIGRAM, threshold=0.5)

# Examples
detector.predict("The quick brown fox")  # False - valid trigrams
detector.predict("jjjqqq")               # True - impossible trigrams
detector.predict("xqzjxq")               # True - impossible trigrams
```

### 21. English Word Validation (`ENGLISH_WORD_VALIDATION`)

**Implementation Logic**: Validates words against an English dictionary using pyspellchecker.

> **Note**: Requires optional dependency. Install with `pip install pygarble[spellchecker]`
> Consider using `WORD_LOOKUP` instead for zero-dependency operation.

```python
detector = GarbleDetector(Strategy.ENGLISH_WORD_VALIDATION, threshold=0.5)

# Examples
detector.predict("hello world")              # False - valid words
detector.predict("asdfghjkl qwertyuiop")    # True - invalid words
```

### 12. Character Frequency (`CHARACTER_FREQUENCY`)

**Implementation Logic**: Analyzes character frequency distribution. Garbled text often has skewed distributions.

```python
detector = GarbleDetector(Strategy.CHARACTER_FREQUENCY, threshold=0.5)

# Examples
detector.predict("aaaaaaa")         # True - high 'a' frequency
detector.predict("normal text")     # False - balanced distribution
```

### 13. Statistical Analysis (`STATISTICAL_ANALYSIS`)

**Implementation Logic**: Analyzes the ratio of alphabetic to non-alphabetic characters.

```python
detector = GarbleDetector(Strategy.STATISTICAL_ANALYSIS, threshold=0.5)

# Examples
detector.predict("123456789")       # True - no alphabetic chars
detector.predict("normal text")     # False - mostly alphabetic
```

### 14. Word Length (`WORD_LENGTH`)

**Implementation Logic**: Checks average word length against normal English patterns.

```python
detector = GarbleDetector(Strategy.WORD_LENGTH, threshold=0.5)

# Examples
detector.predict("supercalifragilistic")  # True - very long word
detector.predict("short words here")       # False - normal lengths
```

## Ensemble Detector

Combine multiple strategies for better accuracy using voting:

```python
from pygarble import EnsembleDetector, Strategy

# Default ensemble (uses best-performing strategies)
ensemble = EnsembleDetector()
print(ensemble.predict("asdfghjkl"))  # True

# Custom strategies
ensemble = EnsembleDetector(
    strategies=[
        Strategy.KEYBOARD_PATTERN,
        Strategy.VOWEL_RATIO,
        Strategy.ENTROPY_BASED,
    ],
    voting="majority"  # or "average" or "weighted"
)

# Weighted voting
ensemble = EnsembleDetector(
    strategies=[Strategy.KEYBOARD_PATTERN, Strategy.VOWEL_RATIO],
    voting="weighted",
    weights=[0.7, 0.3]
)

# Batch processing
texts = ["Hello world", "asdfghjkl", "qwertyuiop"]
results = ensemble.predict(texts)
probas = ensemble.predict_proba(texts)
```

**Voting Modes**:
- `majority`: Text is garbled if >50% of strategies agree
- `average`: Average probability across all strategies
- `weighted`: Weighted average using custom weights
- `any`: High recall - text is garbled if ANY strategy flags it (best F1 score)
- `all`: High precision - text is garbled only if ALL strategies agree

## Advanced Usage

### Input Validation

The library validates inputs automatically:

```python
# Threshold must be between 0 and 1
detector = GarbleDetector(Strategy.KEYBOARD_PATTERN, threshold=1.5)
# Raises: ValueError: threshold must be between 0.0 and 1.0

# Threads must be positive
detector = GarbleDetector(Strategy.KEYBOARD_PATTERN, threads=0)
# Raises: ValueError: threads must be a positive integer
```

### Batch Processing with Threading

```python
detector = GarbleDetector(Strategy.KEYBOARD_PATTERN, threads=4)

# Process 1000 texts in parallel
texts = ["text"] * 1000
results = detector.predict(texts)
```

### Custom Pattern Matching

```python
custom_patterns = {
    'email': r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
    'phone': r'\d{3}-\d{3}-\d{4}',
}

detector = GarbleDetector(
    Strategy.PATTERN_MATCHING,
    patterns=custom_patterns,
    override_defaults=True  # Use only custom patterns
)
```

## API Reference

### GarbleDetector

```python
GarbleDetector(
    strategy: Strategy,
    threshold: float = 0.5,
    threads: Optional[int] = None,
    **kwargs
)
```

**Parameters:**
- `strategy`: Detection strategy to use
- `threshold`: Probability threshold (0.0-1.0) for binary predictions
- `threads`: Number of threads for batch processing
- `**kwargs`: Strategy-specific parameters

**Methods:**
- `predict(X)`: Returns `bool` or `List[bool]`
- `predict_proba(X)`: Returns `float` or `List[float]`

### EnsembleDetector

```python
EnsembleDetector(
    strategies: Optional[List[Strategy]] = None,
    threshold: float = 0.5,
    voting: str = "majority",  # "majority", "average", "weighted", "any", "all"
    weights: Optional[List[float]] = None,
    threads: Optional[int] = None,
    **kwargs
)
```

### Strategy Enum

```python
class Strategy(Enum):
    # Core strategies (zero dependencies)
    MARKOV_CHAIN = "markov_chain"              # Recommended - Best F1 score
    NGRAM_FREQUENCY = "ngram_frequency"        # Trigram analysis
    WORD_LOOKUP = "word_lookup"                # Zero dependencies dictionary
    SYMBOL_RATIO = "symbol_ratio"              # Symbol/number detection
    REPETITION = "repetition"                  # Pattern repetition
    HEX_STRING = "hex_string"                  # Hash/UUID detection
    COMPRESSION_RATIO = "compression_ratio"    # v0.4.0 - Compression-based
    MOJIBAKE = "mojibake"                      # v0.4.0 - Encoding corruption
    PRONOUNCEABILITY = "pronounceability"      # v0.4.0 - Phonotactic rules
    UNICODE_SCRIPT = "unicode_script"          # v0.4.0 - Homoglyph detection

    # High-precision strategies (v0.5.0)
    BIGRAM_PROBABILITY = "bigram_probability"  # NEW v0.5.0 - 100% precision
    LETTER_POSITION = "letter_position"        # NEW v0.5.0 - 99% precision
    CONSONANT_SEQUENCE = "consonant_sequence"  # NEW v0.5.0 - Consonant runs
    VOWEL_PATTERN = "vowel_pattern"            # NEW v0.5.0 - 100% precision
    LETTER_FREQUENCY = "letter_frequency"      # NEW v0.5.0 - 100% precision
    RARE_TRIGRAM = "rare_trigram"              # NEW v0.5.0 - 100% precision

    # Legacy strategies
    CHARACTER_FREQUENCY = "character_frequency"
    WORD_LENGTH = "word_length"
    PATTERN_MATCHING = "pattern_matching"
    STATISTICAL_ANALYSIS = "statistical_analysis"
    ENTROPY_BASED = "entropy_based"
    VOWEL_RATIO = "vowel_ratio"
    KEYBOARD_PATTERN = "keyboard_pattern"

    # Strategy with optional dependency
    ENGLISH_WORD_VALIDATION = "english_word_validation"  # Requires: pygarble[spellchecker]
```

## Architecture

```
pygarble/
├── __init__.py
├── core.py                      # GarbleDetector & EnsembleDetector
├── data/                        # Embedded training data
│   ├── words.py                 # 50K English words
│   ├── bigrams.py               # Character transition probabilities
│   └── trigrams.py              # Common English trigrams
└── strategies/
    ├── base.py                  # BaseStrategy with shared utilities
    ├── markov_chain.py          # Markov chain detection
    ├── ngram_frequency.py       # Trigram frequency analysis
    ├── word_lookup.py           # Dictionary lookup (zero deps)
    ├── symbol_ratio.py          # Symbol/number detection
    ├── repetition.py            # Pattern repetition
    ├── hex_string.py            # Hash/UUID detection
    ├── compression_ratio.py     # Compression-based detection
    ├── mojibake.py              # Encoding corruption detection
    ├── pronounceability.py      # Phonotactic rules
    ├── unicode_script.py        # Homoglyph/script detection
    ├── bigram_probability.py    # NEW v0.5.0: Impossible bigrams
    ├── letter_position.py       # NEW v0.5.0: Position constraints
    ├── consonant_sequence.py    # NEW v0.5.0: Consonant runs
    ├── vowel_pattern.py         # NEW v0.5.0: Vowel sequences
    ├── letter_frequency.py      # NEW v0.5.0: Letter distribution
    ├── rare_trigram.py          # NEW v0.5.0: Impossible trigrams
    ├── character_frequency.py
    ├── word_length.py
    ├── pattern_matching.py      # Regex patterns + keyboard detection
    ├── statistical_analysis.py
    ├── entropy_based.py         # Shannon entropy + bigram analysis
    ├── english_word_validation.py  # pyspellchecker (optional)
    ├── vowel_ratio.py           # Vowel analysis + consonant clusters
    └── keyboard_pattern.py      # N-gram + keyboard row detection
```

## Dependencies

**Core library**: Zero dependencies - works with Python 3.8+ only

**Optional dependency**:
- `pygarble[spellchecker]`: pyspellchecker for English word validation
  - pyspellchecker>=0.7.0

## Development

```bash
# Clone and setup
git clone https://github.com/brightertiger/pygarble.git
cd pygarble
pip install -r requirements.txt -r requirements-dev.txt

# Run tests
pytest tests/ -v

# Run benchmark
python regression/benchmark.py

# Linting
flake8 pygarble/
black pygarble/
mypy pygarble/
```

## Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### Adding New Strategies

1. Create a new file in `pygarble/strategies/`
2. Inherit from `BaseStrategy`
3. Implement `_predict_impl()` and `_predict_proba_impl()`
4. Add to `strategies/__init__.py` and `core.py`
5. Add tests in `tests/`

## License

MIT License - see [LICENSE](LICENSE) for details.

## Changelog

### 0.5.0 (Current)
- **6 New High-Precision Strategies** designed to minimize false positives:
  - `BIGRAM_PROBABILITY`: Detects impossible letter pairs (qx, jj, xz) - **100% precision**
  - `LETTER_POSITION`: Detects letters in impossible positions - **99% precision**
  - `CONSONANT_SEQUENCE`: Detects impossibly long consonant runs (6+)
  - `VOWEL_PATTERN`: Detects invalid vowel sequences - **100% precision**
  - `LETTER_FREQUENCY`: Detects text dominated by rare letters - **100% precision**
  - `RARE_TRIGRAM`: Detects impossible trigrams - **100% precision**
- **Redesigned Default Ensemble**: Uses majority voting with high-precision strategies for **99.5% precision**
- **Expanded Benchmark**: 1,644 test cases (420 internal + 1,224 external validation)
- **External Validation**: Added dictionary words and generated gibberish to detect overfitting
- **Source Attribution**: Benchmark now tracks internal vs external data sources
- **Overfitting Analysis**: Benchmark compares internal vs external performance

### 0.4.0
- **4 New Specialized Strategies**:
  - `COMPRESSION_RATIO`: Detects garbled text using zlib compression analysis (best for long text)
  - `MOJIBAKE`: Detects encoding corruption with **100% precision** (UTF-8 decoded as Latin-1, replacement characters)
  - `PRONOUNCEABILITY`: Detects unpronounceable text using English phonotactic rules (forbidden consonant clusters)
  - `UNICODE_SCRIPT`: Detects homoglyph attacks with **100% precision** (Cyrillic/Greek chars disguised as Latin)
- **Expanded Benchmark**: 263 test cases across 44 categories (up from 200/34)
- **New Test Categories**: mojibake_encoding, replacement_chars, homoglyph_attacks, mixed_scripts, unpronounceable, long_random_text, legitimate_unicode, spam_patterns
- **Zero Dependencies**: All new strategies use only Python stdlib (zlib, unicodedata, re)

### 0.3.1
- **New Strategies**: Added `SYMBOL_RATIO`, `REPETITION`, and `HEX_STRING` strategies for specialized detection
- **New Voting Modes**: Added `any` (high recall) and `all` (high precision) voting modes for EnsembleDetector
- **Removed**: `LANGUAGE_DETECTION` strategy (FastText dependency had NumPy 2.0 compatibility issues)
- **Production-Grade Robustness**:
  - Thread safety with timeout and exception handling in batch processing
  - Division by zero protection across all strategy calculations
  - Parameter validation for all strategy-specific parameters
  - Pre-compiled regex patterns for improved performance
- **Comprehensive Edge Case Tests**: 77 new tests covering parameter validation, type errors, Unicode handling, and boundary conditions

### 0.3.0
- **Zero Dependencies**: Core library now works without any external dependencies
- **New Markov Chain Strategy**: Character-level Markov chain trained on 300K+ English words
- **New N-gram Frequency Strategy**: Trigram analysis using 2000 most common English trigrams
- **New Word Lookup Strategy**: 50K embedded English word dictionary (replaces pyspellchecker dependency)
- **Embedded Training Data**: Pre-computed bigrams, trigrams, and word sets included in package
- **Optional Dependencies**: FastText and pyspellchecker moved to optional extras
- **Lightweight Package**: ~190KB wheel size (well under 5MB limit)
- **Data Source**: Training data from Peter Norvig's word frequency list (MIT licensed)

### 0.2.0
- **New Keyboard Pattern Strategy**: Best-performing strategy with 69.9% F1 score
- **New Vowel Ratio Strategy**: Highest precision (95.45%) with consonant cluster detection
- **EnsembleDetector**: Built-in ensemble with majority/average/weighted voting
- **Enhanced Entropy Strategy**: Added bigram frequency analysis using common English bigrams
- **Enhanced Pattern Matching**: Added keyboard row patterns, consonant clusters, alternating patterns
- **Input Validation**: Validates threshold (0-1) and threads parameters
- **Type Hints**: Full type annotation throughout the codebase
- **Regression Tests**: 117 test cases across 20 categories with benchmarking
- **Performance**: Regex patterns compiled once at initialization

### 0.1.0
- Initial release with 7 detection strategies
- Scikit-learn-like interface
- Probability scoring
- Modular architecture
