# dotfilter

> Boolean algebra on document collections - lifting single-document logic to sets

## Overview

`dotfilter` represents the Collections dimension's core capability: lifting Truth pillar operations to work on collections. It filters documents based on callable predicates while providing lazy evaluation through QuerySet operations.

This tool bridges single-document validation (Truth pillar) with collection-level operations, enabling set operations like union, intersection, and difference on filtered results.

## Installation

```bash
# As part of dotsuite
pip install dotsuite

# Or from source
cd src/collections/dotfilter
pip install -e .
```

## Core Functions

### `filter_docs(docs, predicate)`

Filter a collection of documents using a callable predicate. The predicate is a function that takes a document and returns `True` to keep it.

```python
from collections.dotfilter.core import filter_docs

docs = [
    {"user": "alice", "role": "admin", "active": True},
    {"user": "bob", "role": "user", "active": True},
    {"user": "charlie", "role": "admin", "active": False}
]

# Filter using a lambda predicate
admins = filter_docs(docs, lambda d: d.get("role") == "admin")
# Returns: [{"user": "alice", ...}, {"user": "charlie", ...}]

active_admins = filter_docs(docs, lambda d: d.get("role") == "admin" and d.get("active"))
# Returns: [{"user": "alice", ...}]
```

### `filter_by_path(docs, path, value)`

Simple filtering by path equality. Uses `dotequals` internally.

```python
from collections.dotfilter.core import filter_by_path

# Direct path filtering
active_users = filter_by_path(docs, "active", True)
# Returns: [{"user": "alice", ...}, {"user": "bob", ...}]

admin_users = filter_by_path(docs, "role", "admin")
# Returns: [{"user": "alice", ...}, {"user": "charlie", ...}]
```

### `filter_by_existence(docs, path)`

Filter documents where a path exists. Uses `dotexists` internally.

```python
from collections.dotfilter.core import filter_by_existence

docs = [
    {"user": "alice", "email": "alice@example.com"},
    {"user": "bob"},
    {"user": "charlie", "email": "charlie@example.com"}
]

with_email = filter_by_existence(docs, "email")
# Returns: [{"user": "alice", ...}, {"user": "charlie", ...}]
```

### `combine_filters(docs, *predicates, mode="and")`

Combine multiple filter predicates with AND or OR logic.

```python
from collections.dotfilter.core import combine_filters

users = [
    {"name": "Alice", "role": "admin", "active": True},
    {"name": "Bob", "role": "user", "active": True},
    {"name": "Charlie", "role": "admin", "active": False}
]

# AND mode: all predicates must match
active_admins = combine_filters(
    users,
    lambda d: d.get("role") == "admin",
    lambda d: d.get("active") is True,
    mode="and"
)
# Returns: [{"name": "Alice", ...}]

# OR mode: any predicate can match
admin_or_active = combine_filters(
    users,
    lambda d: d.get("role") == "admin",
    lambda d: d.get("active") is True,
    mode="or"
)
# Returns: all three users
```

## QuerySet Class

`QuerySet` provides lazy evaluation with set operations. Filtering only happens when results are materialized (iterated, counted, etc.).

### Creating and Filtering

```python
from collections.dotfilter.core import QuerySet

docs = [
    {"name": "Alice", "role": "admin", "active": True},
    {"name": "Bob", "role": "user", "active": True},
    {"name": "Charlie", "role": "admin", "active": False}
]

# Create QuerySets (lazy - no evaluation yet)
all_docs = QuerySet(docs)
admins = all_docs.filter(lambda d: d.get("role") == "admin")
active = all_docs.filter(lambda d: d.get("active") is True)

# Exclude (lazy)
non_admins = all_docs.exclude(lambda d: d.get("role") == "admin")
```

### Set Operations

```python
# Set operations (still lazy)
active_admins = admins.intersect(active)         # admins AND active
admin_or_active = admins.union(active)            # admins OR active
inactive_admins = admins.difference(active)       # admins NOT active

# Operator shortcuts
active_admins = admins & active     # intersect
admin_or_active = admins | active   # union
inactive_admins = admins - active   # difference
```

### Materializing Results

```python
# Evaluate when needed
result = list(active_admins)         # Iterate to list
result = active_admins.list()        # Same thing via method

count = active_admins.count()        # Count matching documents
has_any = active_admins.exists()     # True if any match
first = active_admins.first()        # First matching document or None
```

Results are cached after first evaluation -- subsequent iterations reuse the cache.

## Command Line

```bash
# Filter JSONL by path value
dotfilter users.jsonl --path role --value admin

# Filter by existence of a path (repeatable)
dotfilter data.jsonl --exists email
dotfilter data.jsonl --exists email --exists phone

# Python expression filter (document available as 'd')
dotfilter data.jsonl --expr "d['age'] > 18 and d['status'] == 'active'"

# Combine path and existence filters
dotfilter data.jsonl --path role --value admin --exists email

# Count results instead of outputting documents
dotfilter data.jsonl --path active --value true --count

# Invert the filter (exclude matches)
dotfilter data.jsonl --path role --value guest --invert

# Combine filters with OR logic (default is AND)
dotfilter data.jsonl --path role --value admin --exists premium --mode or

# Read from stdin
cat users.jsonl | dotfilter - --path active --value true

# Write to output file
dotfilter users.jsonl --path role --value admin --output admins.jsonl
```

### Options

| Flag | Description |
|------|-------------|
| `input` | JSONL file path or `-` for stdin (positional) |
| `-p`, `--path` | Filter by path value (use with `--value`) |
| `-v`, `--value` | Expected value for `--path` filter (parsed as JSON) |
| `-e`, `--exists` | Filter where path exists (repeatable) |
| `-x`, `--expr` | Python expression filter (document is `d`) |
| `-m`, `--mode` | How to combine multiple filters: `and` (default) or `or` |
| `-c`, `--count` | Output count instead of documents |
| `-i`, `--invert` | Invert the filter (exclude matching documents) |
| `-o`, `--output` | Output file (default: stdout) |

Input and output use JSONL format (one JSON object per line).

## Common Use Cases

### 1. User Management

```python
from collections.dotfilter.core import filter_by_path, filter_docs

users = [
    {"id": 1, "name": "Alice", "role": "admin", "active": True, "login_count": 42},
    {"id": 2, "name": "Bob", "role": "user", "active": True, "login_count": 5},
    {"id": 3, "name": "Charlie", "role": "admin", "active": False, "login_count": 0},
    {"id": 4, "name": "Diana", "role": "moderator", "active": True, "login_count": 15}
]

# Find active admins
active_admins = filter_docs(users, lambda d: d["role"] == "admin" and d["active"])

# Find users who need activation
inactive = filter_by_path(users, "active", False)

# Find power users
power_users = filter_docs(users, lambda d: d.get("login_count", 0) > 10)
```

### 2. Lazy Set Operations

Combine filters efficiently:

```python
from collections.dotfilter.core import QuerySet

products = QuerySet([
    {"name": "Laptop", "price": 999, "category": "electronics", "in_stock": True},
    {"name": "Mouse", "price": 25, "category": "electronics", "in_stock": True},
    {"name": "Desk", "price": 299, "category": "furniture", "in_stock": False},
    {"name": "Chair", "price": 199, "category": "furniture", "in_stock": True}
])

# Define filters (lazy)
electronics = products.filter(lambda d: d["category"] == "electronics")
in_stock = products.filter(lambda d: d["in_stock"])
affordable = products.filter(lambda d: d["price"] < 200)

# Combine filters (still lazy)
available_electronics = electronics & in_stock
budget_items = affordable & in_stock

# Only evaluates when needed
for item in available_electronics:
    print(f"{item['name']}: ${item['price']}")
```

## Philosophy

`dotfilter` embodies the Collections dimension philosophy:
1. **Homomorphic Lifting**: Truth operations on documents lift to set operations on collections
2. **Lazy Evaluation**: QuerySets defer computation until needed
3. **Composability**: Set operations combine naturally
4. **Simplicity**: Complex filtering through simple boolean algebra

## Performance Notes

- **Lazy Evaluation**: QuerySets don't evaluate until iteration
- **Caching**: Results are cached after first evaluation
- **Set Operations**: Optimized using object identity for deduplication

## See Also

- [`dotquery`](../truth/dotquery.md) - Single-document boolean queries
- [`dotany`](../truth/dotany.md) - Check if any document matches
- [`dotall`](../truth/dotall.md) - Check if all documents match
- [`dotrelate`](../collections/dotrelate.md) - Relational operations on collections
