---
type: skill
domain: data
links: [Corpus_Callosum/CONSTITUTION.md, PULSE_OS.md]
---

# DATA_DYNAMO.md
> **ROLE:** DATA ENGINEER
> **TRIGGER:** "BIG DATA"

## Protocol

1. **Map the data flow.** Document every ETL step: source, transform, destination. Identify where data enters PULSE (session logs, agent output, task board) and where it lands (brain files, evidence, state).
2. **Validate at ingestion.** Every incoming record must pass: schema check (required fields present), completeness (no null required fields), type check (strings are strings, ints are ints). Reject and log invalid records.
3. **Check data quality metrics.** After each pipeline run, compute: completeness (% non-null required fields), consistency (% records matching expected schema), freshness (time since last update). Alert on threshold violations.
4. **Optimize throughput.** Batch writes where possible (flush every 50 records or 30 seconds). Use append-only writes to avoid file lock contention. Profile pipeline bottlenecks with timing logs.
5. **Archive and tier.** Hot data (<7 days) stays in working files. Warm data (7-30 days) moves to archive directory. Cold data (>30 days) compresses and moves to long-term storage.

## Metrics & Thresholds
- **Completeness:** >98% of required fields non-null per batch
- **Consistency:** >99% of records match expected schema
- **Freshness:** Pipeline output <5 min old during active sessions
- **Throughput:** Process >100 records/sec for batch, <500ms latency for streaming
- **Error rate:** <1% of ingested records rejected per batch
- **Batch flush:** Every 50 records or 30 seconds, whichever comes first

## Good Example
```python
def validate_record(record: dict) -> bool:
    """Validate an incoming data record at ingestion."""
    required = ["timestamp", "agent_id", "content", "type"]
    # Completeness: all required fields present and non-null
    if not all(record.get(f) for f in required):
        return False
    # Consistency: type must be a known value
    if record["type"] not in ("LEARNING", "FAILURE", "PATTERN", "DECISION"):
        return False
    # Freshness: reject records older than 24h
    age = time.time() - record["timestamp"]
    if age > 86400:
        return False
    return True
```

## Anti-Patterns
- **No validation at ingestion:** Trusting upstream data blindly. Garbage propagates downstream and corrupts the brain.
- **Unbounded file growth:** Appending forever without rotation or archival. Files >10MB slow down every agent read.
- **Silent data loss:** Dropping invalid records without logging them. Always log rejections with reason for debugging.

## Tools
- `python pulse/brain/brain_query.py --query "data pipeline <component>"`
- `python pulse/brain/pulse_save.py --agent <you> --learnings "pipeline processed N records, Q% quality"`
- `python pulse/tasks/pulse_relay.py --tasks "Data quality audit:data:fast"`

## Verification Checklist
- [ ] All data sources have schema validation at ingestion point
- [ ] Completeness >98% and consistency >99% measured per batch
- [ ] Pipeline freshness <5 min verified during active sessions
- [ ] Rejected records logged with reason (not silently dropped)
- [ ] Data tiering active: hot (<7d), warm (7-30d), cold (>30d archived)
