# Data Directory

**⚠️ PRIVACY NOTICE:** This directory contains processed personal conversation data. **NEVER commit these files to git!**

---

## Directory Structure

```
data/
├── raw/                    # Original chat exports
│   └── conversations.json  # ChatGPT export (JSON)
│
├── preprocessed/          # Cleaned & truncated conversations
│   └── conversations_clean.json
│
├── observations/          # Extracted observations
│   ├── observations_raw.json        # All observations (unfiltered)
│   ├── observations.json            # Filtered & clustered
│   └── embeddings.json              # Observation embeddings
│
├── learnings/             # Synthesized learnings
│   └── learnings.json
│
├── cognitions/            # High-level cognitions
│   └── cognitions.json
│
└── persona/               # Final persona output
    ├── persona.json       # Structured (with metadata)
    └── persona.md         # Human-readable
```

---

## File Descriptions

### `raw/conversations.json`

Raw ChatGPT export in "mapping" format.

**What to put here:**
1. Go to ChatGPT → Settings → Data controls → Export data
2. Download the ZIP file
3. Extract `conversations.json`
4. Place it in `data/raw/`

**Format:** ChatGPT's native export format (nested mapping objects)

---

### `preprocessed/conversations_clean.json`

Clean, truncated, AI-ready conversations.

**Generated by:** Step 1 (`01_transform_filter.py`)

**Format:**
```json
{
  "export_date": "2025-10-16T...",
  "filter_cutoff_days": 180,
  "remove_code_blocks": true,
  "truncation_applied": true,
  "truncation_strategy": "head_2000_tail_3000",
  "truncated_count": 257,
  "total_conversations": 1000,
  "total_messages": 8543,
  "total_tokens": 3245678,
  "conversations": [
    {
      "id": "...",
      "title": "...",
      "created_at": 1683420143.0,
      "updated_at": 1683420200.0,
      "message_count": 15,
      "truncated": true,
      "original_message_count": 428,
      "truncated_message_count": 413,
      "metrics": {
        "total_tokens": 5010,
        "original_total_tokens": 79529,
        "user_tokens": 1200,
        "assistant_tokens": 3810,
        ...
      },
      "messages": [
        {
          "id": "msg1",
          "timestamp": 1683420143.0,
          "role": "user",
          "text": "...",
          "token_count": 42
        },
        ...
        {
          "role": "system",
          "text": "[385 MESSAGES TRUNCATED FOR BREVITY]",
          "token_count": 10
        },
        ...
      ]
    }
  ]
}
```

**Changes from raw:**
- Converted from mapping → messages array
- Filtered by date (last N days)
- Code blocks removed (if enabled)
- **Smart truncation:** Long conversations truncated to 5K tokens (2K start + 3K end)
- Clear truncation markers inserted
- Token counts and metrics calculated

---

### `observations/observations.json`

Extracted observations with privacy filtering and clustering.

**Generated by:** Step 2 (`02_extract_cluster_observations.py`)

**Format:**
```json
{
  "export_date": "2025-10-16T...",
  "mode": "full",
  "observations": [
    {
      "id": "observation_00001",
      "title": "...",
      "content": "...",
      "keywords": ["machine-learning", "data-processing"],
      "entities": ["Python", "TensorFlow", "AWS"],
      "category": "technical",
      "privacy": "low",
      "conversation_id": "...",
      "conversation_title": "...",
      "timestamp": 1683420143.0,
      "main_language": "en",
      "cluster_id": "Category.TECHNICAL_5",
      "cluster_number": 5
    }
  ],
  "extraction": {
    "prompt_template": "observation_extraction",
    "model": "gemini-2.5-flash-lite",
    "total_extracted": 845,
    "success_rate": 0.845
  },
  "clustering": {
    "algorithm": "kmeans",
    "by_category": true,
    "total_clusters": 40,
    "categories": {...}
  }
}
```

**Key Fields:**
- `privacy`: `low` | `medium` | `high` (filtered by category-specific rules)
- `category`: `technical` | `professional` | `personal`
- `cluster_id`: String identifier (e.g., "Category.TECHNICAL_5")
- `cluster_number`: Numeric cluster index within category
- `timestamp`: Unix timestamp (float) of conversation
- `keywords`: Abstract concepts (lowercase-kebab-case)
- `entities`: Concrete names (Title Case)

---

### `learnings/learnings.json`

Synthesized learning patterns from observation clusters.

**Generated by:** Step 3 (`03_synthesize_cluster_learnings.py`)

**Format:**
```json
{
  "learnings": [
    {
      "id": "learning_00001",
      "title": "...",
      "content": "...",
      "keywords": ["system-design", "scalability"],
      "entities": ["Docker", "Kubernetes"],
      "category": "technical",
      "confidence": "high",
      "source_observation_ids": ["observation_00001", "observation_00002"],
      "cluster_id": "Category.TECHNICAL_2",
      "first_observed": 1737849600.0,
      "last_observed": 1760204800.0,
      "source_observation_count": 12,
      "cognition_cluster_id": "cognition_0"
    }
  ],
  "synthesis": {
    "prompt_template": "learning_synthesis",
    "model": "gemini-2.5-flash",
    "total_learnings": 67
  },
  "clustering": {
    "total_clusters": 10,
    "target_clusters": 10
  }
}
```

---

### `cognitions/cognitions.json`

High-level cognition patterns from learning clusters.

**Generated by:** Step 4 (`04_synthesize_cognitions.py`)

**Format:**
```json
{
  "cognitions": [
    {
      "title": "Pragmatic approach to system design and scalability",
      "content": "...",
      "domains": ["technical", "professional"],
      "stability": "stable",
      "priority": "core",
      "keywords": ["architecture", "performance", "scalability"],
      "entities": ["Docker", "Kubernetes", "AWS"],
      "source_learning_ids": ["learning_00001", "learning_00002"],
      "cluster_id": "cognition_3",
      "first_observed": 1737590400.0,
      "last_observed": 1760131200.0,
      "source_learning_count": 8
    }
  ],
  "synthesis": {
    "prompt_template": "cognition_synthesis",
    "model": "gemini-2.5-flash",
    "total_cognitions": 18
  }
}
```

**Key Fields:**
- `title`: Short descriptive title (5-10 words)
- `content`: Core principle/cognition (2-3 paragraphs)
- `domains`: Applicable domains (technical, professional, personal)
- `stability`: stable | evolving | emerging
- `priority`: core | important | peripheral
- `keywords`: Abstract concepts (lowercase-kebab-case)
- `entities`: Shared entities from source learnings (frequency ≥2, top 10, sorted by occurrence)
- `cluster_id`: String identifier (e.g., "cognition_3")
- Timestamps: Unix timestamps (float) derived from source learnings

---

### `persona/persona.json` & `persona.md`

Final user persona for AI system prompts.

**Generated by:** Step 5 (`05_build_persona.py`)

**JSON Format:**
```json
{
  "generated_at": "2025-10-12T...",
  "source_cognitions": 22,
  "model": "gemini-2.5-flash",
  "persona": {
    "personal_profile": "...",
    "communication_preferences": "...",
    "professional_profile": "..."
  }
}
```

**Markdown Format:**
```markdown
# User Persona Information for AI

> **Generated:** 2025-10-12
> **Source:** 22 cognitions
> **Model:** gemini-2.5-flash

## Personal Profile
...

## Communication Preferences
...

## Professional Profile
...
```

**Usage:** Copy `persona.md` content into AI system prompts!

---

## Privacy & Security

### What's Protected

All files in this directory are **automatically ignored by git** via `.gitignore`:

```gitignore
data/raw/
data/preprocessed/
data/observations/
data/learnings/
data/cognitions/
data/persona/
data/memory_db/
```

### What to Keep Private

- ✅ Raw conversations (personal chat history)
- ✅ All processed outputs (observations, learnings, cognitions)
- ✅ Persona files (contain personal profile)
- ✅ Memory database (memg-core storage)
- ✅ Any CSV/analysis files you generate

### What's Safe to Share

- ✅ Empty directory structure
- ✅ This README
- ✅ `.gitkeep` files (placeholder for empty dirs)

---

## Regenerating Data

To regenerate from scratch:

```bash
# Clean everything except raw data
cd rebrain
rm -rf data/preprocessed/ data/observations/ data/learnings/ data/cognitions/ data/persona/ data/memory_db/

# Re-run pipeline
scripts/pipeline/cli.sh all
```

To regenerate from a specific step:

```bash
# Re-cluster observations without re-extracting
scripts/pipeline/cli.sh step2 --cluster-only

# Re-synthesize learnings with new config
rm -rf data/learnings/
scripts/pipeline/cli.sh step3
```

---

## File Sizes (Typical)

For 1000 conversations (last year):

- `raw/conversations.json`: ~80-150 MB (original export)
- `preprocessed/conversations_clean.json`: ~30-50 MB (truncated, code removed)
- `observations/observations_raw.json`: ~3-5 MB (all observations)
- `observations/observations.json`: ~2-4 MB (filtered & clustered)
- `observations/embeddings.json`: ~15-25 MB (embedding vectors)
- `learnings/learnings.json`: ~500 KB-1 MB
- `cognitions/cognitions.json`: ~30-50 KB
- `persona/persona.json`: ~2-3 KB
- `persona/persona.md`: ~2-3 KB
- `memory_db/`: ~30-40 MB (Qdrant + Kuzu databases)

Total: ~150-250 MB for full pipeline output

---

## Troubleshooting

**"conversations.json not found"**
- Place your ChatGPT export in `data/raw/conversations.json`
- Or specify custom path: `scripts/pipeline/cli.sh step1 -i path/to/export.json`

**"Permission denied"**
- Ensure you have write permissions in `data/` directory
- Check disk space (need ~200 MB free)

**"Out of memory"**
- Reduce `date_cutoff_days` in `config/pipeline.yaml`
- Process fewer conversations at once

---

**See Also:**
- `scripts/pipeline/README.md` - Pipeline documentation
- `config/pipeline.yaml` - Configuration settings
- `.gitignore` - What's protected from git

