# Reference SQL Storage Module

This module handles the storage, processing, and analysis of SQL query history files. It provides functionality to extract SQL statements from files, analyze them with LLM, and store them in a searchable knowledge base.


## Data Flow

```
SQL Files → File Processor → LLM Analysis → Storage
     ↓              ↓             ↓           ↓
Comment-SQL     Validation    Metadata    Vector DB
  Pairs         + Cleaning   Extraction   + Search
```

### Processing Pipeline

1. **File Processing**: Extract comment-SQL pairs from `.sql` files
2. **SQL Type Filtering**: Only process SELECT queries (skip INSERT/UPDATE/DELETE/etc.)
3. **Validation**: Validate SQL syntax using multiple dialects (MySQL, Hive, Spark)
4. **LLM Analysis**: Extract business metadata using SqlSummaryAgenticNode
5. **Storage**: Store enriched data in LanceDB for vector search
6. **Indexing**: Create search indices for efficient retrieval

## Configuration

### Build Modes

- **`overwrite`**: Replace all existing data
- **`incremental`**: Only process new items (based on SQL+comment hash)
- **`validate-only`**: Only validate SQL files without LLM analysis or storage

### Performance Tuning

- **`pool_size`**: Number of concurrent async tasks for LLM analysis (default: 1)
- **Parallel processing**: Items are processed asynchronously with semaphore-controlled concurrency

## Data Schema

### Input Format (from SQL files)
```python
{
    "sql": "SELECT * FROM users WHERE active = 1",  # Cleaned and validated SQL
    "comment": "Query active users for analysis",  # Extracted from -- comments
    "filepath": "/path/to/source.sql"               # Source file path
}
```

**Notes:**
- Only SELECT queries are processed (other SQL types are skipped)
- SQL is validated against MySQL, Hive, and Spark dialects
- Comments can be empty (will be processed with empty comment field)

### Output Format (after LLM analysis)
```python
{
    "id": "md5_hash_of_sql_and_comment",
    "name": "active_users",                        # Generated by LLM
    "sql": "SELECT * FROM users WHERE active = 1",
    "comment": "Query active users for analysis",
    "filepath": "/path/to/source.sql",
    "summary": "Retrieve all active user records for business analysis",  # Generated by LLM
    "search_text": "active_users: Retrieve all active user records...",   # Used for vector embedding
    "subject_path": ["user_analysis", "user_behavior", "activity_tracking"],  # Generated by LLM
    "tags": "users,active,analysis",               # Generated by LLM
    "created_at": "2025-01-15T10:30:00",           # Auto-generated timestamp
    "vector": [0.1, 0.2, ...]                      # Embedding vector from search_text
}
```

**Required fields for storage:**
- `name`, `sql`, `summary`, `search_text`, `subject_path` (all generated by SqlSummaryAgenticNode)
- Items missing any required fields will be skipped with a warning

## Usage

```bash
# Basic usage - initialize reference SQL with overwrite mode
python -m datus.main bootstrap-kb \
  --namespace your_namespace \
  --components reference_sql \
  --sql_dir /path/to/sql/directory \
  --kb_update_strategy overwrite

# Incremental update - only process new SQL files
python -m datus.main bootstrap-kb \
  --namespace your_namespace \
  --components reference_sql \
  --sql_dir /path/to/sql/directory \
  --kb_update_strategy incremental

# High performance - use 8 parallel threads for LLM analysis
python -m datus.main bootstrap-kb \
  --namespace your_namespace \
  --components reference_sql \
  --sql_dir /path/to/sql/directory \
  --kb_update_strategy overwrite \
  --pool_size 8

# Debug mode with detailed logging
python -m datus.main bootstrap-kb \
  --namespace your_namespace \
  --components reference_sql \
  --sql_dir /path/to/sql/directory \
  --kb_update_strategy overwrite \
  --debug

# Validate-only mode - process and validate SQL files without LLM analysis or storage
python -m datus.main bootstrap-kb \
  --namespace your_namespace \
  --components reference_sql \
  --sql_dir /path/to/sql/directory \
  --validate-only
```
