# Market Data Ingestion System

A high-performance market data ingestion pipeline for processing real-time stock prices, trades, and market events with ultra-low latency requirements.

## Overview

This sample demonstrates:
- **Ultra-high frequency processing**: 100,000+ messages per second
- **Market data validation**: Price reasonability, market hours, circuit breakers
- **Technical indicators**: Real-time calculation of SMA, EMA, RSI, MACD
- **Time-series optimization**: Optimized for TimescaleDB and ClickHouse
- **Regulatory compliance**: MiFID II, Reg NMS compliance

## Architecture

```
Market Data        AtomIngest           Database
Feeds         →    Pipeline        →    PostgreSQL/SQLite
(FIX, Binary)      (Streaming)          (Optimized for Time-Series)
    ↓                   ↓                     ↓
Real-time         Technical           Analytics &
Prices           Indicators          Reporting
```

## Features

### Performance Optimizations
- ✅ Zero-copy message processing
- ✅ Memory-mapped file handling
- ✅ Batch insertions with hypertables
- ✅ Parallel processing workers
- ✅ Lock-free data structures

### Market Data Features
- ✅ Multi-exchange data normalization
- ✅ Corporate actions processing
- ✅ Circuit breaker detection
- ✅ After-hours trading support
- ✅ Tick-by-tick data preservation

### Quality Controls
- ✅ Price reasonability checks
- ✅ Volume spike detection
- ✅ Duplicate message handling
- ✅ Out-of-order message correction
- ✅ Market hours validation

## Quick Start

### Prerequisites

```bash
# PostgreSQL database
docker run -d --name postgres -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:15

# Note: TimescaleDB, ClickHouse, and Redis integration planned for future versions
```

### Setup Database

```bash
# Create PostgreSQL schema
psql postgresql://postgres:password@localhost:5432/postgres < schema.sql

# Verify tables created
psql postgresql://postgres:password@localhost:5432/postgres -c "\dt"
```

### Run Sample Data

```bash
# High-frequency tick data ingestion
atomingest run pipeline.yaml --input sample_data/market_ticks.csv --stream

# Historical data backfill
atomingest run pipeline.yaml --input sample_data/historical_data.parquet --batch-size 50000
```

## Configuration

### Pipeline Configuration (`pipeline.yaml`)

Optimized for ultra-high frequency market data with:
- Streaming processing mode
- Time-series database optimizations
- Real-time technical indicators
- Market hours validation

### Market Data Schema

Supports multiple data types:
- **Level 1**: Best bid/offer, last trade
- **Level 2**: Full order book depth
- **Trades**: Individual trade records
- **OHLCV**: Aggregated bar data
- **Corporate Actions**: Splits, dividends, etc.

## Sample Data

### Market Ticks (`market_ticks.csv`)
Ultra-high frequency tick data with:
- Microsecond timestamps
- Bid/ask spreads
- Trade volumes
- Market maker IDs

### Historical Data (`historical_data.parquet`)
Daily OHLCV data for backtesting:
- 10+ years of S&P 500 data
- Adjusted for splits/dividends
- Multiple asset classes

### Corporate Actions (`corporate_actions.csv`)
Historical corporate actions for price adjustments

## Testing

### Performance Benchmarks
```bash
# Throughput testing
pytest tests/test_performance.py --benchmark --messages 1000000

# Latency testing
pytest tests/test_latency.py --p99-target 1ms

# Memory usage profiling
python tests/memory_profiler.py
```

### Market Data Validation
```bash
# Price reasonability tests
pytest tests/test_market_validation.py -k price_checks

# Market hours validation
pytest tests/test_market_validation.py -k market_hours

# Corporate actions processing
pytest tests/test_corporate_actions.py -v
```

## Monitoring

### Key Metrics
- Messages processed per second
- End-to-end latency (p50, p95, p99)
- Memory usage and GC pressure
- Database write throughput
- Technical indicator calculation time

### Alerts
- Processing latency > 1ms (p99)
- Message loss detected
- Price spike beyond circuit breakers
- Database connection failures
- Market data feed disconnections

## Integration

### Supported Data Sources
- **FIX Protocol**: Industry standard for institutional feeds
- **Binary Protocols**: Exchange-specific formats (ITCH, FAST, etc.)
- **WebSocket**: Real-time retail feeds
- **REST APIs**: For reference data and snapshots
- **Files**: CSV, Parquet, Avro for historical data

### Supported Databases
- **PostgreSQL**: Full-featured relational database (recommended)
- **SQLite**: Lightweight database for development and testing

*Note: TimescaleDB, ClickHouse, and InfluxDB support planned for future versions.*

### External Integrations
- **Bloomberg API**: Professional market data
- **Refinitiv Eikon**: Institutional data feeds
- **Alpha Vantage**: Retail market data API
- **IEX Cloud**: Real-time and historical data

## Advanced Features

### Real-Time Technical Indicators

```yaml
# Configure technical indicators
technical_indicators:
  sma:
    periods: [5, 10, 20, 50, 200]

  ema:
    periods: [12, 26]
    alpha: 0.1

  rsi:
    period: 14

  macd:
    fast: 12
    slow: 26
    signal: 9

  bollinger_bands:
    period: 20
    std_dev: 2
```

### Corporate Actions Processing

```yaml
# Automatic price adjustments
corporate_actions:
  splits:
    auto_adjust: true
    notification_webhook: "${SPLITS_WEBHOOK}"

  dividends:
    ex_dividend_adjustment: true
    reinvestment_calculation: true

  spinoffs:
    create_new_symbols: true
    adjust_historical_prices: true
```

### Market Hours & Calendar

```yaml
# Market hours validation
market_hours:
  NYSE:
    regular: "09:30-16:00 EST"
    extended: "04:00-20:00 EST"

  NASDAQ:
    regular: "09:30-16:00 EST"
    extended: "04:00-20:00 EST"

  LSE:
    regular: "08:00-16:30 GMT"

  holidays_calendar: "NYSE"
  reject_outside_hours: false
  flag_extended_hours: true
```

## Performance Tuning

### High-Frequency Optimizations

```yaml
performance:
  mode: "ultra_high_frequency"

  # Zero-copy processing
  zero_copy: true
  memory_mapped_io: true

  # Batch processing
  batch_size: 10000
  batch_timeout_ms: 10

  # Parallel processing
  worker_threads: 16
  io_threads: 4

  # Memory management
  preallocate_buffers: true
  buffer_pool_size: "1GB"
  gc_pressure_threshold: 0.8
```

### Database Optimizations

```yaml
database:
  # Connection pooling
  pool_size: 50
  pool_pre_ping: true

  # TimescaleDB specific
  chunk_time_interval: "1 hour"
  compression_enabled: true
  compression_interval: "7 days"

  # Bulk operations
  bulk_insert_size: 50000
  parallel_workers: 8
  disable_wal_during_load: true
```

## Compliance & Regulation

### MiFID II Compliance

```yaml
mifid_ii:
  # Best execution reporting
  execution_reporting: true

  # Transaction reporting
  transaction_reporting:
    enabled: true
    reporting_deadline_hours: 24

  # Clock synchronization
  clock_sync:
    ntp_servers: ["time.google.com"]
    max_drift_microseconds: 100
```

### Reg NMS Compliance

```yaml
reg_nms:
  # National best bid/offer
  nbbo_calculation: true

  # Order protection rule
  trade_through_protection: true

  # Access rule compliance
  fair_access_monitoring: true
```

## Troubleshooting

### Common Performance Issues

**High Latency**
```bash
# Check message queue depths
redis-cli INFO

# Monitor database performance
SELECT * FROM pg_stat_user_tables WHERE relname = 'market_data';

# Profile application
python -m cProfile -o profile.stats main.py
```

**Memory Issues**
```bash
# Monitor memory usage
python -m memory_profiler main.py

# Check for memory leaks
valgrind --tool=memcheck python main.py

# Analyze garbage collection
python -X dev main.py
```

**Data Quality Issues**
```bash
# Check for gaps in data
SELECT symbol,
       timestamp,
       LAG(timestamp) OVER (PARTITION BY symbol ORDER BY timestamp) as prev_timestamp,
       timestamp - LAG(timestamp) OVER (PARTITION BY symbol ORDER BY timestamp) as gap
FROM market_data
WHERE gap > INTERVAL '1 second'
ORDER BY gap DESC;

# Validate price movements
SELECT symbol, timestamp, price,
       LAG(price) OVER (PARTITION BY symbol ORDER BY timestamp) as prev_price,
       (price / LAG(price) OVER (PARTITION BY symbol ORDER BY timestamp) - 1) * 100 as price_change_pct
FROM market_data
WHERE ABS((price / LAG(price) OVER (PARTITION BY symbol ORDER BY timestamp) - 1) * 100) > 10
ORDER BY price_change_pct DESC;
```

## Deployment

### Production Deployment

```bash
# Kubernetes deployment with high-performance settings
kubectl apply -f k8s/market-data-deployment.yaml

# Monitor performance
kubectl top pods -l app=market-data-ingestion
```

### Environment Variables

```bash
# Database connections
export DB_URL="postgresql://user:pass@postgres:5432/marketdata"
# Note: Redis, TimescaleDB, and other databases planned for future versions

# Market data feeds
export BLOOMBERG_API_KEY="your_api_key"
export REFINITIV_TOKEN="your_token"

# Performance tuning
export WORKER_THREADS=16
export BATCH_SIZE=10000
export BUFFER_SIZE=1073741824  # 1GB
```

## Support

For high-performance market data ingestion support:
1. Check performance tuning guidelines
2. Review TimescaleDB optimization docs
3. Consult [Fintech User Guide](../../docs/fintech_user_guide.md)
4. Open GitHub issue with [market-data] tag
