# 🧬 SynthNeura

SynthNeura is an early-stage, modular AI and data-engineering platform designed to accelerate clinical research, pharmaceutical intelligence, and personalized medicine.

Core objective:
SynthNeura is built to become a reliable pharma intelligence engine, not just a scraper or a chatbot. It autonomously ingests biomedical data, normalizes it into structured knowledge, and produces reliable, repeatable insights for research, strategy, and decision-making.

Built with a production-grade engineering mindset: clean architecture, strict linting, automated CI/CD, and reproducible workflows.

---

## 🚀 Current Capabilities (V2)

### 🔍 Clinical Trials Ingestion
- Fetches trials from ClinicalTrials.gov (API v2)
- Normalizes heterogeneous responses into a clean internal schema
- Handles missing fields and unexpected response shapes
- Persists structured data to SQLite for local analysis
- Tracks run metadata and stores change history

### 🧱 Normalization & Storage Pipeline
- Strongly typed domain model (`ClinicalTrial`)
- Robust normalization for legacy + modern API formats
- Schema-safe inserts with idempotent upserts
- Storage abstraction via sinks (SQLite default)
- History table for prior versions of updated trials

### 🖥️ Command Line Interface (CLI)

Run the full pipeline:

```bash
python -m synthneura.ui.cli --query "lung cancer"
```

Write results to JSON or CSV:

```bash
python -m synthneura.ui.cli --query "lung cancer" --output-path out.json
python -m synthneura.ui.cli --query "lung cancer" --output-path out.csv --output-format csv
```

Select a storage backend:

```bash
python -m synthneura.ui.cli --query "lung cancer" --sink sqlite --db-path trials.db
```

Postgres (Supabase/Render/Neon):

```bash
export SYNTHNEURA_DB_URL="postgresql://user:pass@host:5432/dbname"
python -m synthneura.ui.cli --query "lung cancer" --sink postgres --db-url "$SYNTHNEURA_DB_URL"
```

Run AI-assisted extraction (endpoints + biomarkers):

```bash
export SYNTHNEURA_OPENAI_API_KEY="your-key"
python -m synthneura.ui.cli --query "lung cancer" --extract
```

Prompt versioning (why it exists):
- `prompt_version` tracks which extraction prompt/schema produced each output.
- It prevents confusion between *data changes* and *prompt changes*.
- It enables reproducibility, auditability, and safe upgrades.
- You only bump it when you intentionally change the prompt/schema/normalization rules.

Alternative approaches (not the only way):
- Store a hash of the prompt text instead of a friendly version.
- Store both a version and a prompt hash for stronger traceability.
- Use model version + schema version only (less precise than prompt versioning).

CLI features:
- Configurable logging levels
- Clear user-friendly output
- Structured debug logs for developers
- Summary stats generated from live trial data
- Change detection (new vs updated vs unchanged)
- Change summaries (top changed trials + changed fields)
- AI-assisted extraction for endpoints and biomarkers (optional)

Logging notes:
- `--log-level` controls all module loggers and handlers for the current run.

Output notes:
- JSON outputs include `summary` and `trials`.
- CSV outputs write trials to the CSV and a summary file at `<output>.summary.json` (or pass `--summary-path`).
- Change summaries are written to `<output>.changes.json` when `--output-path` is set (or pass `--changes-path`).
- Use `--since-last-run` to emit diffs since the previous run when writing change summaries.
- Use `--extract` to run AI extraction for endpoints and biomarkers.
- When extraction runs, the change summary JSON includes extraction counts.

---

## 🗄️ Storage Schema (SQLite & Postgres)

Tables created automatically in the SQLite database:

- `clinical_trials` (latest state)
  - `nct_id` (primary key)
  - normalized fields: `title`, `status`, `phase`, `sponsor`, `conditions`, `interventions`, `outcomes`
  - `raw_json` (full payload)
  - `source`, `query`, `ingested_at`, `run_id`

- `clinical_trials_history` (prior versions on update)
  - same fields as `clinical_trials`
  - `valid_from`, `valid_to`

- `runs` (run metadata + counts)
  - `run_id`, `ingested_at`, `source`, `query`
  - `total`, `new_count`, `updated_count`, `unchanged_count`, `failed_count`

- `trial_diffs` (field-level changes for updates)
  - `nct_id`, `run_id`, `ingested_at`
  - `changed_fields`, `before_json`, `after_json`

- `trial_extractions` (AI-extracted facts)
  - `nct_id`, `run_id`, `ingested_at`
  - `endpoints_json`, `biomarkers_json`, `provider`, `model`
  - `prompt_version`, `raw_response`, `validation_status`, `validation_errors`, `confidence`

- `trial_extraction_failures` (failed extraction attempts)
  - `nct_id`, `run_id`, `ingested_at`
  - `provider`, `model`, `error`

Use cases:
- `clinical_trials` answers "what is the latest state?"
- `clinical_trials_history` answers "what changed and when?"
- `runs` answers "what happened in each run?"
- `trial_diffs` answers "which fields changed and how?"
- `trial_extractions` answers "what endpoints/biomarkers are present?"
- `trial_extraction_failures` answers "what failed and why?"

Indexes:
- `clinical_trials(nct_id)`
- `clinical_trials_history(nct_id, valid_from)`
- `trial_diffs(nct_id, ingested_at)`
- `trial_extractions(nct_id, run_id)` (unique)

---

## 🧰 Supabase Postgres Setup (recommended)

1) Create a new Supabase project
2) Copy the **connection string** from the Database settings
3) Set your environment variable:

```bash
export SYNTHNEURA_DB_URL="postgresql://user:pass@host:5432/dbname"
```

4) Run the pipeline with the Postgres sink:

```bash
python -m synthneura.ui.cli --query "lung cancer" --sink postgres
```

The schema is created automatically on first run.

---

## ⚙️ Settings (Environment Variables)

- `SYNTHNEURA_DB_PATH` (default: `trials.db`)
- `SYNTHNEURA_SINK` (default: `sqlite`)
- `SYNTHNEURA_LOG_LEVEL` (default: `INFO`)
- `SYNTHNEURA_OUTPUT_PATH` (default: unset)
- `SYNTHNEURA_OUTPUT_FORMAT` (default: `json`)
- `SYNTHNEURA_SUMMARY_PATH` (default: unset)
- `SYNTHNEURA_CHANGES_PATH` (default: unset)
- `SYNTHNEURA_LLM_PROVIDER` (default: `openai`)
- `SYNTHNEURA_OPENAI_API_KEY` (default: unset)
- `SYNTHNEURA_OPENAI_MODEL` (default: `gpt-4o-mini`)
- `SYNTHNEURA_LLM_RETRIES` (default: `3`)
- `SYNTHNEURA_LLM_BACKOFF_SECONDS` (default: `1.0`)
- `SYNTHNEURA_LLM_REQUEST_DELAY_SECONDS` (default: `0.0`)
- Supported providers: `openai` (implemented), `local-mock` (implemented), `anthropic` and `local` (stubs).
- `SYNTHNEURA_MAX_RESULTS` (default: `10`)

Example:

```bash
export SYNTHNEURA_DB_PATH="data/trials.db"
export SYNTHNEURA_SINK="sqlite"
export SYNTHNEURA_LLM_PROVIDER="openai"
export SYNTHNEURA_OPENAI_API_KEY="your-key"
export SYNTHNEURA_LLM_RETRIES=3
export SYNTHNEURA_LLM_BACKOFF_SECONDS=1.5
export SYNTHNEURA_LLM_REQUEST_DELAY_SECONDS=0.5
export SYNTHNEURA_MAX_RESULTS=50
python -m synthneura.ui.cli --query "lung cancer"
```

Mock extraction (no API key required):

```bash
export SYNTHNEURA_LLM_PROVIDER="local-mock"
python -m synthneura.ui.cli --query "lung cancer" --extract
```

---

## 🧪 Engineering & Quality Standards

Formatting and linting:
- Black
- isort
- Ruff
- Flake8
- Mypy

All checks are enforced via pre-commit hooks locally and in CI.

---

## 🔄 CI/CD Pipeline

Branch strategy:
- `feature/*` / `dev/*` → active development
- `main` → QA / staging
- Releases → production

Pipeline behavior:
- Every push / PR runs pre-commit + tests
- Merge to `main` triggers QA deployment
- Release published triggers production deployment

---

## 🛠️ Developer Workflow

Common commands:

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run all linting & checks (same as CI)
make lint

# Auto-format code
make format

# Run tests
make test
```

Commit flow:

```bash
git add -A
git commit -m "Meaningful message"
git push
```

---

## 🖥️ Dashboard (Local UI)

The dashboard is a lightweight React UI with a tiny Flask API that reads `trials.db`.

### Run the API

```bash
export SYNTHNEURA_DB_PATH="trials.db"
export SYNTHNEURA_API_PORT=5001
python -m synthneura.ui.web.app
```

### Run the frontend

```bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
nvm use 20
cd frontend
npm install
export VITE_API_BASE="http://localhost:5001"
npm run dev -- --host
```

Open the UI at `http://localhost:5173` and the API at `http://localhost:5001`.

---

## 🧠 Design Philosophy

- Explicit typing
- Clear boundaries between ingestion, normalization, and storage
- Fail-fast behavior for invalid or unexpected data
- Minimal but extensible architecture

## 🔭 Long-Term Vision

- Multi-source biomedical data ingestion
- AI-driven analysis (GNNs, transformers, recommendation systems)
- Scalable APIs and enterprise-ready deployments
