{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Full Pipeline: DICOM to Connectivity\n",
    "\n",
    "This notebook demonstrates running the complete neuroimaging pipeline:\n",
    "\n",
    "1. **HeudiConv**: DICOM → BIDS conversion\n",
    "2. **QSIPrep**: Diffusion preprocessing\n",
    "3. **QSIRecon**: Reconstruction and tractography\n",
    "4. **QSIParc**: Parcellation and regional analysis\n",
    "\n",
    "## Overview\n",
    "\n",
    "This end-to-end pipeline takes raw DICOM data and produces:\n",
    "- BIDS-formatted dataset\n",
    "- Preprocessed diffusion images\n",
    "- Structural connectivity matrices\n",
    "- Regional diffusion metrics\n",
    "- Quality control reports at each stage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import time\n",
    "from datetime import datetime\n",
    "from pathlib import Path\n",
    "\n",
    "from voxelops import (\n",
    "    HeudiconvDefaults,\n",
    "    HeudiconvInputs,\n",
    "    QSIParcDefaults,\n",
    "    QSIParcInputs,\n",
    "    QSIPrepDefaults,\n",
    "    QSIPrepInputs,\n",
    "    QSIReconDefaults,\n",
    "    QSIReconInputs,\n",
    "    run_heudiconv,\n",
    "    run_qsiparc,\n",
    "    run_qsiprep,\n",
    "    run_qsirecon,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "Define all paths and parameters for the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data directories -- update these to match your data\n",
    "dicom_dir = Path(\"/data/raw/dicom\")\n",
    "bids_dir = Path(\"/data/bids\")\n",
    "derivatives_dir = Path(\"/data/derivatives\")\n",
    "work_dir = Path(\"/data/work\")\n",
    "\n",
    "# Configuration files\n",
    "heuristic_file = Path(\"/config/heuristics/brain_bank.py\")\n",
    "recon_spec = Path(\"/config/recon_specs/dsi_studio_gqi.json\")\n",
    "fs_license = Path(\"/opt/freesurfer/license.txt\")\n",
    "\n",
    "# Participant to process\n",
    "participant = \"01\"\n",
    "\n",
    "# Create brain bank standard configurations\n",
    "heudiconv_config = HeudiconvDefaults(\n",
    "    overwrite=False,\n",
    "    bids_validator=True,\n",
    ")\n",
    "\n",
    "qsiprep_config = QSIPrepDefaults(\n",
    "    nprocs=16,\n",
    "    mem_mb=32000,\n",
    "    output_resolution=1.6,\n",
    "    longitudinal=False,\n",
    "    fs_license=fs_license,\n",
    ")\n",
    "\n",
    "qsirecon_config = QSIReconDefaults(\n",
    "    nprocs=24,\n",
    "    mem_mb=48000,\n",
    "    fs_license=fs_license,\n",
    ")\n",
    "\n",
    "qsiparc_config = QSIParcDefaults(\n",
    "    n_jobs=4,\n",
    "    n_procs=2,\n",
    ")\n",
    "\n",
    "print(\"Pipeline configuration:\")\n",
    "print(f\"  Participant: {participant}\")\n",
    "print(f\"  DICOM directory: {dicom_dir}\")\n",
    "print(f\"  BIDS directory: {bids_dir}\")\n",
    "print(f\"  Derivatives: {derivatives_dir}\")\n",
    "print(f\"  Work directory: {work_dir}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline Execution\n",
    "\n",
    "### Step 1: DICOM to BIDS Conversion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"STEP 1: DICOM to BIDS Conversion (HeudiConv)\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "heudiconv_inputs = HeudiconvInputs(\n",
    "    dicom_dir=dicom_dir,\n",
    "    participant=participant,\n",
    "    heuristic=heuristic_file,\n",
    "    output_dir=bids_dir,\n",
    ")\n",
    "\n",
    "start_time = time.time()\n",
    "try:\n",
    "    heudiconv_result = run_heudiconv(heudiconv_inputs, heudiconv_config)\n",
    "    elapsed = time.time() - start_time\n",
    "\n",
    "    print(f\"\\nHeudiConv completed successfully in {elapsed/60:.1f} minutes\")\n",
    "    print(f\"  BIDS directory: {heudiconv_result['expected_outputs'].bids_dir}\")\n",
    "    print(f\"  Exit code: {heudiconv_result['exit_code']}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"\\nHeudiConv failed: {e}\")\n",
    "    print(\"Pipeline aborted.\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Diffusion Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\n\" + \"=\" * 60)\n",
    "print(\"STEP 2: Diffusion Preprocessing (QSIPrep)\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "qsiprep_inputs = QSIPrepInputs(\n",
    "    bids_dir=heudiconv_result[\"expected_outputs\"].bids_dir,\n",
    "    participant=participant,\n",
    "    output_dir=derivatives_dir / \"qsiprep\",\n",
    "    work_dir=work_dir / \"qsiprep\",\n",
    ")\n",
    "\n",
    "start_time = time.time()\n",
    "try:\n",
    "    qsiprep_result = run_qsiprep(qsiprep_inputs, qsiprep_config)\n",
    "    elapsed = time.time() - start_time\n",
    "\n",
    "    print(f\"\\n✓ QSIPrep completed successfully in {elapsed/60:.1f} minutes\")\n",
    "    print(f\"  Output directory: {qsiprep_result['expected_outputs'].qsiprep_dir}\")\n",
    "    print(f\"  HTML report: {qsiprep_result['expected_outputs'].html_report}\")\n",
    "    print(f\"  Exit code: {qsiprep_result['exit_code']}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"\\n✗ QSIPrep failed: {e}\")\n",
    "    print(\"Pipeline aborted.\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Reconstruction and Tractography"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\n\" + \"=\" * 60)\n",
    "print(\"STEP 3: Reconstruction and Tractography (QSIRecon)\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "qsirecon_inputs = QSIReconInputs(\n",
    "    qsiprep_dir=qsiprep_result[\"expected_outputs\"].qsiprep_dir,\n",
    "    participant=participant,\n",
    "    recon_spec=recon_spec,\n",
    "    output_dir=derivatives_dir / \"qsirecon\",\n",
    "    work_dir=work_dir / \"qsirecon\",\n",
    ")\n",
    "\n",
    "start_time = time.time()\n",
    "try:\n",
    "    qsirecon_result = run_qsirecon(qsirecon_inputs, qsirecon_config)\n",
    "    elapsed = time.time() - start_time\n",
    "\n",
    "    print(f\"\\n✓ QSIRecon completed successfully in {elapsed/60:.1f} minutes\")\n",
    "    print(f\"  Output directory: {qsirecon_result['expected_outputs'].qsirecon_dir}\")\n",
    "    print(f\"  HTML report: {qsirecon_result['expected_outputs'].html_report}\")\n",
    "    print(f\"  Exit code: {qsirecon_result['exit_code']}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"\\n✗ QSIRecon failed: {e}\")\n",
    "    print(\"Pipeline aborted.\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Parcellation and Regional Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\n\" + \"=\" * 60)\n",
    "print(\"STEP 4: Parcellation and Regional Analysis (QSIParc)\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "qsiparc_inputs = QSIParcInputs(\n",
    "    qsirecon_dir=qsirecon_result[\"expected_outputs\"].qsirecon_dir,\n",
    "    participant=participant,\n",
    "    output_dir=derivatives_dir / \"qsiparc\",\n",
    ")\n",
    "\n",
    "start_time = time.time()\n",
    "try:\n",
    "    qsiparc_result = run_qsiparc(qsiparc_inputs, qsiparc_config)\n",
    "    elapsed = time.time() - start_time\n",
    "\n",
    "    print(f\"\\nQSIParc completed successfully in {elapsed/60:.1f} minutes\")\n",
    "    print(f\"  Output directory: {qsiparc_result['expected_outputs'].output_dir}\")\n",
    "    print(f\"  Output files: {len(qsiparc_result.get('output_files', []))}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"\\nQSIParc failed: {e}\")\n",
    "    print(\"Pipeline aborted.\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\n\" + \"=\" * 60)\n",
    "print(\"PIPELINE SUMMARY\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "# Collect all results\n",
    "results = [\n",
    "    (\"HeudiConv\", heudiconv_result),\n",
    "    (\"QSIPrep\", qsiprep_result),\n",
    "    (\"QSIRecon\", qsirecon_result),\n",
    "    (\"QSIParc\", qsiparc_result),\n",
    "]\n",
    "\n",
    "# Calculate total time\n",
    "total_duration = sum(\n",
    "    r.get(\"duration_seconds\", 0) for _, r in results\n",
    ")\n",
    "\n",
    "print(f\"\\nParticipant: {participant}\")\n",
    "print(f\"Total processing time: {total_duration/3600:.1f} hours\\n\")\n",
    "\n",
    "# Individual step timings\n",
    "print(\"Step-by-step breakdown:\")\n",
    "for name, result in results:\n",
    "    duration = result.get(\"duration_seconds\", 0)\n",
    "    success = result.get(\"success\", False)\n",
    "    status = \"OK\" if success else \"FAIL\"\n",
    "    pct = (duration / total_duration * 100) if total_duration else 0\n",
    "    print(f\"  [{status}] {name:12s}: {duration/60:6.1f} minutes ({pct:5.1f}%)\")\n",
    "\n",
    "# Output locations\n",
    "print(\"\\nOutput locations:\")\n",
    "print(f\"  BIDS dataset: {heudiconv_result['expected_outputs'].bids_dir}\")\n",
    "print(f\"  QSIPrep: {qsiprep_result['expected_outputs'].qsiprep_dir}\")\n",
    "print(f\"  QSIRecon: {qsirecon_result['expected_outputs'].qsirecon_dir}\")\n",
    "print(f\"  QSIParc: {qsiparc_result['expected_outputs'].output_dir}\")\n",
    "\n",
    "# QC Reports\n",
    "print(\"\\nQuality control reports:\")\n",
    "print(f\"  QSIPrep: {qsiprep_result['expected_outputs'].html_report}\")\n",
    "print(f\"  QSIRecon: {qsirecon_result['expected_outputs'].html_report}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Pipeline Record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create comprehensive pipeline record\n",
    "pipeline_record = {\n",
    "    \"participant\": participant,\n",
    "    \"pipeline_version\": \"2.0\",\n",
    "    \"execution_date\": datetime.now().isoformat(),\n",
    "    \"total_duration_seconds\": total_duration,\n",
    "    \"total_duration_hours\": total_duration / 3600,\n",
    "    \"steps\": {\n",
    "        \"heudiconv\": heudiconv_result,\n",
    "        \"qsiprep\": qsiprep_result,\n",
    "        \"qsirecon\": qsirecon_result,\n",
    "        \"qsiparc\": qsiparc_result,\n",
    "    },\n",
    "    \"outputs\": {\n",
    "        \"bids_dir\": str(heudiconv_result[\"expected_outputs\"].bids_dir),\n",
    "        \"qsiprep_dir\": str(qsiprep_result[\"expected_outputs\"].qsiprep_dir),\n",
    "        \"qsirecon_dir\": str(qsirecon_result[\"expected_outputs\"].qsirecon_dir),\n",
    "        \"qsiparc_dir\": str(qsiparc_result[\"expected_outputs\"].output_dir),\n",
    "    },\n",
    "}\n",
    "\n",
    "# Save to JSON\n",
    "record_dir = Path(\"/data/records/pipeline\")\n",
    "record_dir.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "record_file = record_dir / f\"{participant}_pipeline.json\"\n",
    "\n",
    "with open(record_file, \"w\") as f:\n",
    "    json.dump(pipeline_record, f, indent=2, default=str)\n",
    "\n",
    "print(f\"Pipeline record saved to: {record_file}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Processing Multiple Participants\n",
    "\n",
    "Process multiple participants through the full pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_full_pipeline(participant, configs):\n",
    "    \"\"\"Run the full pipeline for a single participant.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    participant : str\n",
    "        Participant ID (without 'sub-' prefix).\n",
    "    configs : dict\n",
    "        Dict with 'heudiconv', 'qsiprep', 'qsirecon', 'qsiparc' config objects.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    dict\n",
    "        Results from all steps.\n",
    "    \"\"\"\n",
    "    results = {}\n",
    "    pipeline_start = time.time()\n",
    "\n",
    "    try:\n",
    "        # Step 1: HeudiConv\n",
    "        print(\"  [1/4] Running HeudiConv...\")\n",
    "        heudiconv_inputs = HeudiconvInputs(\n",
    "            dicom_dir=dicom_dir,\n",
    "            participant=participant,\n",
    "            heuristic=heuristic_file,\n",
    "            output_dir=bids_dir,\n",
    "        )\n",
    "        results[\"heudiconv\"] = run_heudiconv(heudiconv_inputs, configs[\"heudiconv\"])\n",
    "\n",
    "        # Step 2: QSIPrep\n",
    "        print(\"  [2/4] Running QSIPrep...\")\n",
    "        qsiprep_inputs = QSIPrepInputs(\n",
    "            bids_dir=results[\"heudiconv\"][\"expected_outputs\"].bids_dir,\n",
    "            participant=participant,\n",
    "        )\n",
    "        results[\"qsiprep\"] = run_qsiprep(qsiprep_inputs, configs[\"qsiprep\"])\n",
    "\n",
    "        # Step 3: QSIRecon\n",
    "        print(\"  [3/4] Running QSIRecon...\")\n",
    "        qsirecon_inputs = QSIReconInputs(\n",
    "            qsiprep_dir=results[\"qsiprep\"][\"expected_outputs\"].qsiprep_dir,\n",
    "            participant=participant,\n",
    "            recon_spec=recon_spec,\n",
    "        )\n",
    "        results[\"qsirecon\"] = run_qsirecon(qsirecon_inputs, configs[\"qsirecon\"])\n",
    "\n",
    "        # Step 4: QSIParc\n",
    "        print(\"  [4/4] Running QSIParc...\")\n",
    "        qsiparc_inputs = QSIParcInputs(\n",
    "            qsirecon_dir=results[\"qsirecon\"][\"expected_outputs\"].qsirecon_dir,\n",
    "            participant=participant,\n",
    "        )\n",
    "        results[\"qsiparc\"] = run_qsiparc(qsiparc_inputs, configs[\"qsiparc\"])\n",
    "\n",
    "        # Success\n",
    "        pipeline_duration = time.time() - pipeline_start\n",
    "        results[\"success\"] = True\n",
    "        results[\"pipeline_duration\"] = pipeline_duration\n",
    "\n",
    "        return results\n",
    "\n",
    "    except Exception as e:\n",
    "        pipeline_duration = time.time() - pipeline_start\n",
    "        results[\"success\"] = False\n",
    "        results[\"error\"] = str(e)\n",
    "        results[\"pipeline_duration\"] = pipeline_duration\n",
    "        return results\n",
    "\n",
    "\n",
    "# Process multiple participants\n",
    "participants = [\"01\", \"02\", \"03\"]\n",
    "\n",
    "configs = {\n",
    "    \"heudiconv\": heudiconv_config,\n",
    "    \"qsiprep\": qsiprep_config,\n",
    "    \"qsirecon\": qsirecon_config,\n",
    "    \"qsiparc\": qsiparc_config,\n",
    "}\n",
    "\n",
    "batch_results = []\n",
    "\n",
    "print(f\"\\nProcessing {len(participants)} participants through full pipeline...\\n\")\n",
    "\n",
    "for i, participant in enumerate(participants, 1):\n",
    "    print(f\"[{i}/{len(participants)}] Processing participant {participant}\")\n",
    "\n",
    "    result = run_full_pipeline(participant, configs)\n",
    "    batch_results.append(result)\n",
    "\n",
    "    if result[\"success\"]:\n",
    "        duration = result[\"pipeline_duration\"]\n",
    "        print(f\"  Completed in {duration/3600:.1f} hours\\n\")\n",
    "    else:\n",
    "        print(f\"  Failed: {result['error']}\\n\")\n",
    "\n",
    "# Batch summary\n",
    "successful = sum(1 for r in batch_results if r[\"success\"])\n",
    "total_time = sum(r[\"pipeline_duration\"] for r in batch_results)\n",
    "\n",
    "print(\"=\" * 60)\n",
    "print(\"BATCH PROCESSING SUMMARY\")\n",
    "print(\"=\" * 60)\n",
    "print(f\"Total participants: {len(batch_results)}\")\n",
    "print(f\"  Successful: {successful}\")\n",
    "print(f\"  Failed: {len(batch_results) - successful}\")\n",
    "print(f\"Total processing time: {total_time/3600:.1f} hours\")\n",
    "if batch_results:\n",
    "    print(f\"Average time per participant: {total_time/len(batch_results)/3600:.1f} hours\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "After completing the full pipeline:\n",
    "\n",
    "1. **Quality Control**: Review HTML reports for each step\n",
    "2. **Data Validation**: Check that all expected outputs exist\n",
    "3. **Statistical Analysis**: Analyze connectivity matrices and regional metrics\n",
    "4. **Visualization**: Create figures for publication\n",
    "5. **Database Integration**: Save execution records to your brain bank database\n",
    "\n",
    "## Tips for Production\n",
    "\n",
    "- **Workflow Manager**: Use Airflow, Prefect, or Nextflow for scheduling\n",
    "- **Error Handling**: Implement retry logic and notifications\n",
    "- **Resource Management**: Monitor disk space and memory usage\n",
    "- **Parallel Processing**: Run independent participants in parallel\n",
    "- **Checkpointing**: Save results after each step to enable restarts\n",
    "- **Logging**: Store all execution records for audit trail\n",
    "- **Version Control**: Pin Docker image versions for reproducibility"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}