{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# QSIPrep Basics: Diffusion MRI Preprocessing\n",
    "\n",
    "This notebook demonstrates how to use the QSIPrep runner for diffusion MRI preprocessing.\n",
    "\n",
    "## Overview\n",
    "\n",
    "QSIPrep configures pipelines for processing diffusion-weighted MRI (dMRI) data. It performs:\n",
    "- Head motion correction\n",
    "- Susceptibility distortion correction\n",
    "- Eddy current correction\n",
    "- Spatial normalization\n",
    "- Gradient direction correction\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "- Docker installed and running\n",
    "- BIDS-formatted diffusion data\n",
    "- FreeSurfer license file (required)\n",
    "- Sufficient disk space for working directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "from voxelops import (\n",
    "    QSIPrepDefaults,\n",
    "    QSIPrepInputs,\n",
    "    run_qsiprep,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input paths -- update these to match your data\n",
    "bids_dir = Path(\"/data/bids/\")\n",
    "participant = \"01\"\n",
    "fs_license = Path(\"/opt/freesurfer/license.txt\")\n",
    "\n",
    "# Output paths (optional)\n",
    "output_dir = Path(\"/data/derivatives/qsiprep/\")\n",
    "work_dir = Path(\"/data/work/qsiprep/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Usage\n",
    "\n",
    "### Option 1: Use Default Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create inputs\n",
    "inputs = QSIPrepInputs(\n",
    "    bids_dir=bids_dir,\n",
    "    participant=participant,\n",
    "    output_dir=output_dir,\n",
    "    work_dir=work_dir,\n",
    ")\n",
    "\n",
    "# Run with defaults (8 cores, 16GB RAM)\n",
    "result = run_qsiprep(\n",
    "    inputs,\n",
    "    fs_license=fs_license,  # Required\n",
    ")\n",
    "\n",
    "print(f\"Success: {result['success']}\")\n",
    "print(f\"Duration: {result['duration_human']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run with more resources\n",
    "result = run_qsiprep(\n",
    "    inputs,\n",
    "    fs_license=fs_license,\n",
    "    nprocs=16,  # Use 16 cores\n",
    "    mem_gb=32,  # Allocate 32GB RAM\n",
    ")\n",
    "\n",
    "print(f\"Success: {result['success']}\")\n",
    "print(f\"Used {result['config'].nprocs} cores and {result['config'].mem_gb}GB RAM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 3: Custom Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run with more resources\n",
    "result = run_qsiprep(\n",
    "    inputs,\n",
    "    fs_license=fs_license,\n",
    "    nprocs=16,       # Use 16 cores\n",
    "    mem_mb=32000,    # Allocate 32GB RAM\n",
    ")\n",
    "\n",
    "print(f\"Success: {result['success']}\")\n",
    "print(f\"Used {result['config'].nprocs} cores and {result['config'].mem_mb}MB RAM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration Priority\n",
    "\n",
    "Parameters can be specified in multiple ways with the following priority (highest to lowest):\n",
    "\n",
    "1. **Keyword arguments** (highest priority)\n",
    "2. **Environment variables** (if enabled)\n",
    "3. **Config object**\n",
    "4. **Defaults** (lowest priority)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create custom brain bank configuration\n",
    "config = QSIPrepDefaults(\n",
    "    nprocs=24,\n",
    "    mem_mb=64000,\n",
    "    output_resolution=1.5,  # 1.5mm isotropic\n",
    "    anatomical_template=[\"MNI152NLin2009cAsym\", \"T1w\"],  # Multiple spaces\n",
    "    longitudinal=True,  # Longitudinal processing\n",
    "    skip_bids_validation=False,  # Always validate BIDS\n",
    "    fs_license=fs_license,\n",
    "    docker_image=\"pennlinc/qsiprep:latest\",\n",
    ")\n",
    "\n",
    "# Run with custom config\n",
    "result = run_qsiprep(inputs, config)\n",
    "\n",
    "print(f\"Success: {result['success']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect Execution Record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Config says 16 cores, but keyword arg overrides to 32\n",
    "config = QSIPrepDefaults(\n",
    "    nprocs=16,\n",
    "    mem_mb=32000,\n",
    "    fs_license=fs_license,\n",
    ")\n",
    "\n",
    "result = run_qsiprep(\n",
    "    inputs,\n",
    "    config,\n",
    "    nprocs=32,  # This wins!\n",
    ")\n",
    "\n",
    "print(f\"Actually used {result['config'].nprocs} cores\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Expected Outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execution metadata\n",
    "print(\"Execution Details:\")\n",
    "print(f\"  Tool: {result['tool']}\")\n",
    "print(f\"  Participant: {result['participant']}\")\n",
    "print(f\"  Start: {result['start_time']}\")\n",
    "print(f\"  End: {result['end_time']}\")\n",
    "print(f\"  Duration: {result['duration_human']} ({result['duration_seconds']:.1f}s)\")\n",
    "print(f\"  Success: {result['success']}\")\n",
    "print(f\"  Exit code: {result['exit_code']}\")\n",
    "\n",
    "# Configuration used\n",
    "print(\"\\nConfiguration:\")\n",
    "config_used = result[\"config\"]\n",
    "print(f\"  Cores: {config_used.nprocs}\")\n",
    "print(f\"  Memory: {config_used.mem_mb}MB\")\n",
    "print(f\"  Output resolution: {config_used.output_resolution}mm\")\n",
    "print(f\"  Anatomical template: {config_used.anatomical_template}\")\n",
    "print(f\"  Longitudinal: {config_used.longitudinal}\")\n",
    "print(f\"  Docker image: {config_used.docker_image}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View HTML Report\n",
    "\n",
    "QSIPrep generates a comprehensive HTML quality control report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "# Display HTML report in notebook\n",
    "if outputs.html_report.exists():\n",
    "    IFrame(src=str(outputs.html_report), width=900, height=600)\n",
    "else:\n",
    "    print(f\"HTML report not found at: {outputs.html_report}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List Output Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List all preprocessed files\n",
    "if outputs.participant_dir.exists():\n",
    "    print(f\"Files in {outputs.participant_dir}:\\n\")\n",
    "\n",
    "    for f in sorted(outputs.participant_dir.rglob(\"*\")):\n",
    "        if f.is_file():\n",
    "            rel_path = f.relative_to(outputs.participant_dir)\n",
    "            size_mb = f.stat().st_size / (1024 * 1024)\n",
    "            print(f\"  {rel_path} ({size_mb:.1f} MB)\")\n",
    "else:\n",
    "    print(\"Participant directory not found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error Handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from voxelops.exceptions import (\n",
    "    FreeSurferLicenseError,\n",
    "    InputValidationError,\n",
    "    ProcedureExecutionError,\n",
    ")\n",
    "\n",
    "try:\n",
    "    result = run_qsiprep(\n",
    "        inputs,\n",
    "        fs_license=fs_license,\n",
    "        nprocs=16,\n",
    "    )\n",
    "    print(f\"Success: {result['success']}\")\n",
    "\n",
    "except InputValidationError as e:\n",
    "    print(f\"Input validation failed: {e}\")\n",
    "    print(\"  - Check that BIDS directory exists\")\n",
    "    print(\"  - Check that participant exists in BIDS directory\")\n",
    "\n",
    "except FreeSurferLicenseError as e:\n",
    "    print(f\"FreeSurfer license error: {e}\")\n",
    "    print(\n",
    "        \"  - Obtain license from https://surfer.nmr.mgh.harvard.edu/registration.html\"\n",
    "    )\n",
    "\n",
    "except ProcedureExecutionError as e:\n",
    "    print(f\"Execution failed: {e}\")\n",
    "    print(f\"  - Check log file: {result.get('log_file')}\")\n",
    "    print(\"  - Check stderr output\")\n",
    "    if \"stderr\" in result:\n",
    "        print(f\"\\nStderr (last 500 chars):\\n{result['stderr'][-500:]}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"Unexpected error: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# Get list of participants from BIDS directory\n",
    "participant_dirs = sorted(bids_dir.glob(\"sub-*\"))\n",
    "participants = [d.name.replace(\"sub-\", \"\") for d in participant_dirs if d.is_dir()]\n",
    "\n",
    "print(f\"Found {len(participants)} participants: {participants}\\n\")\n",
    "\n",
    "# Process each participant\n",
    "results = []\n",
    "config = QSIPrepDefaults(\n",
    "    nprocs=16,\n",
    "    mem_gb=32,\n",
    "    fs_license=fs_license,\n",
    ")\n",
    "\n",
    "for participant in participants:\n",
    "    print(f\"Processing participant {participant}...\")\n",
    "\n",
    "    inputs = QSIPrepInputs(\n",
    "        bids_dir=bids_dir,\n",
    "        participant=participant,\n",
    "    )\n",
    "\n",
    "    try:\n",
    "        start = time.time()\n",
    "        result = run_qsiprep(inputs, config)\n",
    "        elapsed = time.time() - start\n",
    "\n",
    "        results.append(result)\n",
    "        print(f\"  ✓ Success in {elapsed/60:.1f} minutes\\n\")\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"  ✗ Failed: {e}\\n\")\n",
    "        results.append(\n",
    "            {\n",
    "                \"participant\": participant,\n",
    "                \"success\": False,\n",
    "                \"error\": str(e),\n",
    "            }\n",
    "        )\n",
    "\n",
    "# Summary\n",
    "successful = sum(1 for r in results if r.get(\"success\"))\n",
    "total_time = sum(\n",
    "    r.get(\"duration_seconds\", 0) for r in results if \"duration_seconds\" in r\n",
    ")\n",
    "\n",
    "print(f\"\\n{'='*60}\")\n",
    "print(f\"Processed {len(results)} participants:\")\n",
    "print(f\"  ✓ Successful: {successful}\")\n",
    "print(f\"  ✗ Failed: {len(results) - successful}\")\n",
    "print(f\"  Total time: {total_time/3600:.1f} hours\")\n",
    "print(f\"  Average time: {total_time/len(results)/60:.1f} minutes per participant\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Execution Records to Database\n",
    "\n",
    "The execution records are plain Python dicts, making them easy to store in any database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# Get list of participants from BIDS directory\n",
    "participant_dirs = sorted(bids_dir.glob(\"sub-*\"))\n",
    "participants = [d.name.replace(\"sub-\", \"\") for d in participant_dirs if d.is_dir()]\n",
    "\n",
    "print(f\"Found {len(participants)} participants: {participants}\\n\")\n",
    "\n",
    "# Process each participant\n",
    "results = []\n",
    "config = QSIPrepDefaults(\n",
    "    nprocs=16,\n",
    "    mem_mb=32000,\n",
    "    fs_license=fs_license,\n",
    ")\n",
    "\n",
    "for participant in participants:\n",
    "    print(f\"Processing participant {participant}...\")\n",
    "\n",
    "    inputs = QSIPrepInputs(\n",
    "        bids_dir=bids_dir,\n",
    "        participant=participant,\n",
    "    )\n",
    "\n",
    "    try:\n",
    "        start = time.time()\n",
    "        result = run_qsiprep(inputs, config)\n",
    "        elapsed = time.time() - start\n",
    "\n",
    "        results.append(result)\n",
    "        print(f\"  Success in {elapsed/60:.1f} minutes\\n\")\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"  Failed: {e}\\n\")\n",
    "        results.append(\n",
    "            {\n",
    "                \"participant\": participant,\n",
    "                \"success\": False,\n",
    "                \"error\": str(e),\n",
    "            }\n",
    "        )\n",
    "\n",
    "# Summary\n",
    "successful = sum(1 for r in results if r.get(\"success\"))\n",
    "total_time = sum(\n",
    "    r.get(\"duration_seconds\", 0) for r in results if \"duration_seconds\" in r\n",
    ")\n",
    "\n",
    "print(f\"\\n{'='*60}\")\n",
    "print(f\"Processed {len(results)} participants:\")\n",
    "print(f\"  Successful: {successful}\")\n",
    "print(f\"  Failed: {len(results) - successful}\")\n",
    "print(f\"  Total time: {total_time/3600:.1f} hours\")\n",
    "if results:\n",
    "    print(f\"  Average time: {total_time/len(results)/60:.1f} minutes per participant\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reproduce a Previous Run\n",
    "\n",
    "Since the exact Docker command is stored, you can reproduce any run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a previous execution record\n",
    "previous_record = json.loads((records_dir / \"01.json\").read_text())\n",
    "\n",
    "# The exact command is stored\n",
    "cmd = previous_record[\"command\"]\n",
    "print(\"To reproduce this exact run, execute:\")\n",
    "print()\n",
    "print(\" \".join(cmd))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "After QSIPrep preprocessing:\n",
    "\n",
    "1. Review the HTML QC report\n",
    "2. Check the preprocessed images\n",
    "3. Proceed to reconstruction with QSIRecon (see `03_qsirecon_basics.ipynb`)\n",
    "\n",
    "## Tips\n",
    "\n",
    "- **Resource allocation**: QSIPrep is memory-intensive. Allocate at least 16GB RAM.\n",
    "- **Working directory**: Use fast local storage (SSD) for the work directory.\n",
    "- **Longitudinal processing**: Enable for subjects with multiple sessions.\n",
    "- **BIDS validation**: Always validate your BIDS dataset before running.\n",
    "- **FreeSurfer license**: Required even if not running FreeSurfer reconstruction.\n",
    "- **Output spaces**: Choose spaces based on downstream analysis needs.\n",
    "- **Docker version**: Pin a specific image version for reproducibility."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "VoxelOps",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}