{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0",
   "metadata": {
    "papermill": {
     "duration": 0.056606,
     "end_time": "2025-12-08T15:10:42.241501",
     "exception": false,
     "start_time": "2025-12-08T15:10:42.184895",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# QC Dashboards and Visualizations\n",
    "\n",
    "This tutorial shows how to generate visual QC dashboards from the command line.\n",
    "\n",
    "## Dashboard Options\n",
    "\n",
    "- **`--dashboard PATH`** - Generate a single PNG dashboard image\n",
    "- **`--dashboard-dir DIR`** - Generate a full HTML dashboard site (ideal for GitHub Pages)\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "Dashboard generation requires optional visualization dependencies:\n",
    "\n",
    "```bash\n",
    "pip install linkml-data-qc[viz]\n",
    "```\n",
    "\n",
    "This installs `matplotlib` and `seaborn` for creating charts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-1",
   "metadata": {
    "papermill": {
     "duration": 0.014315,
     "end_time": "2025-12-08T15:10:42.293562",
     "exception": false,
     "start_time": "2025-12-08T15:10:42.279247",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Setup: Create Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cell-2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:53.206416Z",
     "iopub.status.busy": "2025-12-08T18:50:53.206327Z",
     "iopub.status.idle": "2025-12-08T18:50:53.220893Z",
     "shell.execute_reply": "2025-12-08T18:50:53.220613Z"
    },
    "papermill": {
     "duration": 1.3014,
     "end_time": "2025-12-08T15:10:43.610820",
     "exception": false,
     "start_time": "2025-12-08T15:10:42.309420",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Schema created!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Create a test schema\n",
    "cat > /tmp/dashboard_schema.yaml << 'EOF'\n",
    "id: https://example.org/dashboard-demo\n",
    "name: dashboard_demo\n",
    "prefixes:\n",
    "  linkml: https://w3id.org/linkml/\n",
    "imports:\n",
    "  - linkml:types\n",
    "default_range: string\n",
    "\n",
    "classes:\n",
    "  Dataset:\n",
    "    attributes:\n",
    "      name:\n",
    "        required: true\n",
    "      description:\n",
    "        recommended: true\n",
    "      samples:\n",
    "        multivalued: true\n",
    "        range: Sample\n",
    "\n",
    "  Sample:\n",
    "    attributes:\n",
    "      id:\n",
    "        required: true\n",
    "        identifier: true\n",
    "      name:\n",
    "        recommended: true\n",
    "      tissue_type:\n",
    "        recommended: true\n",
    "      ontology_term:\n",
    "        description: Ontology mapping for this sample\n",
    "        recommended: true\n",
    "      collection_date:\n",
    "        recommended: true\n",
    "EOF\n",
    "echo \"Schema created!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cell-3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:53.222234Z",
     "iopub.status.busy": "2025-12-08T18:50:53.222144Z",
     "iopub.status.idle": "2025-12-08T18:50:53.231662Z",
     "shell.execute_reply": "2025-12-08T18:50:53.231357Z"
    },
    "papermill": {
     "duration": 1.18356,
     "end_time": "2025-12-08T15:10:44.836877",
     "exception": false,
     "start_time": "2025-12-08T15:10:43.653317",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data created!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Create test data with varying compliance\n",
    "cat > /tmp/dashboard_data.yaml << 'EOF'\n",
    "name: Tissue Expression Study\n",
    "description: Analysis of gene expression across tissue types\n",
    "samples:\n",
    "  - id: SAMP001\n",
    "    name: Liver Sample 1\n",
    "    tissue_type: liver\n",
    "    ontology_term: UBERON:0002107\n",
    "    collection_date: \"2024-01-15\"\n",
    "  - id: SAMP002\n",
    "    name: Heart Sample 1\n",
    "    tissue_type: heart\n",
    "    ontology_term: UBERON:0000948\n",
    "    # Missing collection_date\n",
    "  - id: SAMP003\n",
    "    name: Brain Sample 1\n",
    "    tissue_type: brain\n",
    "    # Missing ontology_term and collection_date\n",
    "  - id: SAMP004\n",
    "    # Missing name, tissue_type\n",
    "    ontology_term: UBERON:0000955\n",
    "    collection_date: \"2024-02-20\"\n",
    "  - id: SAMP005\n",
    "    name: Kidney Sample 1\n",
    "    # Missing tissue_type, ontology_term, collection_date\n",
    "  - id: SAMP006\n",
    "    name: Lung Sample 1\n",
    "    tissue_type: lung\n",
    "    ontology_term: UBERON:0002048\n",
    "    collection_date: \"2024-03-01\"\n",
    "  - id: SAMP007\n",
    "    # All recommended fields missing except required id\n",
    "  - id: SAMP008\n",
    "    name: Spleen Sample 1\n",
    "    tissue_type: spleen\n",
    "    ontology_term: UBERON:0002106\n",
    "    collection_date: \"2024-03-15\"\n",
    "EOF\n",
    "echo \"Data created!\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-4",
   "metadata": {
    "papermill": {
     "duration": 0.008133,
     "end_time": "2025-12-08T15:10:44.848456",
     "exception": false,
     "start_time": "2025-12-08T15:10:44.840323",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Basic Dashboard Generation\n",
    "\n",
    "Generate a dashboard with a single command using `--dashboard`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cell-5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:53.233084Z",
     "iopub.status.busy": "2025-12-08T18:50:53.232985Z",
     "iopub.status.idle": "2025-12-08T18:50:54.698834Z",
     "shell.execute_reply": "2025-12-08T18:50:54.698309Z"
    },
    "papermill": {
     "duration": 2.710069,
     "end_time": "2025-12-08T15:10:47.561985",
     "exception": false,
     "start_time": "2025-12-08T15:10:44.851916",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/dashboard_data.yaml\n",
      "Target Class: Dataset\n",
      "Global Compliance: 100.0% (1/1)\n",
      "We"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ighted Compliance: 100.0%\n",
      "\n",
      "Summary by Slot:\n",
      "  description: 100.0%\n",
      "\n",
      "Detailed Path Scores:\n",
      "  (root) (D"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ataset): 100.0%\n",
      "    - description: OK\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/basic_dashboard.png\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\nDashboard saved:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--@ 1 cjm  wheel    79K Dec  8 10:50 /tmp/basic_dashboard.png\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "linkml-data-qc /tmp/dashboard_data.yaml \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    --dashboard /tmp/basic_dashboard.png\n",
    "\n",
    "echo \"\\nDashboard saved:\"\n",
    "ls -lh /tmp/basic_dashboard.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-6",
   "metadata": {
    "papermill": {
     "duration": 0.00143,
     "end_time": "2025-12-08T15:10:47.565178",
     "exception": false,
     "start_time": "2025-12-08T15:10:47.563748",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Dashboard with Configuration\n",
    "\n",
    "Use a configuration file to set weights and thresholds that will be reflected in the dashboard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cell-7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:54.700866Z",
     "iopub.status.busy": "2025-12-08T18:50:54.700717Z",
     "iopub.status.idle": "2025-12-08T18:50:54.712008Z",
     "shell.execute_reply": "2025-12-08T18:50:54.711663Z"
    },
    "papermill": {
     "duration": 0.013483,
     "end_time": "2025-12-08T15:10:47.580074",
     "exception": false,
     "start_time": "2025-12-08T15:10:47.566591",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration created!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Create QC configuration with weights and thresholds\n",
    "cat > /tmp/dashboard_config.yaml << 'EOF'\n",
    "default_weight: 1.0\n",
    "\n",
    "slots:\n",
    "  ontology_term:\n",
    "    weight: 3.0           # Ontology mappings are critical\n",
    "    min_compliance: 80.0  # At least 80% should have mappings\n",
    "  tissue_type:\n",
    "    weight: 2.0           # Tissue type is important\n",
    "    min_compliance: 90.0\n",
    "  name:\n",
    "    weight: 1.5\n",
    "    min_compliance: 95.0\n",
    "  collection_date:\n",
    "    weight: 1.0\n",
    "    min_compliance: 70.0\n",
    "EOF\n",
    "echo \"Configuration created!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cell-8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:54.713523Z",
     "iopub.status.busy": "2025-12-08T18:50:54.713409Z",
     "iopub.status.idle": "2025-12-08T18:50:55.794519Z",
     "shell.execute_reply": "2025-12-08T18:50:55.794176Z"
    },
    "papermill": {
     "duration": 1.15419,
     "end_time": "2025-12-08T15:10:48.735954",
     "exception": false,
     "start_time": "2025-12-08T15:10:47.581764",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/dashboard_data.yaml\n",
      "Target Class: Dataset\n",
      "Global Compliance: 100.0% (1/1)\n",
      "We"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ighted Compliance: 100.0%\n",
      "Config: /tmp/dashboard_config.yaml\n",
      "\n",
      "Summary by Slot:\n",
      "  description: 100.0%"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Detailed Path Scores:\n",
      "  (root) (Dataset): 100.0%\n",
      "    - description: OK\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/configured_dashboard.png\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\nDashboard with thresholds saved:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 cjm  wheel    79K Dec  8 10:50 /tmp/configured_dashboard.png\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Generate dashboard with configuration\n",
    "linkml-data-qc /tmp/dashboard_data.yaml \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    -c /tmp/dashboard_config.yaml \\\n",
    "    --dashboard /tmp/configured_dashboard.png\n",
    "\n",
    "echo \"\\nDashboard with thresholds saved:\"\n",
    "ls -lh /tmp/configured_dashboard.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-9",
   "metadata": {
    "papermill": {
     "duration": 0.001586,
     "end_time": "2025-12-08T15:10:48.739586",
     "exception": false,
     "start_time": "2025-12-08T15:10:48.738000",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Combined Output: Report + Dashboard\n",
    "\n",
    "Generate both a JSON report and a dashboard in one command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cell-10",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:55.796087Z",
     "iopub.status.busy": "2025-12-08T18:50:55.795975Z",
     "iopub.status.idle": "2025-12-08T18:50:56.930939Z",
     "shell.execute_reply": "2025-12-08T18:50:56.930626Z"
    },
    "papermill": {
     "duration": 1.156901,
     "end_time": "2025-12-08T15:10:49.898140",
     "exception": false,
     "start_time": "2025-12-08T15:10:48.741239",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Report written to /tmp/report.json\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/full_dashboard.png\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files generated:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 cjm  wheel    79K Dec  8 10:50 /tmp/full_dashboard.png\n",
      "-rw-r--r--  1 cjm  wheel   887B"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Dec  8 10:50 /tmp/report.json\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "linkml-data-qc /tmp/dashboard_data.yaml \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    -c /tmp/dashboard_config.yaml \\\n",
    "    -f json \\\n",
    "    -o /tmp/report.json \\\n",
    "    --dashboard /tmp/full_dashboard.png\n",
    "\n",
    "echo \"Files generated:\"\n",
    "ls -lh /tmp/report.json /tmp/full_dashboard.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-11",
   "metadata": {
    "papermill": {
     "duration": 0.001672,
     "end_time": "2025-12-08T15:10:49.901806",
     "exception": false,
     "start_time": "2025-12-08T15:10:49.900134",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Comparing Multiple Files\n",
    "\n",
    "When analyzing multiple files, the dashboard shows a comparison view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cell-12",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:56.932493Z",
     "iopub.status.busy": "2025-12-08T18:50:56.932387Z",
     "iopub.status.idle": "2025-12-08T18:50:56.943011Z",
     "shell.execute_reply": "2025-12-08T18:50:56.942758Z"
    },
    "papermill": {
     "duration": 0.013202,
     "end_time": "2025-12-08T15:10:49.917350",
     "exception": false,
     "start_time": "2025-12-08T15:10:49.904148",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Second dataset created!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Create a second dataset with better compliance\n",
    "cat > /tmp/dashboard_data2.yaml << 'EOF'\n",
    "name: Improved Tissue Study\n",
    "description: Updated dataset with better annotation\n",
    "samples:\n",
    "  - id: SAMP101\n",
    "    name: Liver Sample A\n",
    "    tissue_type: liver\n",
    "    ontology_term: UBERON:0002107\n",
    "    collection_date: \"2024-04-01\"\n",
    "  - id: SAMP102\n",
    "    name: Heart Sample A\n",
    "    tissue_type: heart\n",
    "    ontology_term: UBERON:0000948\n",
    "    collection_date: \"2024-04-02\"\n",
    "  - id: SAMP103\n",
    "    name: Brain Sample A\n",
    "    tissue_type: brain\n",
    "    ontology_term: UBERON:0000955\n",
    "    collection_date: \"2024-04-03\"\n",
    "  - id: SAMP104\n",
    "    name: Kidney Sample A\n",
    "    tissue_type: kidney\n",
    "    ontology_term: UBERON:0002113\n",
    "    # Missing collection_date\n",
    "EOF\n",
    "echo \"Second dataset created!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cell-13",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:56.944289Z",
     "iopub.status.busy": "2025-12-08T18:50:56.944207Z",
     "iopub.status.idle": "2025-12-08T18:50:58.082286Z",
     "shell.execute_reply": "2025-12-08T18:50:58.081563Z"
    },
    "papermill": {
     "duration": 1.364419,
     "end_time": "2025-12-08T15:10:51.283729",
     "exception": false,
     "start_time": "2025-12-08T15:10:49.919310",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Multi-File Compliance Report\n",
      "Files Analyzed: 2\n",
      "Global Compliance: 100.0%\n",
      "Weighted Compliance: 100.0%"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Summary by Slot (across all files):\n",
      "  description: 100.0%\n",
      "\n",
      "Summary by Path (across all files):\n",
      "  ("
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root).description: 100.0%\n",
      "\n",
      "Per-File Compliance:\n",
      "  /tmp/dashboard_data.yaml: 100.0%\n",
      "  /tmp/dashboard_"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data2.yaml: 100.0%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/comparison_dashboard.png\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\nComparison dashboard saved:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 cjm  wheel    36K Dec  8 10:50 /tmp/comparison_dashboard.png\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Compare both datasets\n",
    "linkml-data-qc /tmp/dashboard_data.yaml /tmp/dashboard_data2.yaml \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    --dashboard /tmp/comparison_dashboard.png\n",
    "\n",
    "echo \"\\nComparison dashboard saved:\"\n",
    "ls -lh /tmp/comparison_dashboard.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-14",
   "metadata": {
    "papermill": {
     "duration": 0.002663,
     "end_time": "2025-12-08T15:10:51.288977",
     "exception": false,
     "start_time": "2025-12-08T15:10:51.286314",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## CI/CD Integration\n",
    "\n",
    "Combine dashboard generation with CI/CD checks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cell-15",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:58.083730Z",
     "iopub.status.busy": "2025-12-08T18:50:58.083625Z",
     "iopub.status.idle": "2025-12-08T18:50:59.145561Z",
     "shell.execute_reply": "2025-12-08T18:50:59.145248Z"
    },
    "papermill": {
     "duration": 1.734333,
     "end_time": "2025-12-08T15:10:53.025400",
     "exception": false,
     "start_time": "2025-12-08T15:10:51.291067",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/dashboard_data2.yaml\n",
      "Target Class: Dataset\n",
      "Global Compliance: 100.0% (1/1)\n",
      "W"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eighted Compliance: 100.0%\n",
      "\n",
      "Summary by Slot:\n",
      "  description: 100.0%\n",
      "\n",
      "Detailed Path Scores:\n",
      "  (root) ("
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset): 100.0%\n",
      "    - description: OK\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/ci_dashboard.png\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\nCI check passed! Dashboard saved.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Generate dashboard AND fail if below threshold\n",
    "# (using the good dataset so this won't fail)\n",
    "linkml-data-qc /tmp/dashboard_data2.yaml \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    --min-compliance 70 \\\n",
    "    --dashboard /tmp/ci_dashboard.png\n",
    "\n",
    "echo \"\\nCI check passed! Dashboard saved.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cell-16",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:50:59.147090Z",
     "iopub.status.busy": "2025-12-08T18:50:59.146981Z",
     "iopub.status.idle": "2025-12-08T18:51:00.154306Z",
     "shell.execute_reply": "2025-12-08T18:51:00.153944Z"
    },
    "papermill": {
     "duration": 1.997012,
     "end_time": "2025-12-08T15:10:55.026045",
     "exception": false,
     "start_time": "2025-12-08T15:10:53.029033",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/dashboard_data.yaml\n",
      "Target Class: Dataset\n",
      "Global Compliance: 100.0% (1/1)\n",
      "We"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ighted Compliance: 100.0%\n",
      "Config: /tmp/dashboard_config.yaml\n",
      "\n",
      "Summary by Slot:\n",
      "  description: 100.0%"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Detailed Path Scores:\n",
      "  (root) (Dataset): 100.0%\n",
      "    - description: OK\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/failed_ci_dashboard.png\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# This would fail CI due to threshold violations\n",
    "# (but we capture the exit code to show it)\n",
    "linkml-data-qc /tmp/dashboard_data.yaml \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    -c /tmp/dashboard_config.yaml \\\n",
    "    --fail-on-violations \\\n",
    "    --dashboard /tmp/failed_ci_dashboard.png || echo \"\\nCI check failed (as expected due to violations)\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-17",
   "metadata": {
    "papermill": {
     "duration": 0.002093,
     "end_time": "2025-12-08T15:10:55.030672",
     "exception": false,
     "start_time": "2025-12-08T15:10:55.028579",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Directory Analysis with Dashboard\n",
    "\n",
    "Analyze all files in a directory and generate a summary dashboard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cell-18",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:51:00.155866Z",
     "iopub.status.busy": "2025-12-08T18:51:00.155765Z",
     "iopub.status.idle": "2025-12-08T18:51:01.160750Z",
     "shell.execute_reply": "2025-12-08T18:51:01.160386Z"
    },
    "papermill": {
     "duration": 1.216902,
     "end_time": "2025-12-08T15:10:56.249735",
     "exception": false,
     "start_time": "2025-12-08T15:10:55.032833",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Multi-File Compliance Report\n",
      "Files Analyzed: 2\n",
      "Global Compliance: 100.0%\n",
      "Weighted Compliance: 100.0%"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Summary by Slot (across all files):\n",
      "  description: 100.0%\n",
      "\n",
      "Summary by Path (across all files):\n",
      "  ("
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root).description: 100.0%\n",
      "\n",
      "Per-File Compliance:\n",
      "  /tmp/datasets/study1.yaml: 100.0%\n",
      "  /tmp/datasets/"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "study2.yaml: 100.0%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Dashboard saved to /tmp/directory_dashboard.png\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\nDirectory analysis dashboard saved:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 cjm  wheel    33K Dec  8 10:51 /tmp/directory_dashboard.png\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Create a directory with multiple data files\n",
    "mkdir -p /tmp/datasets\n",
    "cp /tmp/dashboard_data.yaml /tmp/datasets/study1.yaml\n",
    "cp /tmp/dashboard_data2.yaml /tmp/datasets/study2.yaml\n",
    "\n",
    "# Analyze directory\n",
    "linkml-data-qc /tmp/datasets/ \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    --pattern \"*.yaml\" \\\n",
    "    --dashboard /tmp/directory_dashboard.png\n",
    "\n",
    "echo \"\\nDirectory analysis dashboard saved:\"\n",
    "ls -lh /tmp/directory_dashboard.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mzy4ze4jj9",
   "metadata": {},
   "source": [
    "## HTML Dashboard Site (GitHub Pages)\n",
    "\n",
    "For multi-file analysis, use `--dashboard-dir` to generate a complete HTML dashboard site:\n",
    "\n",
    "- **Sortable file list** - Files sorted by compliance (worst first) to prioritize curation\n",
    "- **Priority badges** - Top 10 worst files highlighted with priority numbers\n",
    "- **Detailed charts** - Individual slot breakdown for priority files\n",
    "- **JSON export** - Machine-readable `reports.json` for automation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0nhfzk3tcp7l",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:51:01.162388Z",
     "iopub.status.busy": "2025-12-08T18:51:01.162277Z",
     "iopub.status.idle": "2025-12-08T18:51:02.296990Z",
     "shell.execute_reply": "2025-12-08T18:51:02.296720Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Multi-File Compliance Report\n",
      "Files Analyzed: 2\n",
      "Global Compliance: 100.0%\n",
      "Weighted Compliance: 100.0%"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Summary by Slot (across all files):\n",
      "  description: 100.0%\n",
      "\n",
      "Summary by Path (across all files):\n",
      "  ("
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root).description: 100.0%\n",
      "\n",
      "Per-File Compliance:\n",
      "  /tmp/datasets/study1.yaml: 100.0%\n",
      "  /tmp/datasets/"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "study2.yaml: 100.0%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "HTML dashboard generated at /tmp/html_dashboard/index.html\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\nHTML dashboard generated:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 208\n",
      "drwxr-xr-x    7 cjm   wheel    224 Dec  8 10:51 \u001b[1m\u001b[36m.\u001b[m\u001b[m\n",
      "drwxrwxrwt  172 root  whe"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "el   5504 Dec  8 10:51 \u001b[30m\u001b[42m..\u001b[m\u001b[m\n",
      "-rw-r--r--    1 cjm   wheel  33654 Dec  8 10:51 comparison"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".png\n",
      "-rw-r--r--    1 cjm   wheel  27637 Dec  8 10:51 detail_0.png\n",
      "-rw-r--r--    1 cjm   wheel  28008"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Dec  8 10:51 detail_1.png\n",
      "-rw-r--r--    1 cjm   wheel   6973 Dec  8 10:51 index.html\n",
      "-rw-r--r--    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 cjm   wheel    344 Dec  8 10:51 reports.json\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Generate HTML dashboard for directory\n",
    "linkml-data-qc /tmp/datasets/ \\\n",
    "    -s /tmp/dashboard_schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    --pattern \"*.yaml\" \\\n",
    "    --dashboard-dir /tmp/html_dashboard\n",
    "\n",
    "echo \"\\nHTML dashboard generated:\"\n",
    "ls -la /tmp/html_dashboard/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "sap3ol4k1zm",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:51:02.298246Z",
     "iopub.status.busy": "2025-12-08T18:51:02.298155Z",
     "iopub.status.idle": "2025-12-08T18:51:02.310007Z",
     "shell.execute_reply": "2025-12-08T18:51:02.309700Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  {\n",
      "    \"file\": \"study1.yaml\",\n",
      "    \"global_compliance\": 100.0,\n",
      "    \"weighted_compliance\": 100.0,\n",
      " "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   \"total_checks\": 1,\n",
      "    \"total_populated\": 1,\n",
      "    \"violations\": 0\n",
      "  },\n",
      "  {\n",
      "    \"file\": \"study2.yam"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "l\",\n",
      "    \"global_compliance\": 100.0,\n",
      "    \"weighted_compliance\": 100.0,\n",
      "    \"total_checks\": 1,\n",
      "    \"to"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tal_populated\": 1,\n",
      "    \"violations\": 0\n",
      "  }\n",
      "]"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# View the JSON report for automation\n",
    "cat /tmp/html_dashboard/reports.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ir8be4v9j2f",
   "metadata": {},
   "source": [
    "The HTML dashboard can be directly deployed to GitHub Pages. The `reports.json` file contains all file metrics sorted by compliance, useful for:\n",
    "\n",
    "- CI/CD automation scripts\n",
    "- Custom downstream analysis\n",
    "- Integration with other reporting tools"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-19",
   "metadata": {
    "papermill": {
     "duration": 0.003938,
     "end_time": "2025-12-08T15:10:56.256516",
     "exception": false,
     "start_time": "2025-12-08T15:10:56.252578",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Summary of Generated Dashboards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cell-20",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-08T18:51:02.311368Z",
     "iopub.status.busy": "2025-12-08T18:51:02.311278Z",
     "iopub.status.idle": "2025-12-08T18:51:02.325333Z",
     "shell.execute_reply": "2025-12-08T18:51:02.325048Z"
    },
    "papermill": {
     "duration": 0.016446,
     "end_time": "2025-12-08T15:10:56.276165",
     "exception": false,
     "start_time": "2025-12-08T15:10:56.259719",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All generated dashboards:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--@ 1 cjm  wheel    79K Dec  8 10:50 /tmp/basic_dashboard.png\n",
      "-rw-r--r--  1 cjm  wheel    79"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K Dec  8 10:50 /tmp/ci_dashboard.png\n",
      "-rw-r--r--  1 cjm  wheel    86K Dec  7 18:45 /tmp/cli_dashboard"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".png\n",
      "-rw-r--r--  1 cjm  wheel    36K Dec  8 10:50 /tmp/comparison_dashboard.png\n",
      "-rw-r--r--  1 cjm  w"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "heel    79K Dec  8 10:50 /tmp/configured_dashboard.png\n",
      "-rw-r--r--  1 cjm  wheel    33K Dec  8 10:51 "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/directory_dashboard.png\n",
      "-rw-r--r--  1 cjm  wheel    79K Dec  8 10:51 /tmp/failed_ci_dashboard.p"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ng\n",
      "-rw-r--r--  1 cjm  wheel    79K Dec  8 10:50 /tmp/full_dashboard.png\n",
      "-rw-r--r--  1 cjm  wheel    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79K Dec  7 12:03 /tmp/qc_dashboard.png\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo \"All generated dashboards:\"\n",
    "ls -lh /tmp/*dashboard*.png 2>/dev/null || echo \"No dashboards found\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-21",
   "metadata": {
    "papermill": {
     "duration": 0.002595,
     "end_time": "2025-12-08T15:10:56.281589",
     "exception": false,
     "start_time": "2025-12-08T15:10:56.278994",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Key CLI Options for Dashboards\n",
    "\n",
    "| Option | Description |\n",
    "|--------|-------------|\n",
    "| `--dashboard PATH` | Generate single dashboard PNG image |\n",
    "| `--dashboard-dir DIR` | Generate HTML dashboard site (for GitHub Pages) |\n",
    "| `-c, --config PATH` | Config file with weights/thresholds (reflected in dashboard) |\n",
    "| `-o, --output PATH` | Also save text/JSON/CSV report |\n",
    "| `--min-compliance N` | Fail if compliance < N% (dashboard still generated) |\n",
    "| `--fail-on-violations` | Fail if thresholds violated (dashboard still generated) |\n",
    "\n",
    "## HTML Dashboard Structure\n",
    "\n",
    "When using `--dashboard-dir`, the following files are generated:\n",
    "\n",
    "```\n",
    "dashboard/\n",
    "├── index.html       # Main dashboard page\n",
    "├── comparison.png   # Slot compliance across files\n",
    "├── detail_0.png     # Detailed chart for worst file\n",
    "├── detail_1.png     # Detailed chart for 2nd worst file\n",
    "├── ...              # Up to 10 detail charts\n",
    "└── reports.json     # All metrics for automation\n",
    "```\n",
    "\n",
    "Files are sorted by compliance (worst first) to help prioritize curation efforts.\n",
    "\n",
    "## CI/CD Example with HTML Dashboard\n",
    "\n",
    "```bash\n",
    "# GitHub Actions workflow\n",
    "linkml-data-qc data/ \\\n",
    "    -s schema.yaml \\\n",
    "    -t Dataset \\\n",
    "    -c qc_config.yaml \\\n",
    "    --dashboard-dir artifacts/qc_dashboard \\\n",
    "    --fail-on-violations\n",
    "\n",
    "# Deploy artifacts/qc_dashboard to GitHub Pages\n",
    "```\n",
    "\n",
    "This generates a complete dashboard site as a CI artifact, then fails if thresholds are violated."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 15.194605,
   "end_time": "2025-12-08T15:10:56.400700",
   "environment_variables": {},
   "exception": null,
   "input_path": "docs/notebooks/06_dashboards.ipynb",
   "output_path": "docs/notebooks/06_dashboards.ipynb",
   "parameters": {},
   "start_time": "2025-12-08T15:10:41.206095",
   "version": "2.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
