{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df7d19b1",
   "metadata": {
    "papermill": {
     "duration": 0.001996,
     "end_time": "2025-12-07T04:52:04.247173",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.245177",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# QC for Tabular Data (CSV/TSV)\n",
    "\n",
    "This tutorial shows how to use linkml-data-qc with CSV and TSV files - common formats for tabular datasets.\n",
    "\n",
    "## When to Use Tabular Formats\n",
    "\n",
    "CSV/TSV is great for:\n",
    "- Flat datasets where each row is an instance\n",
    "- Data exported from spreadsheets or databases\n",
    "- Large datasets with many similar records\n",
    "\n",
    "## How It Works\n",
    "\n",
    "- Each row in the CSV/TSV becomes an instance of your target class\n",
    "- Column headers map to slot names\n",
    "- Empty cells are treated as missing (not populated)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de804726",
   "metadata": {
    "papermill": {
     "duration": 0.001171,
     "end_time": "2025-12-07T04:52:04.249885",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.248714",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Example: Gene Annotation Dataset\n",
    "\n",
    "Let's create a schema for gene annotations and a TSV dataset with varying completeness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "22d3923a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:04.252805Z",
     "iopub.status.busy": "2025-12-07T04:52:04.252671Z",
     "iopub.status.idle": "2025-12-07T04:52:04.267387Z",
     "shell.execute_reply": "2025-12-07T04:52:04.266838Z"
    },
    "papermill": {
     "duration": 0.017323,
     "end_time": "2025-12-07T04:52:04.268331",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.251008",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gene schema created!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat > /tmp/gene_schema.yaml << 'EOF'\n",
    "id: https://example.org/genes\n",
    "name: gene_schema\n",
    "prefixes:\n",
    "  linkml: https://w3id.org/linkml/\n",
    "imports:\n",
    "  - linkml:types\n",
    "default_range: string\n",
    "\n",
    "classes:\n",
    "  Gene:\n",
    "    attributes:\n",
    "      gene_id:\n",
    "        identifier: true\n",
    "        required: true\n",
    "      symbol:\n",
    "        required: true\n",
    "      name:\n",
    "        recommended: true\n",
    "      description:\n",
    "        recommended: true\n",
    "      chromosome:\n",
    "        recommended: true\n",
    "      go_terms:\n",
    "        recommended: true\n",
    "      publications:\n",
    "        recommended: true\n",
    "EOF\n",
    "echo \"Gene schema created!\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61976510",
   "metadata": {
    "papermill": {
     "duration": 0.001227,
     "end_time": "2025-12-07T04:52:04.271085",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.269858",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Create a TSV Dataset\n",
    "\n",
    "Here's a gene annotation file with some missing recommended fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "90343448",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:04.273985Z",
     "iopub.status.busy": "2025-12-07T04:52:04.273847Z",
     "iopub.status.idle": "2025-12-07T04:52:04.289610Z",
     "shell.execute_reply": "2025-12-07T04:52:04.289326Z"
    },
    "papermill": {
     "duration": 0.018104,
     "end_time": "2025-12-07T04:52:04.290283",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.272179",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gene TSV created! Contents:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gene_id\tsymbol\tname\tdescription\tchromosome\tgo_terms\tpublications\n",
      "ENSG00000139618\tBRCA2\tBRCA2 DNA rep"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "air associated\tInvolved in DNA repair\t13\tGO:0006281,GO:0006310\tPMID:8589730\n",
      "ENSG00000141510\tTP53\tTum"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "or protein p53\tTumor suppressor gene\t17\tGO:0006915,GO:0008283\tPMID:2052564\n",
      "ENSG00000157764\tBRAF\tB-Ra"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f proto-oncogene\t\t7\tGO:0006468\t\n",
      "ENSG00000133703\tKRAS\tKRAS proto-oncogene\tGTPase signaling\t12\t\tPMID:2"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "278970\n",
      "ENSG00000171862\tPTEN\tPhosphatase and tensin homolog\t\t10\t\t\n",
      "ENSG00000146648\tEGFR\t\t\t7\tGO:0007169"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tPMID:3490687\n",
      "ENSG00000181143\tMYC\tMYC proto-oncogene\tTranscription factor\t8\tGO:0006355\tPMID:7479893\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat > /tmp/genes.tsv << 'EOF'\n",
    "gene_id\tsymbol\tname\tdescription\tchromosome\tgo_terms\tpublications\n",
    "ENSG00000139618\tBRCA2\tBRCA2 DNA repair associated\tInvolved in DNA repair\t13\tGO:0006281,GO:0006310\tPMID:8589730\n",
    "ENSG00000141510\tTP53\tTumor protein p53\tTumor suppressor gene\t17\tGO:0006915,GO:0008283\tPMID:2052564\n",
    "ENSG00000157764\tBRAF\tB-Raf proto-oncogene\t\t7\tGO:0006468\t\n",
    "ENSG00000133703\tKRAS\tKRAS proto-oncogene\tGTPase signaling\t12\t\tPMID:2278970\n",
    "ENSG00000171862\tPTEN\tPhosphatase and tensin homolog\t\t10\t\t\n",
    "ENSG00000146648\tEGFR\t\t\t7\tGO:0007169\tPMID:3490687\n",
    "ENSG00000181143\tMYC\tMYC proto-oncogene\tTranscription factor\t8\tGO:0006355\tPMID:7479893\n",
    "EOF\n",
    "echo \"Gene TSV created! Contents:\"\n",
    "cat /tmp/genes.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31adf492",
   "metadata": {
    "papermill": {
     "duration": 0.001229,
     "end_time": "2025-12-07T04:52:04.292993",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.291764",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Analyze the TSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1026d9c9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:04.296308Z",
     "iopub.status.busy": "2025-12-07T04:52:04.296169Z",
     "iopub.status.idle": "2025-12-07T04:52:04.787082Z",
     "shell.execute_reply": "2025-12-07T04:52:04.786692Z"
    },
    "papermill": {
     "duration": 0.493829,
     "end_time": "2025-12-07T04:52:04.788049",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.294220",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/genes.tsv\n",
      "Target Class: Gene\n",
      "Global Compliance: 77.1% (27/35)\n",
      "Weighted Compl"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iance: 77.1%\n",
      "\n",
      "Summary by Slot:\n",
      "  chromosome: 100.0%\n",
      "  description: 57.1%\n",
      "  go_terms: 71.4%\n",
      "  name: 8"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.7%\n",
      "  publications: 71.4%\n",
      "\n",
      "Aggregated Scores by List Path:\n",
      "  [].chromosome: 100.0% (7/7)\n",
      "  [].descr"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iption: 57.1% (4/7)\n",
      "  [].go_terms: 71.4% (5/7)\n",
      "  [].name: 85.7% (6/7)\n",
      "  [].publications: 71.4% (5/7)"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Detailed Path Scores:\n",
      "  [0] (Gene): 100.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome: "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [1] (Gene): 100.0%\n",
      "    - name: OK\n",
      "    - description: "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [2] (Gene): 60.0%\n",
      "    - name: OK"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    - description: MISSING\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: MISSING\n",
      "  [3"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "] (Gene): 80.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: MISSING\n",
      "  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  - publications: OK\n",
      "  [4] (Gene): 40.0%\n",
      "    - name: OK\n",
      "    - description: MISSING\n",
      "    - chromosome:"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " OK\n",
      "    - go_terms: MISSING\n",
      "    - publications: MISSING\n",
      "  [5] (Gene): 60.0%\n",
      "    - name: MISSING\n",
      "    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- description: MISSING\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [6] (Gene): "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publicatio"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ns: OK\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "linkml-data-qc /tmp/genes.tsv \\\n",
    "    -s /tmp/gene_schema.yaml \\\n",
    "    -t Gene"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3622eb9b",
   "metadata": {
    "papermill": {
     "duration": 0.001578,
     "end_time": "2025-12-07T04:52:04.791529",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.789951",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Understanding Tabular Paths\n",
    "\n",
    "For CSV/TSV data, each row gets an indexed path:\n",
    "- `[0]` - First row (BRCA2)\n",
    "- `[1]` - Second row (TP53)\n",
    "- etc.\n",
    "\n",
    "The aggregated path `[]` shows compliance across all rows.\n",
    "\n",
    "Let's see the JSON output for details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2066d198",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:04.795585Z",
     "iopub.status.busy": "2025-12-07T04:52:04.795449Z",
     "iopub.status.idle": "2025-12-07T04:52:05.109292Z",
     "shell.execute_reply": "2025-12-07T04:52:05.108949Z"
    },
    "papermill": {
     "duration": 0.318361,
     "end_time": "2025-12-07T04:52:05.111579",
     "exception": false,
     "start_time": "2025-12-07T04:52:04.793218",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"file_path\": \"/tmp/genes.tsv\",\n",
      "    \"target_class\": \"Gene\",\n",
      "    \"schema_path\": \"/tmp/gene_schem"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a.yaml\",\n",
      "    \"global_compliance\": 77.14285714285715,\n",
      "    \"weighted_compliance\": 77.14285714285715,\n",
      " "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   \"total_checks\": 35,\n",
      "    \"total_populated\": 27,\n",
      "    \"path_scores\": [\n",
      "        {\n",
      "            \"path\":"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \"[0]\",\n",
      "            \"parent_class\": \"Gene\",\n",
      "            \"item_count\": 1,\n",
      "            \"slot_scores\": "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "                {\n",
      "                    \"path\": \"[0]\",\n",
      "                    \"slot_name\": \"name\",\n",
      "    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                \"populated\": 1,\n",
      "                    \"total\": 1,\n",
      "                    \"percentage\": 10"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "                },\n",
      "                {\n",
      "                    \"path\": \"[0]\",\n",
      "                    \"slo"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t_name\": \"description\",\n",
      "                    \"populated\": 1,\n",
      "                    \"total\": 1,\n",
      "        "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            \"percentage\": 100.0\n",
      "                },\n",
      "                {\n",
      "                    \"path\": \"[0"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "]\",\n",
      "                    \"slot_name\": \"chromosome\",\n",
      "                    \"populated\": 1,\n",
      "             "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       \"total\": 1,\n",
      "                    \"percentage\": 100.0\n",
      "                },\n",
      "                {\n",
      "    "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                \"path\": \"[0]\",\n",
      "                    \"slot_name\": \"go_terms\",\n",
      "                    \"pop"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ulated\": 1,\n",
      "                    \"total\": 1,\n",
      "                    \"percentage\": 100.0\n",
      "                "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "},\n",
      "                {\n",
      "                    \"path\": \"[0]\",\n",
      "                    \"slot_name\": \"publicatio"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ns\",\n",
      "                    \"populated\": 1,\n",
      "                    \"total\": 1,\n",
      "                    \"percen"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tage\": 100.0\n",
      "                }\n",
      "            ],\n",
      "            \"overall_percentage\": 100.0\n",
      "        },\n",
      "   "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     {\n",
      "            \"path\": \"[1]\",\n",
      "            \"parent_class\": \"Gene\",\n",
      "            \"item_count\": 1,\n",
      " "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           \"slot_scores\": [\n",
      "                {\n",
      "                    \"path\": \"[1]\",\n",
      "                   "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \"slot_name\": \"name\",\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "linkml-data-qc /tmp/genes.tsv \\\n",
    "    -s /tmp/gene_schema.yaml \\\n",
    "    -t Gene \\\n",
    "    -f json | python -m json.tool | head -60"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8a5e1f1",
   "metadata": {
    "papermill": {
     "duration": 0.002044,
     "end_time": "2025-12-07T04:52:05.115928",
     "exception": false,
     "start_time": "2025-12-07T04:52:05.113884",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Example: CSV Format\n",
    "\n",
    "CSV works the same way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bd35f5da",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:05.120766Z",
     "iopub.status.busy": "2025-12-07T04:52:05.120617Z",
     "iopub.status.idle": "2025-12-07T04:52:05.433803Z",
     "shell.execute_reply": "2025-12-07T04:52:05.433479Z"
    },
    "papermill": {
     "duration": 0.316966,
     "end_time": "2025-12-07T04:52:05.434954",
     "exception": false,
     "start_time": "2025-12-07T04:52:05.117988",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/genes.csv\n",
      "Target Class: Gene\n",
      "Global Compliance: 85.0% (17/20)\n",
      "Weighted Compl"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iance: 85.0%\n",
      "\n",
      "Summary by Slot:\n",
      "  chromosome: 100.0%\n",
      "  description: 75.0%\n",
      "  go_terms: 75.0%\n",
      "  name: 1"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00.0%\n",
      "  publications: 75.0%\n",
      "\n",
      "Aggregated Scores by List Path:\n",
      "  [].chromosome: 100.0% (4/4)\n",
      "  [].desc"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ription: 75.0% (3/4)\n",
      "  [].go_terms: 75.0% (3/4)\n",
      "  [].name: 100.0% (4/4)\n",
      "  [].publications: 75.0% (3/"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4)\n",
      "\n",
      "Detailed Path Scores:\n",
      "  [0] (Gene): 100.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ": OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [1] (Gene): 100.0%\n",
      "    - name: OK\n",
      "    - description"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ": OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [2] (Gene): 60.0%\n",
      "    - name: "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OK\n",
      "    - description: MISSING\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: MISSING\n",
      "  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3] (Gene): 80.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: MISSING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    - publications: OK\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat > /tmp/genes.csv << 'EOF'\n",
    "gene_id,symbol,name,description,chromosome,go_terms,publications\n",
    "ENSG00000139618,BRCA2,BRCA2 DNA repair associated,Involved in DNA repair,13,\"GO:0006281,GO:0006310\",PMID:8589730\n",
    "ENSG00000141510,TP53,Tumor protein p53,Tumor suppressor gene,17,\"GO:0006915,GO:0008283\",PMID:2052564\n",
    "ENSG00000157764,BRAF,B-Raf proto-oncogene,,7,GO:0006468,\n",
    "ENSG00000133703,KRAS,KRAS proto-oncogene,GTPase signaling,12,,PMID:2278970\n",
    "EOF\n",
    "\n",
    "linkml-data-qc /tmp/genes.csv \\\n",
    "    -s /tmp/gene_schema.yaml \\\n",
    "    -t Gene"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d7eeff0",
   "metadata": {
    "papermill": {
     "duration": 0.002147,
     "end_time": "2025-12-07T04:52:05.439552",
     "exception": false,
     "start_time": "2025-12-07T04:52:05.437405",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Setting Thresholds for Tabular Data\n",
    "\n",
    "For large datasets, you might want to ensure a minimum percentage of records have key fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cf988e0b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:05.444393Z",
     "iopub.status.busy": "2025-12-07T04:52:05.444263Z",
     "iopub.status.idle": "2025-12-07T04:52:05.743747Z",
     "shell.execute_reply": "2025-12-07T04:52:05.743354Z"
    },
    "papermill": {
     "duration": 0.303221,
     "end_time": "2025-12-07T04:52:05.744850",
     "exception": false,
     "start_time": "2025-12-07T04:52:05.441629",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compliance Report: /tmp/genes.tsv\n",
      "Target Class: Gene\n",
      "Global Compliance: 77.1% (27/35)\n",
      "Weighted Compl"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iance: 78.0%\n",
      "Config: /tmp/gene_qc_config.yaml\n",
      "\n",
      "Summary by Slot:\n",
      "  chromosome: 100.0%\n",
      "  description: "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "57.1%\n",
      "  go_terms: 71.4%\n",
      "  name: 85.7%\n",
      "  publications: 71.4%\n",
      "\n",
      "Threshold Violations (1):\n",
      "  [].name: 85"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".7% < 90.0% (shortfall: 4.3%)\n",
      "\n",
      "Aggregated Scores by List Path:\n",
      "  [].chromosome: 100.0% (7/7)\n",
      "  [].de"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scription: 57.1% (4/7)\n",
      "  [].go_terms: 71.4% (5/7)\n",
      "  [].name: 85.7% (6/7)\n",
      "  [].publications: 71.4% (5"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/7)\n",
      "\n",
      "Detailed Path Scores:\n",
      "  [0] (Gene): 100.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosom"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e: OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [1] (Gene): 100.0%\n",
      "    - name: OK\n",
      "    - descriptio"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n: OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [2] (Gene): 60.0%\n",
      "    - name:"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " OK\n",
      "    - description: MISSING\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: MISSING\n",
      " "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [3] (Gene): 80.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: MISSING"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    - publications: OK\n",
      "  [4] (Gene): 40.0%\n",
      "    - name: OK\n",
      "    - description: MISSING\n",
      "    - chromoso"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "me: OK\n",
      "    - go_terms: MISSING\n",
      "    - publications: MISSING\n",
      "  [5] (Gene): 60.0%\n",
      "    - name: MISSING\n",
      " "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   - description: MISSING\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publications: OK\n",
      "  [6] (Gene"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "): 100.0%\n",
      "    - name: OK\n",
      "    - description: OK\n",
      "    - chromosome: OK\n",
      "    - go_terms: OK\n",
      "    - publica"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tions: OK\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat > /tmp/gene_qc_config.yaml << 'EOF'\n",
    "default_weight: 1.0\n",
    "\n",
    "slots:\n",
    "  name:\n",
    "    weight: 2.0\n",
    "    min_compliance: 90.0  # At least 90% of genes should have names\n",
    "  go_terms:\n",
    "    weight: 1.5\n",
    "    min_compliance: 70.0  # At least 70% should have GO annotations\n",
    "  publications:\n",
    "    weight: 1.0\n",
    "    min_compliance: 60.0  # At least 60% should have publications\n",
    "EOF\n",
    "\n",
    "linkml-data-qc /tmp/genes.tsv \\\n",
    "    -s /tmp/gene_schema.yaml \\\n",
    "    -t Gene \\\n",
    "    -c /tmp/gene_qc_config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d0a72e",
   "metadata": {
    "papermill": {
     "duration": 0.002576,
     "end_time": "2025-12-07T04:52:05.750370",
     "exception": false,
     "start_time": "2025-12-07T04:52:05.747794",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Analyzing a Directory of TSV Files\n",
    "\n",
    "If you have multiple TSV files, analyze them together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "05fcd1c7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:05.756864Z",
     "iopub.status.busy": "2025-12-07T04:52:05.756703Z",
     "iopub.status.idle": "2025-12-07T04:52:06.056987Z",
     "shell.execute_reply": "2025-12-07T04:52:06.056044Z"
    },
    "papermill": {
     "duration": 0.306021,
     "end_time": "2025-12-07T04:52:06.059150",
     "exception": false,
     "start_time": "2025-12-07T04:52:05.753129",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Multi-File Compliance Report\n",
      "Files Analyzed: 2\n",
      "Global Compliance: 82.2%\n",
      "Weighted Compliance: 82.2%\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary by Slot (across all files):\n",
      "  chromosome: 100.0%\n",
      "  description: 66.7%\n",
      "  go_terms: 77.8%\n",
      "  na"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "me: 88.9%\n",
      "  publications: 77.8%\n",
      "\n",
      "Summary by Path (across all files):\n",
      "  [].chromosome: 100.0%\n",
      "  [].de"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scription: 66.7%\n",
      "  [].go_terms: 77.8%\n",
      "  [].name: 88.9%\n",
      "  [].publications: 77.8%\n",
      "\n",
      "Per-File Compliance"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":\n",
      "  /tmp/genes.tsv: 77.1%\n",
      "  /tmp/genes2.tsv: 100.0%\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Create a second TSV file\n",
    "cat > /tmp/genes2.tsv << 'EOF'\n",
    "gene_id\tsymbol\tname\tdescription\tchromosome\tgo_terms\tpublications\n",
    "ENSG00000105329\tTGFB1\tTransforming growth factor beta 1\tCytokine signaling\t19\tGO:0008083\tPMID:3016893\n",
    "ENSG00000127479\tVEGFA\tVascular endothelial growth factor A\tAngiogenesis regulator\t6\tGO:0001525\tPMID:2587564\n",
    "EOF\n",
    "\n",
    "linkml-data-qc /tmp \\\n",
    "    -s /tmp/gene_schema.yaml \\\n",
    "    -t Gene \\\n",
    "    --pattern \"genes*.tsv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df237ea9",
   "metadata": {
    "papermill": {
     "duration": 0.018522,
     "end_time": "2025-12-07T04:52:06.084699",
     "exception": false,
     "start_time": "2025-12-07T04:52:06.066177",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Finding Incomplete Records\n",
    "\n",
    "Use CSV output to identify which records need attention:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "92663750",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-12-07T04:52:06.130325Z",
     "iopub.status.busy": "2025-12-07T04:52:06.129866Z",
     "iopub.status.idle": "2025-12-07T04:52:06.738120Z",
     "shell.execute_reply": "2025-12-07T04:52:06.737801Z"
    },
    "papermill": {
     "duration": 0.624753,
     "end_time": "2025-12-07T04:52:06.738883",
     "exception": false,
     "start_time": "2025-12-07T04:52:06.114130",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Records with lowest compliance ===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "/tmp/genes.tsv,[2],Gene,description,0,1,0.0\r\n",
      "/tmp/genes.tsv,[2],Gene,publications,0,1,0.0\r\n",
      "/tmp/gen"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "es.tsv,[3],Gene,go_terms,0,1,0.0\r\n",
      "/tmp/genes.tsv,[4],Gene,description,0,1,0.0\r\n",
      "/tmp/genes.tsv,[4],Ge"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ne,go_terms,0,1,0.0\r\n",
      "/tmp/genes.tsv,[4],Gene,publications,0,1,0.0\r\n",
      "/tmp/genes.tsv,[5],Gene,descripti"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "on,0,1,0.0\r\n",
      "/tmp/genes.tsv,[5],Gene,name,0,1,0.0\r\n",
      "file,path,class,slot,populated,total,percentage\r\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo \"=== Records with lowest compliance ===\"\n",
    "linkml-data-qc /tmp/genes.tsv \\\n",
    "    -s /tmp/gene_schema.yaml \\\n",
    "    -t Gene \\\n",
    "    -f csv | grep -v \"\\[\\]\" | sort -t, -k7 -n | head -10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36145993",
   "metadata": {
    "papermill": {
     "duration": 0.003573,
     "end_time": "2025-12-07T04:52:06.745479",
     "exception": false,
     "start_time": "2025-12-07T04:52:06.741906",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## CI/CD Integration for Tabular Data\n",
    "\n",
    "Ensure your data meets quality standards before releases:\n",
    "\n",
    "```bash\n",
    "# Fail if less than 80% of genes have names\n",
    "linkml-data-qc data/genes.tsv \\\n",
    "    -s schema/gene.yaml \\\n",
    "    -t Gene \\\n",
    "    --min-compliance 80\n",
    "```\n",
    "\n",
    "## Key Takeaways\n",
    "\n",
    "1. **File extension matters** - Use `.csv`, `.tsv`, or `.tab` for tabular files\n",
    "2. **Each row = one instance** - Compliance is calculated per row, then aggregated\n",
    "3. **Empty cells = missing** - Empty strings in CSV/TSV are treated as unpopulated\n",
    "4. **Use patterns** - Analyze multiple files with `--pattern \"*.tsv\"`\n",
    "5. **Same config works** - Weight and threshold configs work identically for tabular data"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 3.603731,
   "end_time": "2025-12-07T04:52:06.870626",
   "environment_variables": {},
   "exception": null,
   "input_path": "docs/notebooks/04_tabular_data.ipynb",
   "output_path": "docs/notebooks/04_tabular_data.ipynb",
   "parameters": {},
   "start_time": "2025-12-07T04:52:03.266895",
   "version": "2.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}