{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "# How to generate the `genemap.txt` file\n",
    "\n",
    "Using UCSC refGene for gene definition, Rutgers Map for genetic distances, and linear interpolation for those that cannot be found in the database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "## Gene range file\n",
    "\n",
    "Downloaded [`refGene.txt.gz` from UCSC](http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/refGene.txt.gz), currently version March 01, 2020. Gene range file generated using [this workflow](https://gaow.github.io/cnv-gene-mapping/dsc/20190627_Clean_RefGene.html) written by Min Qiao when she was at UChicago. Please refer to the link for trickiness converting `refGene.txt.gz` to gene ranges.\n",
    "\n",
    "The output is a 4-column file,\n",
    "\n",
    "```\n",
    "chr start end gene_name\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "Python3"
   },
   "source": [
    "## Genetic distance map file\n",
    "\n",
    "Rutgers genetic maps downloaded from [here](http://compgen.rutgers.edu/downloads/rutgers_map_v3.zip). Preprocessing scripts below were mostly written by Hang Dai when he was at Baylor. I put some in a workflow script to better organize them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "SoS"
   },
   "outputs": [],
   "source": [
    "# Copied from Hang Dai's preprocessing scripts in 2014\n",
    "# to SoS workflow, with minor data formatting adjustments\n",
    "\n",
    "# add_chr_to_original_file\n",
    "[preprocess_1]\n",
    "depends: executable('bgzip')\n",
    "parameter: chrom = list()\n",
    "if len(chrom) == 0: chrom = list(range(1,23)) + ['X']\n",
    "input: for_each = 'chrom'\n",
    "output: f'RUMap_chr{_chrom}.txt.gz'\n",
    "bash: expand = '${ }'\n",
    "    awk -F'\\t' -v chromosome=\"${_chrom}\" 'BEGIN {OFS=\"\\t\"} {if (NR==1) {print \"#chr\",$1,$2,$3,$6,$7,$8,$9} else {if ($2==\"SNP\") {print chromosome,$1,$2,$3,$6,$7,$8,$9}}}' RUMapv3_B137_chr${_chrom if _chrom != 'X' else 23}.txt | sort -k5 -g | bgzip -c > ${_output}\n",
    "\n",
    "# make_tabix_index_file.sh\n",
    "[preprocess_2]\n",
    "output: f'{_input}.tbi'\n",
    "bash: expand = '${ }'\n",
    "    tabix  -s1 -b5 -e5 -c# ${_input}\n",
    "\n",
    "# chr_min_max_dict\n",
    "[preprocess_3]\n",
    "input: group_by='all'\n",
    "python: expand = '${ }'\n",
    "    import subprocess\n",
    "    chr_min_max_dict={}\n",
    "    for item in [${_input:nr,}]:\n",
    "        print(item)\n",
    "        command='zcat {} | head -2 | tail -1'.format(item)\n",
    "        p=subprocess.Popen(command, universal_newlines=True, shell=True, stdout=subprocess.PIPE)\n",
    "        out=p.stdout.read().split('\\t')  #a list\n",
    "        min_pos=out[4]\n",
    "        command='zcat {} | tail -1'.format(item)\n",
    "        p=subprocess.Popen(command, universal_newlines=True, shell=True, stdout=subprocess.PIPE)\n",
    "        out=p.stdout.read().split('\\t')  #a list\n",
    "        max_pos=out[4]\n",
    "        chr_min_max_dict[item]=[min_pos, max_pos]\n",
    "    print(chr_min_max_dict)\n",
    "    print(len(chr_min_max_dict))\n",
    "\n",
    "[liftover_download: provides = ['hg19ToHg38.over.chain.gz', 'liftOver']]\n",
    "download:\n",
    "\thttps://hgdownload.soe.ucsc.edu/gbdb/hg19/liftOver/hg19ToHg38.over.chain.gz\n",
    "\thttp://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/liftOver\n",
    "bash:\n",
    "\tchmod +x liftOver\n",
    "\n",
    "[liftover_genemap]\n",
    "depends: 'hg19ToHg38.over.chain.gz', 'liftOver'\n",
    "parameter: genemap = 'genemap.hg19.txt'\n",
    "input: genemap\n",
    "output: f'{_input:nn}.hg38.txt'\n",
    "bash: expand = '${ }'\n",
    "\tawk '{print \"chr\"$1,$2,$3,$4}' ${_input} > ${_output:nn}.hg19.bed\n",
    "\t./liftOver ${_output:nn}.hg19.bed hg19ToHg38.over.chain.gz ${_output:nn}.hg38.bed ${_output:nn}.unlifted.bed\n",
    "python: expand = '${ }'\n",
    "\tgenemap = dict([(x.split()[3], x.strip().split()) for x in open(${_input:r}).readlines()])\n",
    "\tnew_coord = dict([(x.split()[3], x.strip().split()) for x in open('${_output:nn}.hg38.bed').readlines()])\n",
    "\ttotal = len(genemap)\n",
    "\tunmapped = 0\n",
    "\tfor k in list(genemap.keys()):\n",
    "\t\tif k in new_coord:\n",
    "\t\t\tgenemap[k][0] = new_coord[k][0][3:]\n",
    "\t\t\tgenemap[k][1] = new_coord[k][1]\n",
    "\t\t\tgenemap[k][2] = new_coord[k][2]\n",
    "\t\telse:\n",
    "\t\t\tdel genemap[k]\n",
    "\t\t\tunmapped += 1\n",
    "\tprint(f'{unmapped} units failed to be mapped to hg38.')\n",
    "\twith open(${_output:r}, 'w') as f:\n",
    "\t\tf.write('\\n'.join(['\\t'.join(x) for x in genemap.values()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "Python3"
   },
   "source": [
    "To use it, after downloading and decompressing Rutgers Map data, run:\n",
    "\n",
    "```\n",
    "sos run genemap.ipynb preprocess\n",
    "python genetic_pos_searcher.py genemap.txt\n",
    "mv CM_genemap.txt genemap.hg19.txt\n",
    "sos run genemap.ipynb liftover_genemap --genemap genemap.hg19.txt\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SoS",
   "language": "sos",
   "name": "sos"
  },
  "language_info": {
   "codemirror_mode": "sos",
   "file_extension": ".sos",
   "mimetype": "text/x-sos",
   "name": "sos",
   "nbconvert_exporter": "sos_notebook.converter.SoS_Exporter",
   "pygments_lexer": "sos"
  },
  "sos": {
   "kernels": [
    [
     "SoS",
     "sos",
     "",
     "",
     "sos"
    ]
   ],
   "version": "0.21.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
