Metadata-Version: 2.1
Name: clinvar-this
Version: 0.1.0
Summary: ClinVar Submission via API Made Easy
Home-page: https://github.com/bihealth/clinvar_this
Author: Manuel Holtgrewe
Author-email: manuel.holtgrewe@bih-charite.de
License: MIT license
Keywords: clinvar
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: AUTHORS.rst

[![codecov](https://codecov.io/gh/bihealth/clinvar-this/branch/main/graph/badge.svg?token=059T45KAQM)](https://codecov.io/gh/bihealth/clinvar-this)
[![Documentation Status](https://readthedocs.org/projects/clinvar-this/badge/?version=latest)](https://clinvar-this.readthedocs.io/en/latest/?badge=latest)

# ClinVar This!

ClinVar Submission via API Made Easy

- Free software: MIT license
- Documentation: https://clinvar-this.readthedocs.io/en/latest/

## Getting Started

You will need some experience with VCF files, ClinVar, and the Linux/Mac command line.

### Obtain ClinVar API Key

First of all, you need to register your organisation with NCBI, request a service account, and obtain an API key.
Skip any step if you have already completed it.

1. [Register your organisation with NCBI as they document](https://www.ncbi.nlm.nih.gov/clinvar/docs/api_http/)
2. Send an email to clinvar@ncbi.nlm.nih.gov to request a service account for your organisation.
3. Once you have a service account, create an API key as outlined [at the top of the NCBI ClinVar API documentation](https://www.ncbi.nlm.nih.gov/clinvar/docs/api_http/).

### Install clinvar-this

You can either install the [PyPi package clinvar-this](https://pypi.org/project/clinvar-this/):

```
# pip install clinvar-this
```

Or you install via [conda/bioconda](http://bioconda.github.io/).

```
# conda install -c clinvar-this
```

Check that your installation worked:

```
# clinvar-this --help
Usage: clinvar-this [OPTIONS] COMMAND [ARGS]...

  Main entry point for CLI via click.

Options:
  --verbose / --no-verbose
  --profile TEXT            The profile to use
  --help                    Show this message and exit.

Commands:
  batch   Sub comment category ``batch ...``
  config  Sub command category ``varfish-this config ...``
```

### Configure your API Token

```
# clinvar-this config set auth_token YOUR_AUTH_TOKEN_HERE
```

Check that this worked:

```
# clinvar-this config dump
# path: /home/holtgrem_c/.config/clinvar-this/config.toml
[default]
auth_token = "YOUR_AUTH_TOKEN_HERE"
```

### Prepare a clinvar-this TSV file.

You will need the following header in the first line

- `ASSEMBLY` - the assembly used, e.g., `GRCh37`, `hg19`, `GRCh38`, `hg38`
- `CHROM` - the chromosomal position without `chr` prefix, e.g., `1`
- `POS` - the 1-based position of the first base in `REF` column
- `REF` - the reference allele of your variant
- `ALT` - the alternative allele of your variant
- `OMIM` - the OMIM id of the carrier's condition (not the OMIM gene ID), e.g., `619325`.
  Leave empty or use `not provided` if you have no OMIM ID.
- `MOI` - mode of inheritance, e.g., `Autosomal dominant inheritance` or `Autosomal recessive inheritance`
- `CLIN_SIG` - clinical significance, e.g. `Pathogenic`, or `Likely benign`
- `CLIN_EVAL` - optional, date of late clinical evaluation, e.g. `2022-12-02`, leave empty to fill with the date of today
- `CLIN_COMMENT` - optional, a comment on the clinical significance, e.g., `ACMG Class IV; PS3, PM2_sup, PP4`
- `KEY` - optional, a local key to identify the variant/condition pair.
  Filled automatically with a UUID if missing, recommeded to leave empty.
- `HPO` - List of HPO terms separated by comma or semicolon, any space will be stripped.
  E.g., `HP:0004322; HP:0001263`.

The following shows an example.

```
ASSEMBLY	CHROM	POS	REF	ALT	OMIM	MOI	CLIN_SIG	HPO
GRCh37	19	48183936	C	CA	619325	Autosomal	dominant	inheritance	Likely	pathogenic	HP:0004322;HP:0001263
```

Note that you must use TAB characters (`\t`) for separating the file.

### Import the TSV file into clinvar-this

Use the `batch import` command to import the TSV file into the local clinvar-this storage.

```
# clinvar-this batch import --name=BATCHNAME DATA_FILE.tsv
```

If you do not specify the `--name` parameter then clinvar-this will generate one based on the current time.
This will create a new batch storage folder below `~/.local/share/clinvar-this/default` with the batch name and place a file `payload.$timestamp.json` there.
This corresponds to the data that will be uploaded into ClinVar.

You can now import another TSV file or change your TSV file and re-import it to apply the changes.

### Submit via ClinVar API

Use `batch submit BATCHNAME` to submit the data to the ClinVar API.

```
# clinvar-this batch submit BATCHNAME
```

This will create a new file `submission-response.$timestamp.json` in the batch storage folder.
This file stores the identifier of the ClinVar submission.
This information is subsequently used in `batch retrieve`.

### Retrieve ClinVar API Submission Result

You can now use the following command to query the ClinVar API for the status of your submission.

```
# clinvar batch retrieve BATCHNAME
```

It will get the submission ID from the latest `submission-response.*.json` file (using lexicographic file name comparison) and query the ClinVar API.
The API response will be written to `retrieve-response.$timestamp.json`.
In the case that the API has processed your submission, clinvar-this will create a new `payload.$timestamp.json` file to reflect the change.
You will probably have to wait a few or many minutes until the processing finishes.
This will store any error message or ClinVar SCV.

### Obtain SCV or Error Message

You could now look at the `payload.$timestamp.json` file to see the full server response.
It is more convenient, however, to export the results to a TSV file again which will display the SCV identifiers and any error message:

```
# clinvar-this batch export BATCHNAME DATA_FILE.reply.tsv
```

The [ClinVar API documentation](https://www.ncbi.nlm.nih.gov/clinvar/docs/api_http/) says that variants submitted via the API do not have to pass manual curation.
That is, the server will perform a number of checks.
If your variants pass all checks then you will directly obtain an SCV and the variants will become publically available on the next Sunday.

### Rinse and Repeat

In the case of a partial success, update the exported TSV file and submit it again until you are happy.

## Caveats

- **The `--use-testing` and `--dry-run` mode.**
  When enabling `--use-testing`, an alternative API endpoint provided by ClinVar will be used.
  This endpoint may use a different schema than the official endpoint (e.g., this has happened in November 2022).
  ClinVar has previously notified their submitters via email without official news posts.


# Changelog

## 0.1.0 (2022-12-02)


### Features

* add basic config management in CLI ([#32](https://www.github.com/bihealth/clinvar-this/issues/32)) ([#33](https://www.github.com/bihealth/clinvar-this/issues/33)) ([c903546](https://www.github.com/bihealth/clinvar-this/commit/c903546751fe6c59fcc20770b6a1ad1ac88fbd86))
* add functions for managing batch data ([#34](https://www.github.com/bihealth/clinvar-this/issues/34), [#37](https://www.github.com/bihealth/clinvar-this/issues/37)) ([#35](https://www.github.com/bihealth/clinvar-this/issues/35)) ([0c7e0f9](https://www.github.com/bihealth/clinvar-this/commit/0c7e0f9b65dc0b2189c302065cdebdc2489ea842))
* add sphinx-based documentation ([#30](https://www.github.com/bihealth/clinvar-this/issues/30)) ([#31](https://www.github.com/bihealth/clinvar-this/issues/31)) ([d10adc5](https://www.github.com/bihealth/clinvar-this/commit/d10adc50a720548b580b7aa6bd52477012bb6d29))
* add tests for submission messages ([#19](https://www.github.com/bihealth/clinvar-this/issues/19)) ([#22](https://www.github.com/bihealth/clinvar-this/issues/22)) ([b168d47](https://www.github.com/bihealth/clinvar-this/commit/b168d47e0b3aed8d3a706ddbd28ddbb635ad86b4))
* add unit tests for submission messages ([#19](https://www.github.com/bihealth/clinvar-this/issues/19)) ([#20](https://www.github.com/bihealth/clinvar-this/issues/20)) ([1c4e11a](https://www.github.com/bihealth/clinvar-this/commit/1c4e11aa6eda5ebce0a66e8f21f17ea89c7ed7e2))
* adding mypy type checking ([#11](https://www.github.com/bihealth/clinvar-this/issues/11)) ([#12](https://www.github.com/bihealth/clinvar-this/issues/12)) ([700994a](https://www.github.com/bihealth/clinvar-this/commit/700994a113f2441bba76e3dc95adccf9fed6a156))
* adjust to ClinVar API change (Nov 2022) ([#47](https://www.github.com/bihealth/clinvar-this/issues/47)) ([#48](https://www.github.com/bihealth/clinvar-this/issues/48)) ([0e4fb50](https://www.github.com/bihealth/clinvar-this/commit/0e4fb508bda571c0c0afdab417e9052667f46ea1))
* allow annotation with HPO terms in TSV format ([#50](https://www.github.com/bihealth/clinvar-this/issues/50)) ([#56](https://www.github.com/bihealth/clinvar-this/issues/56)) ([0b0da41](https://www.github.com/bihealth/clinvar-this/commit/0b0da41e938e464e97ea1be7edcad6c666860f1e))
* allow import of extra columns ([#53](https://www.github.com/bihealth/clinvar-this/issues/53)) ([#54](https://www.github.com/bihealth/clinvar-this/issues/54)) ([616bfe7](https://www.github.com/bihealth/clinvar-this/commit/616bfe7867fe7ddd1c898f7582b1a353069b7ccd))
* completing enums ([#23](https://www.github.com/bihealth/clinvar-this/issues/23)) ([#24](https://www.github.com/bihealth/clinvar-this/issues/24)) ([9198983](https://www.github.com/bihealth/clinvar-this/commit/919898305a6155408118cd43a982c78153976879))
* implement api models ([#4](https://www.github.com/bihealth/clinvar-this/issues/4)) ([#5](https://www.github.com/bihealth/clinvar-this/issues/5)) ([3690c36](https://www.github.com/bihealth/clinvar-this/commit/3690c36b25fc98e520a401f8755a48b3451a972f))
* implement attrs-based message models ([#1](https://www.github.com/bihealth/clinvar-this/issues/1)) ([#2](https://www.github.com/bihealth/clinvar-this/issues/2)) ([f253c24](https://www.github.com/bihealth/clinvar-this/commit/f253c248cf35f749ace3c68633eb8d0357349688))
* implement enums for submission messages ([#14](https://www.github.com/bihealth/clinvar-this/issues/14)) ([#16](https://www.github.com/bihealth/clinvar-this/issues/16)) ([201cbe4](https://www.github.com/bihealth/clinvar-this/commit/201cbe4e055e0d013efe6f85a91b5ec339c77838))
* implement internal models for submissions ([#9](https://www.github.com/bihealth/clinvar-this/issues/9)) ([#26](https://www.github.com/bihealth/clinvar-this/issues/26)) ([be04e40](https://www.github.com/bihealth/clinvar-this/commit/be04e40e15adba180827fdc808c17d7d1647edb7))
* implement minimal TSV format ([#17](https://www.github.com/bihealth/clinvar-this/issues/17)) ([#18](https://www.github.com/bihealth/clinvar-this/issues/18)) ([960827d](https://www.github.com/bihealth/clinvar-this/commit/960827d9be757e5cc1cfe537451e3a34c738b256))
* implement models for extra files ([#6](https://www.github.com/bihealth/clinvar-this/issues/6)) ([#13](https://www.github.com/bihealth/clinvar-this/issues/13)) ([852e4c6](https://www.github.com/bihealth/clinvar-this/commit/852e4c6cd981ae533afbfda0217a4b9e1de8065e))
* implement models for submission message ([#8](https://www.github.com/bihealth/clinvar-this/issues/8)) ([#15](https://www.github.com/bihealth/clinvar-this/issues/15)) ([90afd10](https://www.github.com/bihealth/clinvar-this/commit/90afd105fc0d29c3962529157f38a20e48090a56))
* implement more columns in TSV ([#39](https://www.github.com/bihealth/clinvar-this/issues/39)) ([#55](https://www.github.com/bihealth/clinvar-this/issues/55)) ([6184d76](https://www.github.com/bihealth/clinvar-this/commit/6184d76a2f8261131718bbe69e88fdaf66b3a8de))
* implementing REST clinvar_api.client module ([#28](https://www.github.com/bihealth/clinvar-this/issues/28)) ([#29](https://www.github.com/bihealth/clinvar-this/issues/29)) ([829d907](https://www.github.com/bihealth/clinvar-this/commit/829d907763db617a04d277f297256980d25d51ab))
* store errors from "batch retrieve" ([#59](https://www.github.com/bihealth/clinvar-this/issues/59)) ([#60](https://www.github.com/bihealth/clinvar-this/issues/60)) ([67b8b0c](https://www.github.com/bihealth/clinvar-this/commit/67b8b0ccd85d4a913e7b8f1eb30dc4e96a2af32c))
* write data into profile sub directory ([#52](https://www.github.com/bihealth/clinvar-this/issues/52)) ([e699736](https://www.github.com/bihealth/clinvar-this/commit/e6997367d77586c1337a44fa7b5306ecede52a41))


### Bug Fixes

* interpret TSV OMIM "not provided" ([#57](https://www.github.com/bihealth/clinvar-this/issues/57)) ([#58](https://www.github.com/bihealth/clinvar-this/issues/58)) ([b655097](https://www.github.com/bihealth/clinvar-this/commit/b655097f2bf534b57115e3f9f5d18387ddfb1f18))


### Documentation

* add getting started tutorial to README ([#45](https://www.github.com/bihealth/clinvar-this/issues/45)) ([#61](https://www.github.com/bihealth/clinvar-this/issues/61)) ([3de084b](https://www.github.com/bihealth/clinvar-this/commit/3de084b49c184dfc6fb977b64cf478f03275ef9b))
