# AZDSDR - Data Scientist's Data Reader

[![PyPI version](https://badge.fury.io/py/azdsdr.svg)](https://badge.fury.io/py/azdsdr)

- [AZDSDR - Data Scientist's Data Reader](#azdsdr---data-scientists-data-reader)
  - [Installation](#installation)
  - [Use Dremimo Reader](#use-dremimo-reader)
    - [Step 1. Install Dremio Connector](#step-1-install-dremio-connector)
    - [Step 2. Generate a Personal Access Token(PAT)](#step-2-generate-a-personal-access-tokenpat)
    - [Step 3. Configure driver](#step-3-configure-driver)
    - [Dremio Sample Query](#dremio-sample-query)
  - [Use Kusto Reader](#use-kusto-reader)
    - [Azure CLI Authentication](#azure-cli-authentication)
    - [Kusto Sample Query](#kusto-sample-query)
  - [Thanks](#thanks)

This package includes data reader for DS to access data in a easy way. 

Covered data platforms 

* Dremio
* Kusto

## Installation

Use pip to install the package and all of the dependences

```
pip install -U azdsdr
```

The `-U` will help update your old version to the newest

Or, you can clone the repository and copy over the `readers.py` file to your project folder.  

## Use Dremimo Reader

### Step 1. Install Dremio Connector

You will need to install the Dremio ODBC driver first to use `DremioReader` from this package. 

**For Windows user**

Please download the [dremio-connector](https://github.com/xhinker/azdsdr/tree/main/drivers) file from the drivers folder. 


### Step 2. Generate a [Personal Access Token(PAT)](https://docs.dremio.com/cloud/security/authentication/personal-access-token/#creating-a-token)

- Recommend storing this personal access token in a safe location, such as a user environment variable on your local machine.  
- Start Menu -> “Edit Environment variables For Your Account”.  
- Click “New” under environment variables.  
- Enter a new variable with name “DREMIO_TOKEN” and set the value to the PAT you generated earlier.  


### Step 3. Configure driver
- Go to Start Menu -> “ODBC Data Sources (64-bit)”.
- Under User DSN, click “Add”.
- Add Dremio Connector
- Configure as follows, with your own <alias>@host.com as the username.

![](https://github.com/xhinker/azdsdr/blob/main/README/2022-09-29-13-26-26.png)

- Click Ok/Save

**For Linux and Mac User**

You can download the driver from [Dremio's ODBC Driver](https://www.dremio.com/drivers/odbc/) page. 

### Dremio Sample Query

```python
from azdsdr.readers import DremioReader
import os

username    = "name@host.com"
#token       = "token string"
token       = os.environ.get("DREMIO_TOKEN") 
dr          = DremioReader(username=username,token=token)

sql = '''
select 
    * 
from 
    [workspace].[folder].[tablename]
limit 10
'''
r = dr.run_sql(sql)
```

## Use Kusto Reader

### Azure CLI Authentication

Before running the kusto query, please use 

```
az login
```

To login into Azure using AAD authentication. An authentication refresh token is generated by Azure and stored in your local machine. This token will be revoked after **90 days of inactivity**. 

For More details, read [Sign in with Azure CLI](https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli).

After successufuly authenticated with AAD, you should be able to run the following code without any pop up auth request. 

### Kusto Sample Query

```python 
from readers import KustoReader

cluster = "https://help.kusto.windows.net"
db      = "Samples"
kr = KustoReader(cluster=cluster,db=db)

kql = "StormEvents | take 10"
r = kr.run_kql(kql)
```

The Kusto Reader is test in Windows 10, in theroy should also work in Linux and Mac. 

## Thanks

The Dremio ODBC Reader solution is origin from [KC Munnings](https://github.com/kcm117). Glory and credits belong to KC. 
