# qualtrics-utils

Utilities for qualtrics surveys. Get and sync survey responses, generate codebooks, & c.

## Quickstart 🚀

This project requires Python `^3.10` to run.

### via [`poetry`](https://python-poetry.org/docs/)

Install poetry, then run

> poetry install

And you're done.

## [`surveys`](qualtrics_utils/surveys.py)

### Exporting

Example (get a survey's responses, convert to a pandas DataFrame):

```python
from qualtrics_utils import Surveys

surveys = Surveys(api_token=QUALTRICS_API_TOKEN)

exported_file = surveys.get_responses_df(
    survey_id=SURVEY_ID
)
df = exported_file.data
```

Survey's can be exported to a variety of formats, including:

-   `.csv`
-   `.xlsx`
-   `.json`

And with a variety of parameters. Please see the `ExportCreationRequest` documentation [herein](./qualtrics_utils/surveys_response_import_export_api_client/models/export_creation_request.py)

## [`sync`](qualtrics_utils/sync.py)

Perhaps one of the more useful features hereof is the ability to sync survey responses to the following services:

-   Google Sheets
-   MySQL

Future services will include:

-   PostgreSQL
-   MongoDB
-   AWS S3

Syncing can either be leveraged as a standard python module, or as a CLI tool.

### CLI

Execute the `help` command to see the available options:

```bash
python -m qualtrics_utils.sync --help
```

See also the [config.example.toml](config.example.toml) for an example configuration file.

### Module

Simply import `sync_*` from the `qualtrics_utils.sync` module, and execute the function with the appropriate arguments.

### Syncing information

The process is fairly straightforward:

1. Ensure that the service has two "tables", or datastores for:
    - Survey responses: This defaults to the input base name (defaults to the survey ID) + `_responses`
    - Survey export statuses: This defaults to the input base name (defaults to the survey ID) + `_status`
2. Export the survey responses to the service
3. Update the survey export statuses to reflect the export

This will allow for a sync to pick up where it left off, only exporting newly found responses. Please note, if a first time sync contains enough survey responses to exceed the service's limits (~1.8 GB), the sync will fail. Please see the [Qualtrics documentation](https://api.qualtrics.com/docs/response-exports) for more information.

For example, in google sheets:

```python
from qualtrics_utils import Surveys, sync_sheets
import pandas as pd

# Survey ID or URL
survey_id = ...
# Sheet URL
responses_url = ...

# dict of extra survey arguments for the export creation request
survey_args = ...

sheets = Sheets()

def post_processing_func(df: pd.DataFrame) -> pd.DataFrame:
    # Do some post processing of the responses before syncing here
    return df

sync_sheets(
    survey_id=survey_id,
    surveys=surveys,
    response_post_processing_func=post_processing_func,
    sheet_name=table_name,
    sheet_url=responses_url,
    sheets=sheets,
    **survey_args,
)

```

## Codebook mapping

### [`generate.py`](qualtrics_utils/codebook/generate_codebook.py)

Takes the exported `.qsf` file from Qualtrics and generates a codebook mapping question
IDs to question text and answer choices. The output is a JSON file containing a list of
dictionaries.

Example row:

```json
{
        "question_number": "Q5.10",
        "question_string": "What is your role at this school?",
        "answer_choices": "..."
},
```

### [`map_columns.py`](qualtrics_utils/codebook/map_codebook_columns.py)

Takes a codebook mapping (generated by the above function) and creates conditional
statements to map the question columns into valid Tableau or SQL code. Used to create a
singular question column in the above formats when there are multiple questions in a
single question block (e.g. multiple Likert scale questions).

## OpenAPI client

To handle the majority of the qualtrics API calls, we use the publicly available OpenAPI spec, found on the qualtrics API docs [website]("https://stoplight.io/api/v1/projects/qualtricsv2/publicapidocs/nodes/reference/")

The OpenAPI spec is fed to [openapi-python-client](https://github.com/openapi-generators/openapi-python-client), where it's used to generate a python client(s) for the qualtrics API. These clients (for each of the utilized APIs) come bundled and pre-generated as-is, but to create them anew, one can execute the [create_api_client.py](create_api_client.py) script. This will re-generate the clients and place them in the [qualtrics_utils](qualtrics_utils) directory. Occasionally, the OpenAPI spec is broken and mis-validated from Qualtrics; regenerate at your own risk.
