# ZENETICS SDK

ZENETICS SDK is a Python library that provides a simple interface to interact with ZENETICS API. It provides a command line interface CLI to run Test Suites on your local development machine or in your CI/CD system.

The ZENETICS SDK is intended to be use in combination with the **ZENETICS Platform - Quality Management for LLM Applications**. To get started with LLM testing, sign up at [www.zenetics.io](https:://www.zenetics.io).

## Installation

The ZENETICS SDK is provided as Python module an can be installed easily in your environment.

```commandline
> pip install zenetics

or

> poetry add zenetics
```

## Setup

### Setting The ZENETICS_API_KEY

To access your tests, you need to set the ENV-variable `ZENETICS_API_KEY` in your environment. Alternatively, you can update your .env file if your project supports it.

```bash
export ZENETICS_API_KEY=xxx-xxx-xxx
```

You can access your ZENETICS API Key in the [ZENETICS Portal](https://app.zenetics.io). API keys are application specific, and you need to select the application you want to test. Once you have selected the application, you will find your API Key under the menu item **Settings** in the left sidebar.

The section "API Key - Test Runners" lists all the available API-keys. You can create a new one with simply by clicking **Create API Key**.

![ZENETICS Portal Application Settings](/docs/images/zenetics_portal_application_settings.png)

### Validating the Setup

You can run the check command of the ZENETICS CLI to validate if your setup is working:

```bash
> zenetics check

or

> poetry run zenetics check
```

The command will show a success message or list which parts of the setup are missing.

### Implementing the Generation Function

The ZENETICS Test Runner requires access to the results of your application, and optional metadata to run the evaluations for the selected test cases. ZENETICS offers an easy way to connect your application by creating a Python file that implements a single function: `def generate(input: string) -> Generation". This function receives an input string that represents the input from the test cases and which is used by your application to generate the result.

See the following code example of the basic structure for the generate-function:

```python

from zenetics.models.generation import Generation, TokenUsage, ModelMetadata


# This is the only function that you need to provide. It connects the
# ZENETICS Test Runner with your applications and captures the relevant
# information for the evaluation process.
def generate(input: str) -> Generation:

    # Generate the output using your LLM application.
    #
    # output:            The result generated by your LLM application.
    #                    (type: str)
    #
    # retrieval_context: The optional retrieval context for RAG applications. This
    #                    information is required for Relevance evaluators like
    #                    Answer Relevance, Context Relevance, and Groundedness.
    #                   (type: List[str])
    #
    # Replace `YOUR_APPLICATION.generate` with the function that generates the output
    #
    output, retrieval_context = YOUR_APPLICATION.generate(input)

    # You can also provide additional information that helps you to better understand
    # the performance and characteristics of your model. We recommend that you provide
    # both TokenUsage and ModelMetadata information.
    output, retrieval_context, token_usage, metadata = YOUR_APPLICATION.generate(input)


    # The Generation object contains the necessary information for the
    # ZENETICS Test Runner to evaluate the specific test case and perform the
    # evaluations assigned to to the test suite.
    return Generation(
        output=output,
        retrieval_context=retrieval_context,

        # Optional: Token usage helps you track the tokens required for
        # generating results.
        token_usage=TokenUsage(    # replace this with the data returned from your app.
            input_tokens=864,
            completion_tokens=567,
            total_tokens=1431
        ),
        # Optional: Metadata helps you relate test results to your specific
        # model configurations.
        metadata=ModelMetadata(    # replace this with the actual metadata from your app.
            provider="OpenAI",
            model= "gpt-3.5-turbo",
            parameters={
                "max_tokens": 1024,
                "temperature": 0.7
            },
        )
    )
```

Depending on your application, the `generate` function can call your application directly by importing it into the file, or call your API directly if the version your want to test runs on a server. You are free to provide different implementations and select the specific implementation when you run your test (see the next section: **Usage**).

## Usage

### Running Test Suites

ZENETICS comes with a simple command line interface (CLI) that lets your run you tests. The **run** command of the CLI requires the following parameters:

1. Test Suite Id: Identifier of the test suite you want to run. You can access this in the [ZENETICS Portal](https://app.zenetics.io).
2. Generation File: Path to the generation file you implemented.

You can run a selected test suite with the following example command.

```bash
> zenetics run 1234 src/test/generate.py

or

> poetry zenetics run 1234 src/test/generate.py
```

Make sure that you replace both the test suite id (`1234`) and the filepath to the generation-file (`src/test/generation.py`) with the correct values for your application.

The Test Runner will output a summary of the result in the console.
![ZENETICS Console - Test Run Result](/docs/images/zenetics_cli_run_result.png)

When running tests, you can add the following parameters:

```bash
> zenetics run 1234 src/test/generate.py --verbose --local-only --output-dir=./test-results

or

> poetry zenetics run 1234 src/test/generate.py --verbose --local-only --output-dir=./test-results
```

-   output-dir=Defines the directory for storing the local test run files (./test-results by default)
-   verbose: Prints also passed test cases in the result summary
-   local-only: Only creates a local test run file but no test run in ZENETICS

### Additional Commands

#### Help

You can see the supported commands and their parameters with the help command:

```bash
> zenetics help

or

> poetry zenetics help
```

#### Check

You can check if your setup is working correctly:

```bash
> zenetics check

or

> poetry zenetics check
```

#### List Test Suites

You can list all available test suites for the application linked to the configured ZENETICS_API_KEY:

```bash
> zenetics testsuites

or

> poetry zenetics testsuites
```

This prints out all available test suites for the application.
![ZENETICS Portal Application Settings](/docs/images/zenetics_cli_testsuites.png)
