# eLLMental SDK

eLLMental is a Python library for interacting with a custom evaluation API. This library allows you to set an evaluation context, execute evaluations, push test results, and manage search pipelines and test cases.

# Installation
Since eLLMental is a private repository, you need to install it from a local path. First clone the repo and then you can install the library using pip from the local path:

```sh
pip install /path/to/your/eLLMental
Replace /path/to/your/eLLMental with the actual path to the directory containing setup.py.
```

# Usage
Below is a step-by-step guide on how to use the eLLMental library.

## Initialization
First, import the library and create an instance of the eLLMental class with the base URL of your API:

```python
from Ellmental import Ellmental

base_url = "http://your-api-url.com"
authorization_token= "your bearer toke generated by eLLMental"
eLLMental = Ellmental(base_url, authorization_token)
```
## Search pipeline

### Listing Search Pipelines
List all available search pipelines:

```python
pipelines = eLLMental.list_search_pipelines()
print(pipelines)
```

### Searching with a Pipeline
Perform a search using a specific pipeline and context:

```python
pipeline_id = "your_pipeline_id"
context_id = "your_context_id"
query = "your_query"
results = pipelines = eLLMental.list_search_pipelines()
.search_pipeline_query(pipeline_id, context_id, query)
print(results)
```
## Evaluations
### Setting the Evaluation Context
Set the evaluation context by specifying the test suite ID:

```python
test_suite_id = "your_test_suite_id"
eLLMental.set_evaluation_context(test_suite_id)
```
### Listing Evaluation Cases
List all test cases in the evaluation context:

```python
test_cases = eLLMental.list_evaluation_cases()
print(test_cases)
```

### Starting an Evaluation Execution
Start an evaluation execution:

```python
eLLMental.start_evaluation_execution()
```

### Pushing Evaluation Results
Push the results of a test case:

```python
test_case_id = "your_test_case_id"
test_result = {"some": "result"}
status = "Pass"  # or "Failure"
metadata = {"additional": "info"}
response = eLLMental.push_evaluation_result(test_case_id, test_result, status, metadata)
print(response)
```

### Finishing the Evaluation Execution
Finish the evaluation execution:

```python
response = eLLMental.finish_evaluation_execution()
print(response)
```

### Complete evaluation example:
```python
from Ellmental import Ellmental
import json
import time

def main():
    eLLMental = eLLMental("http://localhost:4000", "API KEY TOKEN") # init the SDK with your instance URL
    eLLMental.set_evaluation_context(2) # init the evaluation context with your evaluation id
    test_cases = eLLMental.list_evaluation_cases() # fetch all your evaluation cases
    eLLMental.start_evaluation_execution() # start a new evaluation execution
    for test_case in test_cases:

        test_input = json.loads(test_case["test_query"])
        expected_result = int(test_case["expected_results"])
        test_result = function_under_test(test_input[0], test_input[1])
        status = "Pass" if expected_result == test_result else "Fail"

        eLLMental.push_evaluation_result(test_case["id"], test_result, status) # notify the evaluation result

    eLLMental.finish_evaluation_execution() # finish the current evaluation execution

def function_under_test(a, b):
    time.sleep(2)
    return a + b

if __name__ == "__main__":
    main()
```
## Analytics
🚧 TO DO 🚧

## Data 
 
🚧 TO DO 🚧
# Development
If you are contributing to the SDK after perfom your changes in order to create a new pip package to be installed / released you have to run the following command:
```sh
python setup.py sdist bdist_wheel
```