# End2End Tests

Tests which

1. Create an engine
2. Run some tests
3. Delete the engine

## Running the Tests

### Dev Requirements

Install the dev requirements if you have not already:

```sh
$ source .venv/bin/activate  # On windows, use `.venv\Scripts\activate`
$ python -m pip install -e '.[dev]'
```

### Config

#### `raiconfig.toml`

If you have a `raiconfig.toml` file, the tests will use the specified engine.

#### Env Vars

Otherwise, the test runner authenticates with keys supplied as environment variables
and creates a new engine for the test session.

Set these variables in the environment (credentials to run against):

```
export RAI_CLIENT_SECRET=...
export RAI_CLIENT_ID=...
```

### Invocation

To run the tests locally, use the `pytest` command:

```
$ pytest -s tests/end2end
```

The command has two parts:

- The `-s` flag instructs `pytest` to display any data tests send to stdout.
- The `tests/end2end` argument tells `pytest` where to search for tests.

Another useful flag is `-k` which can be used to filter tests:

```sh
$ pytest -s -k 'myTest' tests/end2end
```

When you use the `-k` flag, `pytest` searches for tests containing the provided string
in their test function name, file name, or test ID.
Partial matches are accepted.

Here are some other useful flags:

- `--lf` runs only tests that previously failed.
- `--ff` runs previously failed tests first, followed by the rest of the tests.
- `--nf` runs newly added tests first, followed by the rest.

> [!TIP]
> Check out these pytest resources for more details:
> 
> - [How to invoke pytest](https://docs.pytest.org/en/8.0.x/how-to/usage.html)
> - [Complete pytest command-line flag reference](https://docs.pytest.org/en/8.0.x/how-to/usage.html)
> - [How to re-run failed tests and maintain state between test runs](https://docs.pytest.org/en/latest/how-to/cache.html)

## Contributing Tests

There are three steps to follow when adding an end-to-end test.

### Step 1: Add your test file to `test_cases/` folder

Inside of the `tests/end2end/test_cases/` folder are a bunch of `.py` files
that contain small end-to-end programs.
To add a new test, simply drop a new `.py` file into the folder.

As an example, consider the `smoke.py` test:

```python
import relationalai as rai

# Note the config here.
model = rai.Model("test_smoke", config=globals()["test_config"])
Person = model.Type("Person")
Adult = model.Type("Adult")

with model.rule():
    Person.add(name="Joe", age=74)
    Person.add(name="Bob", age=40)
    Person.add(name="Jane", age=10)

with model.rule():
    p = Person()
    p.age >= 18
    p.set(Adult)

with model.rule():
    p = Person()
    with p.age == 10:
        p.set(coolness=100)

with model.query() as select:
    a = Adult()
    10 <= p.age <= 80
    select(a, a.name, a.age)

with model.query() as select:
    p = Person()
    select(p, p.coolness)
```

There are a couple of things to note:

1. The `Model` class's `config` parameter is set to a special global `test_config`.
   This config is created by the test runner and must be passed to the model.
2. In each query, `select()` is called but the return value is not kept.
   Unless you need the results for a different part of the test, you can skip the assignment to `response` variable.
   The results of each query are automatically captured and compared to the snapshots contained in the `snapshots/` folder.

**Snapshots** are files containing the expected output of a query.
You do not need to create them yourself.

### Step 2: Generate snapshots

After you add a new file to the `test_cases/` directory,
invoke the `pytest` runner with the `--snapshot-update` flag to generate the snapshots for the test:

```sh
$ pytest --snapshot-update -s tests/end2end
```

The first time you run a new test file, the test fails with an error:

```sh
ERROR tests/end2end/test_snapshots.py::test_snapshots[smoke]
  - Failed: Snapshot directory was modified: tests/end2end/snapshots/test_snapshots/test_snapshots/smoke
```

That's because new snapshots were generated!
For `smoke.py`, there are two new files in the `snapshots/` directory:

- `snapshots/test_snapshots/test_snapshots/smoke/query0.txt`
- `snapshots/test_snapshots/test_snapshots/smoke/query1.txt`

But you need to verify they're correct.

### Step 3: Verify snapshots are correct

Snapshot files contain a text representation of the results data frame generated by each query in a test file.
The files are named `query<N>.txt` where `<N>` is the 0-based index of the query in order of appearance.

In the `smoke.py` test, there are two queries, so two snapshots are created:

- `query0.txt`:

   ```text
                       adult name  age
   0  3DYw2RQ6sGl/hgEt5fpReA  Joe   74
   1  Sr6rxsd7FExenA/dKVuftw  Bob   40
   ```

- `query1.txt`

   ```text
                      person  coolness
   0  8lLogIzeGtdaNKEDKcHzxQ       100
   ```

Once you look at the snapshots and verify they are correct, you can check in the test.
