# Testset

The testset contains the tests that will be used to judge a problem,
and can be decomposed into two components:

- Its skeleton: which are tests are samples? how other tests are grouped?
- How to generate tests for each group? Are the tests manually defined? Are they generated by a script?

The test skeleton can be specified in the `problem.rbx.yml` file through the `testcases` field. This field
is a list of [`TestcaseGroup`][rbx.box.schema.TestcaseGroup] objects, which describe a group of tests.

Every test group should have an unique name. The name will be used to identify the group of tests when running
{{rbx}} commands. There's a special reserved name, `samples`, which will be used to identify the group of samples.

Below, an example of a very simple, ICPC-style test plan: just two groups, one secret, with tests hidden to the user
and one with the samples.

```yaml title="problem.rbx.yml"
# ...
testcases:
  - name: 'samples'
  - name: 'secret'
```

Of course, we have to add tests to these groups. The rest of this section will be devoted to this topic.

There are 5 different ways of adding tests to group:

+----------------------------+-------------------+-----------------------------------------------------------------------+
|           Method           |       Field       |                              Description                              |
+============================+===================+=======================================================================+
| List of testcases          | `testcases`       | A list of [`Testcase`][rbx.box.schema.Testcase] objects that manually |
|                            |                   | defines a few testcases.                                              |
+----------------------------+-------------------+-----------------------------------------------------------------------+
| A testcase glob            | `testcaseGlob`    | A path glob (string) that matches a set of testcase                   |
|                            |                   | inputs (`.in` files).                                                 |
+----------------------------+-------------------+-----------------------------------------------------------------------+
| A list of generator calls  | `generators`      | A list of [`GeneratorCall`][rbx.box.schema.GeneratorCall] objects,    |
|                            |                   | which describe how to generate the tests for the group.               |
+----------------------------+-------------------+-----------------------------------------------------------------------+
| A static generator script  | `generatorScript` | A path to a `.txt` file -- each of its lines is a                     |
| (*aka* a testplan)         |                   | generator call for a testcase.                                        |
+----------------------------+-------------------+-----------------------------------------------------------------------+
| A dynamic generator script | `generatorScript` | A path to a code or script that, when called, will generate           |
|                            |                   | a static generator script.                                            |
+----------------------------+-------------------+-----------------------------------------------------------------------+

In this section, we'll talk about the two most recommended approaches: using a testcase glob and using a generator script.

## Defining the testset

### Testcase glob

Testcase globbing is the simplest way of adding manually defined tests to a group.

```yaml title="problem.rbx.yml"
testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
```

In the example above, we define a group of samples which will contains tests matching the glob `tests/*.in`.

Thus, if there are 3 files in the `tests` directory, `tests/01.in`, `tests/02.in` and `tests/03.in`, all these
three will be added to the samples group.

!!! warning "Test ordering"

    The order of the tests will be the lexicographical order of the files.

    Be careful to not define tests as `1.in`, `2.in`, ..., `10.in` as this will lead to a test set where
    test `10.in` is executed before test `2.in`.

    Instead, define the tests as `01.in`, `02.in`, ..., `10.in`, using leading zeroes.

### Generator script

*If you haven't read the [Generators section](generators.md) yet, you should read it before proceeding.*

A generator script is a script that will be used to generate tests for a group.

It can be either a static script (in which case we also call it a testplan) or a dynamic script,
and can be specified through the `generatorScript` field of a test group.

#### Static generator script (*aka* testplan)

A static generator script (or a testplan) is a `.txt` file containing a list of line-separated generator calls.

A generator call is simply a pair of `<generator-name> <generator-args...>`, where `<generator-args>` is a list
of space-separated arguments to pass to the generator.

Testplan can also have lines starting with a `#`, denoting this line is a comment and should be ignored, or
even empty lines.

Below there's an example of a testplan for a problem that has two generators, `random` and `small`, and how
to define it in the `problem.rbx.yml` file.

=== "testplan.txt"

    ```
    # Two random tests
    random 10 100
    random 10 1000

    # Two small tests
    small 50
    small 100
    ```

=== "problem.rbx.yml"

    ```yaml hl_lines="4-5"
    testcases:
      - name: 'samples'
        testcaseGlob: 'tests/*.in'
      - name: 'secret'
        generatorScript:
          path: 'testplan.txt'
    ```

#### Dynamic generator script

A dynamic generator script is a code that produces a testplan. Think of a code (in Python, or even in C++)
that produces a testplan file as its output.

Below, there's an example of a dynamic generator script for a problem that has a `random` generator.

=== "testplan.py"

    ```python
    for i in range(10):
        print(f"random {i}")
    ```

=== "problem.rbx.yml"

    ```yaml hl_lines="4-5"
    testcases:
      - name: 'samples'
        testcaseGlob: 'tests/*.in'
      - name: 'secret'
        generatorScript:
          path: 'testplan.py'
    ```

The script spits a testplan with exactly 10 random tests, each one generated from a different argument between
0 and 9.

#### Advanced testplan features

The static generator script (testplan) supports a few advanced features that can be useful in some scenarios.

##### Manual tests (`@input`)

You can define manual tests directly in the testplan using the `@input` directive. This is useful when you want to add
some edge cases that are hard to generate with a generator.

You can specify the input content using a quoted string (single or double quotes) or a block of braced content.

```testplan
# Single line input (not raw)
@input "10 20\n"

# Multi-line input (triple quotes, raw)
@input """10
20
30"""

# Multi-line input (braces, raw, lines around the input are stripped)
# Recommended approach.
@input {
10
20
30
}
```

##### Copying tests (`@copy`)

You can include existing test files in the testplan using the `@copy` directive. This is useful when you have
some manual tests in a directory and you want to include them in the testplan.

```testplan
@copy tests/manual/01.in
@copy tests/manual/02.in
```

##### Grouping tests (`@testgroup`)

You can group tests using the `@testgroup` directive. This is useful when you want to add
the specified tests only to testgroups that have this name. This is useful when different testgroups import the same testplan, but you want to create separate tests for each group.

```testplan
# This test will be part of any testgroup that imports this testplan.
random 5

@testgroup subtask1 {
    # These tests will be part only of testgroups that have the name 'subtask1' and
    # import this testplan.
    random 10
    random 20
    @input "5"
}

@testgroup subtask2 {
    # These tests will be part only of testgroups that have the name 'subtask2' and
    # import this testplan.
    random 100
}
```

### What about the outputs?

Until now, we've just generated the inputs of our testcases. What about the outputs? Where they come from?

By default, {{rbx}} will use the model solution to generate the outputs of the testcases. This will be done
when building the testset.

The model solution is the topmost {{tags.accepted}} solution in the `solutions` field of the `problem.rbx.yml` file.

In some cases, though, it's useful to specify a different output than the one generated by the model solution. You can create an `.ans` file in the same directory as the `.in` file to specify the answer for the problem, and
override the one provided by the model solution (if it exists).

Notice this will affect both the testset itself, and the sample output
rendered in the statement.

There are times, though, where you only want to modify the statement output
without affecting the testset (think of cases where the model solution
output reveals too much about the intended solution). You can create an `.out` file in the same directory as the `.in` file to specify the output for the problem.

You can only create manually crafted outputs for testcases you've defined manually (with
a testcase glob, for instance).

Let's look at the file tree above, and assume we have a testcase glob for samples such as `tests/*.in`.

```bash
- tests/
  - 01.in
  - 01.out  # This will override the output in the statement.
  - 02.in
  - 02.ans  # This will override the answer both in the statement AND in the testset.
```

#### What about interactive problems?

For interactive problems, you can create a `.interaction` file in the very same path (and with the very same name) as the `.in` files you've manually defined.

This file should contain the interaction between the solution and the interactor. Each line of interaction should start
with either `<` or `>` to denote whether this was the output of the interactor or the output of the solution, respectively.

```
< this is an interactor line
< followed by another line
> this is the solution
```

## Building the testset

The command below can be used to build the testset.

```sh
rbx build
```

{{ asciinema("LlhvlLY5Fmee2l1OgCvFQk0xM") }}

This command will build the testset, using the generator scripts to generate the tests for each group. All tests
will be written to the `build/tests` directory, which you can inspect manually in our file system.

This command also accepts an extra verification flag (`-v`), which you can use to control whether validators will
be run after generating the tests or not. The flag defaults to `-v0`, which means no verification will be done.

```sh
rbx build -v1
```

You can read more about the verification level flag in the [verification section](/setters/verification#verification-level) and
about validation in the [Validators section](/setters/verification/validators).

## Visualizing the testset

You can use the `rbx ui` to visualize the testcases that were built through the `rbx build` command.

```sh
rbx ui
```

{{ asciinema("cqUTWgIRFA1P7VsV39uJTorKC") }}

This command will start an interactive UI in your terminal which you can use to browse the testset.

