# Validation

## User guide - Automatic validation
Working with ``scarlet2`` provides users significant freedom when working with
``Observation``, ``Source`` and ``Scene`` objects.
In an attempt to provide guidance without restriction, we've implemented
a suite of validation checks that will run automatically at various points in a
``scarlet2`` workflow.

> Note: These validation checks are designed to be efficient.
> If a validation check is performing poorly, please let us know by creating a
> [GitHub issue](https://github.com/pmelchior/scarlet2/issues) explaining what you're
> experiencing. 

Currently automatic validation occurs when:
* An instance of the ``Observation`` class is created
* An instance of the ``Source`` class is created
* A user calls ``Scene.fit()``


### Validation results
Every individual validation check will return at least one ``ValidationResult``
object which will one of the following types: Info, Warning, or Error.
All of the results will be printed along with information about the results of
the check.

Below is an example screen shot from a jupyter notebook showing the creation of
a ``Observation`` object. All the checks returned ``ValidationInfo`` results with
the exception of the last, which returned a ``ValidationError``.

![Automatic validation after creating an Observation object.](_static/example_obs_validation.png)

> Note: Validation checks will NEVER cause the program to halt.
> i.e. Even a ``ValidationError`` will not stop the execution of ``scarlet2``.

### Toggling automatic validation checks
Automatic validation checks are enabled by default, and because the results of
validation checks will not halt the program, it is generally fine to leave them
enabled.

However, if you're running ``scarlet2`` in a way that the logged output of
validation checks won't be seen, or you feel that the checks are bothersome, they
can be togged as shown here:

```
from scarlet2.validation_utils import set_validation

# turn off automatic validation checks
set_validation(False)

# turn on automatic validation checks
set_validation(True)
```

> Note: Turning off validation is not persistent. i.e. restarting ``scarlet2``
> will re-enable automatic validation.

### Running validation checks manually
If automatic checks are disabled, or you would like to run validation checks
manually, the following functions are available:

* ``check_fit(scene, observation)``
* ``check_observation(observation)``
* ``check_scene(scene, observation, parameters)``
* ``check_source(source)``

To nicely print the validation results, you can use the `print_validation_results`
function. An example of using ``check_observation`` to run Observation validation
checks together with `print_validation_results` to view the results would look
like this:

```
from scarlet2.validation import check_observation
from scarlet2.validation_utils import print_validation_results

# Assuming the automatic validation is toggled off, we'll create an Observation
observation = Observation(...)

# Run the Observation validation checks
validation_results = check_observation(observation)

# Pretty-print the results
print_validation_results(validation_results)
```


## Dev guide - Implementing validation checks
The goal of the validation checks is to provide guidance for users as they work with ``scarlet2``.
Since guidance changes and the code base evolves, we want to encourage developing
and maintaining validation checks.

Here we include two examples to walk a developer through implementing new validation
checks within the framework that has been established.


### Example - Adding a new validation check
Ideally when modifying existing code or extending API functionality, the developer
will add corresponding validation checks.

Here we'll imagine that a hypothetical method has been added to the ``Observation``
class that will take the square root of a new input parameter ``obs_sqrt``.
The user is free to provide any value for the new input parameter.
We'll write a validation check to provide some guardrails for the user.

#### Write the basic validation check
Validation checks are implemented as methods in a ``*Validation`` class that
is co-located with the class it is validating.
For this example all the validation checks for the ``Observation`` class are
contained in the ``observation.py`` module in a class named
``ObservationValidator``.

Well add the following method to the ``ObservationValidator`` class:
```
import numbers

def check_sqrt_parameter(self) -> ValidationResult:
    """Check that the parameter to be passed to the ``obs_sqrt`` function is
    reasonable.

    Returns
    -------
    ValidationResult
        An appropriate ValidationResult subclass based on the value of ``obs_sqrt``.

    obs_sqrt = self.observation.obs_sqrt

    if not isinstance(obs_sqrt, numbers.Number):
        return ValidationError(
            message="The value of `obs_sqrt` is not numeric",
            check=self.__class__.__name__,
            context={"obs_sqrt": obs_sqrt}
        )

    elif(obs_sqrt < 0.):
        return ValidationWarning(
            message="The value of `obs_sqrt` is < 0. This might be a mistake.",
            check=self.__class__.__name__,
            context={"obs_sqrt": obs_sqrt}
        )
    else:
        return ValidationInfo(
            message="The value of `obs_sqrt` looks good.",
            check=self.__class__.__name__,
        )
```

When adding a new validation check, it is required to use the following naming scheme:
``check_<something>``.
Failure to do so will mean the check won't be run automatically with the rest of the checks.

A validation check should always return at least one instance of a subclass of ``ValidationResult``.
Returning a list of ``ValidationResults`` subclasses is also ok.
You should not return an instance of the ``ValidationResults`` directly.
The available subclasses are:
* ValidationInfo - for checks passed without any concerns.
* ValidationWarning - for something concerning, but probably won't prevent downstream tasks from completing.
* ValidationError - for what appears to be a problem, and will likely cause failures in later steps.


#### Write unit tests
In our example, new unit tests should be included in ``.../tests/test_observation_checks.py``.
Remember, it's up to the developer to determine when code has sufficient test coverage.

```
def test_obs_sqrt_returns_error(bad_obs):
    """Test that a non-numeric obs_sqrt returns an error."""
    obs = Observation(
        data=...,
        weights=...,
        channels=...,
        obs_sqrt="scarlet2",
    )

    checker = ObservationValidator(obs)

    results = checker.heck_sqrt_parameter()

    assert isinstance(results, ValidationError)
```

Of course, it would make sense to add a few more unit tests to cover the other
possible return and input types, but this should suffice for this example.

At this point the work of adding a new validation check is complete. Nice job!


### Example - Adding a new ``Validator`` class
There may be times when a completely new validator is required. 
Here we work through an example of writing a suite of validation checks for a
hypothetical ``scarlet2`` class called ``Thing``.
Fortunately the details of the ``Thing`` class aren't important beyond the
assumption that it is part of ``scarlet2``.

#### Write the new validation class
When writing the new ``Validator`` be sure to:

* [Required] Include the ``ValidationMethodCollector`` metaclass in the class definition.
* [Required] Use the naming scheme for validation checks: ``check_<something>``.
* [Encouraged] Use the validator naming scheme ``*Validator``, in this case ``ThingValidator``.
* [Encouraged] Add the new ``Validator`` in the same module (.py file) as the class it is testing. In this case, in ``thing.py`` after the ``Thing`` class.

A minimal implementation of our new validator would look like this:
```
from .validation_utils import (
    ValidationInfo,
    ValidationMethodCollector,
    ValidationWarning
)

class ThingValidator(metaclass=ValidationMethodCollector):
    """Doc string describing the purpose of `ThingValidator`."""

    def __init__(self, thing: Thing):
        self.thing = thing

    # An example check of self.thing's `parameter`.
    def check_thing(self) -> ValidationResult:
        """Doc string explaining the check.

        Returns
        -------
        ValidationResult
            ValidationResult object with info about the check.
        """
        all_good = self.thing.parameter == True
        if all_good:
            return ValidationInfo(
                message="All is good",
                check=self.__class__.__name__,
            )
        else:
            return ValidationWarning(
                message="Things might not be good",
                check=self.__class__.__name__.
                context={"all_good": all_good}
            )
```

#### Write the function to run the tests
To allow users to run the validation checks you'll implement a ``check_thing``
function in ``.../src/scarlet2/validation.py``.
For real examples see the ``check_*`` functions here: [GitHub link](https://github.com/pmelchior/scarlet2/blob/main/src/scarlet2/validation.py).

For our example the function would look like this:
```
def check_thing(thing) -> list[ValidationResults]:
    """Check the ``thing`` with the various validation checks.

    Parameters
    ----------
    thing : Thing
        The ``Thing`` instance to check.

    Returns
    -------
    list[ValidationResults]
        A list of ``ValidationResults`` from the execution of the checks in
        ``ThingValidator``.
    """

    return _check(validation_class=ThingValidator, **{"thing_to_check": thing })
```

> Note: If your validator requires more than one input, you'll need to pass those
> in here. Follow the GitHub link above and look at the ``check_fit`` function as
> an example.

#### Run the validation checks automatically
The following code should be included where appropriate, depending on when the
``Validator`` should run. For example, if the the checks should run after
initializing an object, include the snippet at the end of the ``__init__`` method.

Given that the user has not turned off automatic validation checks, the following
code would execute all the ``Thing`` validation checks and print out the results..

```
# (re)-import `VALIDATION_SWITCH` at runtime to avoid using a static/old value
from .validation_utils import VALIDATION_SWITCH

if VALIDATION_SWITCH:
    # This import happens here to avoid circular dependencies
    from .validation import check_observation

    validation_results = check_thing(self)
    print_validation_results("Observation validation results", validation_results)
```

#### Create a new test suite
Finally be sure to add a test suite for the new ``ThingValidator``.
It's best to add the new file in ``.../tests/scarlet2/test_thing_checks.py`` so
that it will be automatically detected as part of the continuous integration pipelines.

Our example test suite might look something like the following:

```
import pytest
from scarlet2.thing import Thing, ThingValidator
from scarlet2.validation_utils import (
    ValidationInfo,
    ValidationWarning,
    set_validation
)

@pytest.fixture(autouse=True)
def setup_validation()
    # Turn off auto-validation
    set_validation(False)

def test_check_thing():
    # Create an instance of a Thing
    thing = Thing(parameter=True)

    checker = ThingValidator(thing)

    results = checker.check_thing()

    assert isinstance(results, ValidationInfo)

def test_check_thing_warning();
    # Create an instance of a Thing
    thing = Thing(parameter=False)

    checker = ThingValidator(thing)

    results = checker.check_thing()

    assert isinstance(results, ValidationWarning)
```

With the test suite in place, we now have confidence that the logic in our checks
is behaving as expected. Given that we've followed the typical naming scheme for
tests and put this test suite in the correct directory, it should be discovered
automatically and included as part of the continuous integration tests with every
future commit.
