# Changelog

## 1.1.0 (2021-07-15)
### Added
- `ReportAggregator` in `model_quality_report.report_aggregator` and `ExperimentBase` in `model_quality_report.experiment_base`. `ReportAggregator` is designed to aggregate model quality reports from different models that potentially use different input/output data and can not fit into the framework of `ModelComparisonReport`. `ReportAggregator` operates with the list of classes that derive from `ExperimentBase`.   
- Dependency on [`pandera`](https://pandera.readthedocs.io/) and a few basic schemas in `model_quality_report.schemas` for report data validation.

## 1.0.0 (2021-07-14)
### Changed
- Use `.infer_objects()` after merging true and predicted data. This makes computation of quality metrics more robust.
- Quality metrics callables are organized into the dictionary name/metric `all_quality_metrics` in `model_quality_report.quality_metrics` to make easier looping over all metrics and accessing them by name. Regression metrics are accessible separately from `regression_quality_metrics` dictionary in the same module, while classification metrics are stored in `classification_quality_metrics` dictionary.
- Refactor `ModelComparisonReport`. It takes four lists of equal size: `QualityReportBase` instances, `X_data_list`, `y_data_list`, and flat dictionaries `experiment_keys`. The class now has only two public methods:
  - `create_reports` that simply returns the list quality reports for each experiment, and
  - `get_metrics` that extracts a single metrics DataFrame from multiple quality reports obtained above. The resulting DataFrame has experiment keys in corresponding columns. 
- Rename `create_quality_report_and_return_dict` method of quality report to `create_quality_report`. Now it still returns a dictionary, but the data and metrics values are pandas `DataFrame` objects.
- Rename `metrics_to_frame` method  of quality report to `get_metrics`.
- Drop missing observations from either actual or predicted series after merging them and before computing quality metrics in `CrossValidationTimeSeriesQualityReport`. This permits actual and predicted series to be not aligned in case a model produces forecasts of a different length than `y_test`. 
- Rename `mape` into `mean_absolute_percentage_error` in `model_quality_report.quality_metrics.RegressionQualityMetrics`.
- Check that predictions and true values have the same index names.
- Convert model predictions to `pd.Series` only if some other type is returned.
- Major refactoring and renaming of modules and classes.
### Added
- Run `create_reports()` in parallel Using `joblib` if the argument `n_jobs` in initialization of `ModelComparisonReport` is not `None` (see more in [`joblib.Parallel`](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html) documentation).
- New metrics for real valued data: `mean_error`, `median_error`, `mean_sign_error`, `mean_percentage_error`, `median_percentage_error`, `median_absolute_percentage_error`, `mean_absolute_cum_error`, `median_absolute_cum_error`, `mean_absolute_percentage_cum_error`, `median_absolute_percentage_cum_error`, `mean_sign_cum_error`. 
- New metrics for categorical data: `accuracy`, `precision`, `recall` metrics for classification problems.
- Logging to `model_quality_report.model_comparison_report.ModelComparisonReport` and `model_quality_report.quality_report.base.QualityReportBase`.
- Method `create_reports` of `model_quality_report.model_comparison_report.ModelComparisonReport` returns a list of dictionaries `ModelComparisonReportType` that contain experiment key, quality report, and errors string.
- Method `get_true_and_predicted_data` in `model_quality_report.model_comparison_report.ModelComparisonReport` which returns a single DataFrame with true and predicted data collected from all quality reports.
- `ByFrequency` splitter in `model_quality_report.splitters.temporal.cross_validation.by_frequency`.
### Fixed
- Quality metrics with division in the formula now return `np.nan` in case of `ZeroDivisionError`. 
- Catch `TypeError` exceptions in metrics.
### Removed
- Methods `set_model`, `get_model`, `set_splitter`, `get_splitter` from quality report classes.

## 0.2 (2019-08-26)
- New `TimeSeriesCrossValidationDataSplitter` which produces a list of splits of temporal data such that each consecutive train set has one more observation and test set one less. This class can be of use in Time Series analysis, where one wants to produce predictions for several steps ahead starting at different dates, in order to assess predictive power of a model by averaging errors across these dates for each specific horizon.
- New `CrossValidationTimeSeriesQualityReport` for cross-validation time series quality reporting. Collects quality metrics when model predictions are in general non-scalar, e.g. for several time steps ahead.
- New `ModelComparisonReport` takes a splitter, a quality report class, and the list of model/data. It calculates quality metrics for each pair and combines this into one single quality report.
- All splitters return a list of at least one split
- `QualityReportArchetype` calculates quality metrics internally through pandas objects. For that end one needs to implement a new method `_calculate_quality_metrics_as_pandas` for each new quality report that takes true and predicted values as pandas objects and returns pandas objects as well. This allows to add as many dimensions to the report as necessary. For example, one may need to compare several models for several KPI's.

## 0.1 (2019-07-10)
- First release
