{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Module and Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import functools\n",
    "import os\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import SISSOkit.evaluation as evl\n",
    "import SISSOkit.plot as plot\n",
    "import SISSOkit.utils as utils\n",
    "pd.options.display.max_columns, pd.options.display.max_rows = None,None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path=\n",
    "cv_path=\n",
    "regression=evl.Regression(path)\n",
    "regressionCV=evl.RegressionCV(cv_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents of This Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following, there will be the results of:\n",
    "1. Baseline\n",
    "1. Training over whole data set:\n",
    "    1. Descriptors\n",
    "    1. Training errors\n",
    "1. Cross validation results\n",
    "    1. Training errors\n",
    "    1. Prediction errors\n",
    "\n",
    "In the supplementary information:\n",
    "1. Whole data set\n",
    "1. Features\n",
    "1. Operation set\n",
    "1. Feature importance\n",
    "1. Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell contains table and histogram of baseline.\n",
    "\n",
    "Baseline means the model that uses mean value as the prediction of every sample, i.e., prediction = mean(property).\n",
    "\n",
    "In this model, RMSE is the standard deviation of property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (14, 7))\n",
    "plot.baselineplot(regression,bins=50,marker_y=3,marker_shape=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training over whole data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell contains the descriptors and training errors over whole data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descriptors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.descriptors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descriptors:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (10, 7))\n",
    "plot.abs_errors_vs_dimension(regression,selected_errors=('RMSE',),display_baseline=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.total_errors(display_baseline=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Validation Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell contains training errors and prediction errors of cross validation.\n",
    "\n",
    "When using SISSO_analysis.cross_validation to generate CV files, whole data set is divided into 2 files: 'train.dat' for training and 'validation.dat' for cross validation.\n",
    "\n",
    "You should use SISSO to training each CV file and you will get model for each CV file.\n",
    "\n",
    "Training errors are the average training error over all CV files, and training errors of each CV file are the training errors over corresponding 'train.dat'.\n",
    "\n",
    "Prediction errors are the average prediction error over all CV files, and prediction errors of each CV file are the errors using model found by SISSO on the basis of 'train.dat' to predict property of samples in 'validation.dat'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (10, 7))\n",
    "plot.abs_errors_vs_dimension(regressionCV,selected_errors=('RMSE',),display_baseline=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regressionCV.total_errors(display_baseline=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot.errors_details(regressionCV)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,10))\n",
    "plot.boxplot(regressionCV)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regressionCV.total_errors(training=False,display_baseline=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (10, 7))\n",
    "plot.abs_errors_vs_dimension(regressionCV,training=False,selected_errors=('RMSE',),display_baseline=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot.errors_details(regressionCV,training=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,10))\n",
    "plot.boxplot(regressionCV,training=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supplementary Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Whole Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.features_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operation Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.operation_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature percent is the percentage of each feature in top `sub_sis` 1D descriptors.\n",
    "\n",
    "For example, if you set `sub_sis=100` in `SISSO.in`, and SISSO result shows that feature `A` appears in 30 descriptors out of the top 100 (`sub_sis`) features in `./feature_space/Uspace.name`, then the percentage of feature `A` is 30%.\n",
    "\n",
    "For feature percent of cross validation results, it is the average percentage of each CV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.features_percent()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regressionCV.features_percent()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
