\documentclass[british]{article}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{geometry}
\geometry{verbose,tmargin=3.5cm,bmargin=3.5cm,lmargin=3cm,rmargin=3cm}
\usepackage{array}
%\usepackage{multirow}
\usepackage{amstext}
\usepackage{graphicx}
\usepackage{color}
\usepackage[font=small,labelfont=bf]{caption}
%\usepackage{subfigure}

\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}

\newcommand{\mytablefontsize}{7pt}
\newcommand{\mytablebaselineskip}{0.7}
\newcommand{\mytabcolsep}{3pt}

\newcommand{\medianInterval}[1]{}

\makeatletter

\newif\iftest
@@testBool@@

\newif\ifruntime
@@runtimeBool@@

\newif\ifablation
@@ablationBool@@

\newif\iffeatures
@@featuresBool@@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
%% Because html converters don't know tabularnewline
\providecommand{\tabularnewline}{\\}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.

\title{Configuration Report for the Solver @@solver@@ on the Training Instance Set @@instanceSetTrain@@ \iftest and Testing Instance Set @@instanceSetTest@@ \fi in \emph{Sparkle} }
\author{ Automatically generated by \emph{Sparkle} (version: @@sparkleVersion@@) }

\makeatother

\usepackage{babel}
\begin{document}
\maketitle %


\section{Introduction}
\label{sec:Introduction}

\emph{Sparkle} \cite{Hoos15} is a multi-agent problem-solving platform based on Programming by Optimisation (PbO) \cite{Hoos12}, and would provide a number of effective algorithm optimisation techniques (such as automated algorithm configuration, portfolio-based algorithm selection, etc) to accelerate the existing solvers.

This experimental report is automatically generated by \emph{Sparkle}. This report presents experimental results on the scenario of configuring the solver @@solver@@ on the training instance set @@instanceSetTrain@@\iftest~and evaluating it on the testing instance set @@instanceSetTest@@\fi.


\section{Information about the Instance Set(s)}

\begin{itemize}
\item Training set: \textbf{@@instanceSetTrain@@}, consisting of @@numInstanceInTrainingInstanceSet@@ instances
\iftest\item \textbf{@@instanceSetTest@@}, consisting of @@numInstanceInTestingInstanceSet@@ instances\fi
\end{itemize}


\iffeatures
    \section{Feature Extractors}
        There are @@numFeatureExtractors@@ feature extractor(s) included in \emph{Sparkle}, as listed below.

        \begin{enumerate}
        @@featureExtractorList@@
        \end{enumerate}

        \emph{Sparkle} uses all the feature extractors presented above to compute a feature vector for each training instance. Every feature extractor computes a feature vector for each training instance. The final feature vector is then the combination of all computed feature vectors. The cutoff time for feature vector computation on each training instance is set to @@featureComputationCutoffTime@@ seconds. This time is shared equally between the feature extractors.

        The calculated instance features are then passed on to be used in the process of solver configuration. 

\fi


\section{Information about the Configuration Protocol}

The configurator used in \emph{Sparkle} is SMAC ({\em Sequential Model-based Algorithm Configuration}) \cite{HutEtAl11}, and the version of SMAC used in \emph{Sparkle} is 2.10.03.

During the configuration process, \emph{Sparkle} performs @@numSmacRuns@@ independent SMAC runs for configuring the solver @@solver@@ on the training instance set @@instanceSetTrain@@\iffeatures; the instance features of the training instance set were used for configuration\fi; the configuration objective is @@smacObjective@@; the whole configuration time budget is @@smacWholeTimeBudget@@ seconds; the cutoff time for each run is @@smacEachRunCutoffTime@@ seconds.

Each independent run of SMAC would result in one optimised configuration. As a result, \emph{Sparkle} would obtain @@numSmacRuns@@ optimised configurations. Each of these was then evaluated on the entire training set, with one solver run per instance and a cutoff time of @@smacEachRunCutoffTime@@ seconds, and the configuration with the lowest @@performanceMeasure@@ value was selected as the result of the configuration process.

\section{Information about the Optimised Configuration}

After the configuration process mentioned above, \emph{Sparkle} obtained the optimised configuration. The details of the optimised configuration are described below.

\vspace{5mm}

@@optimisedConfiguration@@

\vspace{5mm}

%The PAR value of the optimised configuration on the entire training set is @@optimisedConfigurationTrainingPerformancePAR@@.

\section{Comparison between Configured Version and Default Version on the Training Instance Set}
In order to investigate the performance on the training instance set, \emph{Sparkle} ran the configured version of @@solver@@ and the default version of @@solver@@ on the training instance set. During this phase, each version performed one run per instance with a cutoff time of @@smacEachRunCutoffTime@@ seconds. The results are reported as follows.

\begin{itemize}
    \item \textbf{@@solver@@ (configured)}, @@performanceMeasure@@: @@optimisedConfigurationTrainingPerformancePAR@@
    \item \textbf{@@solver@@ (default)}, @@performanceMeasure@@: @@defaultConfigurationTrainingPerformancePAR@@
\end{itemize}

The empirical comparison between the @@solver@@ (configured) and @@solver@@ (default) on the training set of @@instanceSetTrain@@ is presented in Figure \ref{fig:configured_vs_default_train}.

\begin{figure}[htbp]
\noindent \begin{centering}
@@figure-configured-vs-default-train@@
% \includegraphics[width=0.8\textwidth]{fittedModels}
\par\end{centering}

\caption{Empirical comparison between the @@solver@@ (configured) and @@solver@@ (default) on the training set of @@instanceSetTrain@@.}\label{fig:configured_vs_default_train}
\end{figure}

%\ifruntime

Table \ref{tbl:timeouts_train} shows on how many instances the @@solver@@ (configured) and @@solver@@ (default) timed out (did not solve the instance within the cutoff time of @@smacEachRunCutoffTime@@ seconds) on the training set of @@instanceSetTrain@@, as well as on how many instances both timed out.

    \begin{table}[htbp]
        \centering
            \begin{tabular}{ccc}
                configured & default & overlap \\ \hline
                @@timeoutsTrainConfigured@@ & @@timeoutsTrainDefault@@ & @@timeoutsTrainOverlap@@
            \end{tabular}
            \caption{Number of time-outs for @@solver@@ (configured), @@solver@@ (default), and for how many instances both timed out on the training set of @@instanceSetTrain@@.}
        \label{tbl:timeouts_train}
    \end{table}

%\fi % End ifruntime

\iftest
    \section{Comparison between Configured Version and Default Version on the Testing Instance Set}

    After specifying the optimised configuration, \emph{Sparkle} ran the configured version of @@solver@@ and the default version of @@solver@@ on the testing instance set. During this phase, each version performed one run per instance with a cutoff time of @@smacEachRunCutoffTime@@ seconds. The results are reported as follows.

    \begin{itemize}
        \item \textbf{@@solver@@ (configured)}, @@performanceMeasure@@: @@optimisedConfigurationTestingPerformancePAR@@
        \item \textbf{@@solver@@ (default)}, @@performanceMeasure@@: @@defaultConfigurationTestingPerformancePAR@@
    \end{itemize}

    The empirical comparison between the @@solver@@ (configured) and @@solver@@ (default) on the testing set of @@instanceSetTest@@ is presented in Figure \ref{fig:configured_vs_default_test}.

    \begin{figure}[htbp]
        \noindent
        \begin{centering}
            @@figure-configured-vs-default-test@@
            % \includegraphics[width=0.8\textwidth]{fittedModels}
            \par
        \end{centering}

       \caption{Empirical comparison between the @@solver@@ (configured) and @@solver@@ (default) on the testing set of @@instanceSetTest@@.}\label{fig:configured_vs_default_test}
    \end{figure}

    %\ifruntime

        Table \ref{tbl:timeouts_test} shows on how many instances the @@solver@@ (configured) and @@solver@@ (default) timed out (did not solve the instance within the cutoff time of @@smacEachRunCutoffTime@@ seconds) on the testing set of @@instanceSetTest@@, as well as on how many instances both timed out.

        \begin{table}[htbp]
            \begin{center}
                \begin{tabular}{ccc}
                    configured & default & overlap \\ \hline
                    @@timeoutsTestConfigured@@ & @@timeoutsTestDefault@@ & @@timeoutsTestOverlap@@
                \end{tabular}
            \end{center}
            \caption{Number of time-outs for @@solver@@ (configured), @@solver@@ (default), and for how many instances both timed out on the testing set of @@instanceSetTest@@.}
            \label{tbl:timeouts_test}
        \end{table}

    %\fi % End ifruntime

\fi % End iftest

\ifablation
    \section{Parameter importance via Ablation}

    Ablation analysis~\cite{FawcettHoos16} is performed from the @@solver@@ (default) to @@solver@@ (configured) to see which parameter changes between them contribute most to the improved performance.
    \iftest
    The ablation path uses the training set @@instanceSetTrain@@ and validation is perform on the test set @@instanceSetTest@@.
    \else
    The ablation path is constructed and validated with the training set @@instanceSetTrain@@.
    \fi
    The set of parameters that differ in the two configurations will form the ablation path.
    Starting from the default configuration, the path is computed by performing a sequence of rounds.
    In a round, each available parameter is flipped in the configuration and is validated on its performance.
    The flipped parameter with the best performance in that round, is added to the configuration and the next round starts with the remaining parameters.
    This repeats until all parameters are flipped, which is the best found configuration.
    The analysis resulted in the ablation path presented in Table~\ref{table:ablationpath}.


    \begin{table}[htbp]
        \caption{Ablation path from @@solver@@ (default) to @@solver@@ (configured) where parameters with higher importance are ranked higher.}
        \label{table:ablationpath}
        \begin{center}
        \footnotesize
            @@ablationPath@@
        \end{center}
    \end{table}

\fi % End ablation

\bibliographystyle{plain}
\bibliography{@@bibliographypath@@}

\end{document}
