\documentclass[british]{article}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{geometry}
\geometry{verbose,tmargin=3.5cm,bmargin=3.5cm,lmargin=3cm,rmargin=3cm}
\usepackage{array}
\usepackage{amstext}
\usepackage{graphicx}
\usepackage{color}
\usepackage[font=small,labelfont=bf]{caption}

\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}

\newcommand{\mytablefontsize}{7pt}
\newcommand{\mytablebaselineskip}{0.7}
\newcommand{\mytabcolsep}{3pt}

\newcommand{\medianInterval}[1]{}

\makeatletter

\newif\iftest
@@testBool@@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
%% Because html converters don't know tabularnewline
\providecommand{\tabularnewline}{\\}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.

\title{Algorithm Selection Report}
\author{ \emph{Sparkle} }

\makeatother

\usepackage{babel}
\begin{document}
\maketitle %


\section{Introduction}
\label{sec:Introduction}

\emph{Sparkle} \cite{Hoos15} is a multi-agent problem-solving platform based on Programming by Optimisation (PbO)~\cite{Hoos12}, and would provide a number of effective algorithm optimisation techniques (such as automated algorithm configuration, portfolio-based algorithm selection, etc.) to accelerate the existing solvers.

This experimental report is automatically generated by \emph{Sparkle}. This report presents experimental results for the training instances submitted to \emph{Sparkle}\iftest, as well as for running the \emph{Sparkle} portfolio selector to solve the test instances in the instance set @@testInstanceClass@@\fi.

\section{Experimental Preliminaries}
\label{sec:Experimental_Preliminaries}

This section presents the experimental preliminaries, including the list of solvers, the list of feature extractors, the list of instance sets, information about the experimental setup and information about how to construct a portfolio-based algorithm selector in \emph{Sparkle}.

\subsection{Solvers}
\label{sec:Solvers}
There are @@numSolvers@@ solver(s) included in \emph{Sparkle}, as listed below.

\begin{enumerate} 
@@solverList@@
\end{enumerate}

\subsection{Feature Extractors}
\label{sec:Feature_Extractors}
There are @@numFeatureExtractors@@ feature extractor(s) included in \emph{Sparkle}, as listed below.

\begin{enumerate}
@@featureExtractorList@@
\end{enumerate}

\subsection{Training Instance Set(s)}
\label{sec:Train_Instance_Sets}
There are @@numInstanceClasses@@ instance set(s) included in \emph{Sparkle}, as listed below.

\begin{enumerate}
@@instanceClassList@@
\end{enumerate}

\iftest
\subsection{Test Instance Set}
\label{sec:Test_Instance_Set}

\begin{itemize}
\item @@testInstanceClass@@, consisting of @@numInstanceInTestInstanceClass@@ instances
\end{itemize}
\fi

\subsection{Experimental Setup for Training Phase}
\label{sec:Experimental_Setup_for_Training_Phase}

The experimental setup for the training phase is described below.

\textbf{Feature computation:} \emph{Sparkle} uses all the feature extractors presented above to compute a feature vector for each training instance. Every feature extractor computes a feature vector for each training instance. The final feature vector is then the combination of all computed feature vectors. The cutoff time for feature vector computation on each training instance is set to @@featureComputationCutoffTime@@ seconds. This time is shared equally between the feature extractors.

\textbf{Performance computation:} \emph{Sparkle} runs each solver one time on each training instance. The cutoff time for each performance computation run is set to @@performanceComputationCutoffTime@@ seconds.

\subsection{Constructing the Portfolio-Based Algorithm Selector}
\label{sec:Portfolio}

\emph{Sparkle} saves the results of feature extraction and performance computation described above. This data is then utilised by \emph{Sparkle} to run \emph{AutoFolio}~\cite{LinEtAl15} to automatically construct a portfolio-based per-instance algorithm selector.

\section{Experimental Results on the Training Set(s)}
\label{sec:Experimental_Results_Train}

In this section, the @@metric@@ results for the current portfolio selector in \emph{Sparkle} on solving the training instance set(s) listed in Section~\ref{sec:Train_Instance_Sets} is reported.

\subsection{@@metric@@ Ranking List}
\label{sec:PAR_Ranking}

Below, the solvers are ranked based on the penalised average runtime (@@metric@@). The first ranked solver is also referred to as the single best solver (\emph{SBS}).

\begin{enumerate}
@@PARRankingList@@
\end{enumerate}

Next, the @@metric@@ values for the virtual best solver (\emph{VBS}), i.e., the perfect portfolio selector, and the actual portfolio selector in \emph{Sparkle} are given.

\begin{itemize}
\item \textbf{\emph{VBS}}, @@metric@@: @@VBSPAR@@
\item \textbf{Actual Portfolio Selector in \emph{Sparkle}}, @@metric@@: @@actualPAR@@
\end{itemize}

\subsection{Marginal Contribution Ranking List}
\label{sec:Marginal_Contribution_Ranking}

\emph{Sparkle} uses the concept of marginal contribution~\cite{XuEtAl12} to measure each solver's contribution to the \textbf{\emph{VBS}} and to the \textbf{actual portfolio selector in \emph{Sparkle}}. In this report, the approach described in the literature~\cite{FreEtAl16} is used to compute each solver's marginal contribution.

This resulted in the following ranking of solvers for the marginal contribution to the \textbf{{\em VBS}}:

\begin{enumerate}
@@solverPerfectRankingList@@
\end{enumerate}

And the following ranking for the marginal contribution to the \textbf{actual portfolio selector in \emph{Sparkle}}:

\begin{enumerate} 
@@solverActualRankingList@@
\end{enumerate}


\subsection{Scatter Plot Analysis}

Figure~\ref{fig:sparkle_vs_sbs} shows the empirical comparison between the actual portfolio selector in \emph{Sparkle} and the single best solver (\emph{SBS}).
Figure \ref{fig:sparkle_vs_vbs} shows the empirical comparison between the actual portfolio selector in \emph{Sparkle} and the virtual best solver (\emph{VBS}).

\begin{figure}[htbp]
\noindent \begin{centering}
@@figure-portfolio-selector-sparkle-vs-sbs@@
\par\end{centering}

\caption{Empirical comparison between the actual portfolio selector in \emph{Sparkle} and the \emph{SBS}.}\label{fig:sparkle_vs_sbs}
\end{figure}

\begin{figure}[htbp]
\noindent \begin{centering}
@@figure-portfolio-selector-sparkle-vs-vbs@@
\par\end{centering}

\caption{Empirical comparison between the actual portfolio selector in \emph{Sparkle} and the \emph{VBS}.}\label{fig:sparkle_vs_vbs}
\end{figure}


\iftest
\section{Experimental Results on the Test Set}
\label{sec:Experimental_Results_Test}

In this section, the @@metric@@ results for the current portfolio selector in \emph{Sparkle} on solving the test instance set @@testInstanceClass@@ is reported.

\begin{itemize}
\item \textbf{Actual Portfolio Selector in \emph{Sparkle}}, @@metric@@: @@testActualPAR@@
\end{itemize}
\fi


\bibliographystyle{plain}
\bibliography{@@bibliographypath@@}

\end{document}
