\chapter*{Preface}
\addcontentsline{toc}{chapter}{Preface}

\section*{Why \Stan?}

We%
%
\footnote{In Fall 2010, the ``we'' consisted of Andrew Gelman and his
  crew of Ph.D.\ students (Wei Wang and Vince Dorie), postdocs (Ben
  Goodrich, Matt Hoffman and Michael Malecki), and research staff (Bob
  Carpenter and Daniel Lee).  Previous postdocs whose work directly
  influenced \Stan included Matt Schofield, Kenny Shirley, and Aleks
  Jakulin.  Jiqiang Guo joined as a postdoc in Fall 2011.  Marcus
  Brubaker, a computer science postdoc at Toyota Technical Institute
  at Chicago, joined the development team in early 2012.  Michael
  Betancourt, a physics Ph.D.\ about to start a postdoc at University
  College London, joined the development team in late 2012 after
  months of providing useful feedback on geometry and debugging
  samplers at our meetings. Yuanjun Gao, a statistics graduate student at
  Columbia, and Peter Li, an undergraduate student at Columbia,
  joined the development team in the Fall semester of 2012.}
%
did not set out to build \Stan as it currently exists.  We set out to
apply full Bayesian inference to the sort of multilevel generalized
linear models discussed in Part II of \citep{GelmanHill:2007}.  These
models are structured with grouped and interacted predictors at
multiple levels, hierarchical covariance priors, nonconjugate
coefficient priors, latent effects as in item-response models, and
varying output link functions and distributions.

The models we wanted to fit turned out to be a challenge for current
general-purpose software to fit.  A direct encoding in \BUGS or \JAGS
can grind these tools to a halt.  Matt Schofield found his multilevel
time-series regression of climate on tree-ring measurements wasn't
converging after hundreds of thousands of iterations.

Initially, Aleks Jakulin spent some time working on extending the
Gibbs sampler in the Hierarchical Bayesian Compiler
\citep{DaumeIII:2007}, which as its name suggests, is compiled rather
than interpreted.  But even an efficient and scalable implementation
does not solve the underlying problem that Gibbs sampling does not
fare well with highly correlated posteriors.  We finally realized we
needed a better sampler, not a more efficient implementation.

We briefly considered trying to tune proposals for a random-walk
Metropolis-Hastings sampler, but that seemed too problem specific and
not even necessarily possible without some kind of adaptation rather
than tuning of the proposals. 
 

\section*{The Path to \Stan}

We were at the same time starting to hear more and more about
Hamiltonian Monte Carlo (\HMC) and its ability to overcome some of the
the problems inherent in Gibbs sampling.  Matt Schofield managed to
fit the tree-ring data using a hand-coded implementation of \HMC,
finding it converged in a few hundred iterations.

\HMC appeared promising but was also problematic in that the
Hamiltonian dynamics simulation requires the gradient of the log
posterior.  Although it's possible to do this by hand, it is very
tedious and error prone.  That's when we discovered reverse-mode
algorithmic differentiation, which lets you write down a templated
\Cpp function for the log posterior and automatically compute a proper
analytic gradient up to machine precision accuracy in only a few
multiples of the cost to evaluate the log probability function itself.
We explored existing algorithmic differentiation packages with open
licenses such as {\sc rad} \citep{Gay:2005} and its repackaging in the
Sacado module of the Trilinos toolkit and the {\small CppAD} package in the
{\sc coin-or} toolkit.  But neither package supported very many
special functions (e.g., probability functions, log gamma, inverse logit) or
linear algebra operations (e.g., Cholesky decomposition) and were not
easily and modularly extensible.  

So we built our own reverse-mode algorithmic differentiation package.
But once we'd built our own reverse-mode algorithmic differentiation
package, the problem was that we could not just plug in the
probability functions from a package like Boost because they weren't
templated on all the arguments.  We only needed algorithmic
differentiation variables for parameters, not data or transformed
data, and promotion is very inefficient in both time and memory.  So
we wrote our own fully templated probability functions.  

Next, we integrated the Eigen \Cpp package for matrix operations and
linear algebra functions.  Eigen makes extensive use of expression
templates for lazy evaluation and the curiously recurring template
pattern to implement concepts without virtual function calls.  But we
ran into the same problem with Eigen as with the existing probability
libraries --- it doesn't support mixed operations of algorithmic
differentiation variables and primitives like \code{double}.  This is
a problem we have yet to optimize away as of \Stan version 1.3, but we
have plans to extend Eigen itself to support heterogeneous 
matrix operator types.

At this point (Spring 2011), we were happily fitting models coded 
directly in \Cpp on top of the pre-release versions of the \Stan API. 
Seeing how well this all worked, we set our sights on the generality 
and ease of use of \BUGS.  So we designed a modeling language in which 
statisticians could write their models in familiar notation that could 
be transformed to efficient \Cpp code and then compiled into an 
efficient executable program.
 
The next problem we ran into as we started implementing richer models
is variables with constrained support (e.g., simplexes and covariance
matrices).  Although it is possible to implement \HMC with bouncing
for simple boundary constraints (e.g., positive scale or precision
parameters), it's not so easy with more complex multivariate
constraints.  To get around this problem, we introduced typed
variables and automatically transformed them to unconstrained support
with suitable adjustments to the log probability from the log absolute
Jacobian determinant of the inverse transforms.

Even with the prototype compiler generating models, we still faced a
major hurdle to ease of use.  \HMC requires two tuning parameters
(step size and number of steps) and is very sensitive to how they are
set.  The step size parameter could be tuned during warmup based on
Metropolis rejection rates, but the number of steps was not so easy to
tune while maintaining detailed balance in the sampler.  This led to
the development of the No-U-Turn sampler (\NUTS)
\citep{Hoffman-Gelman:2011, Hoffman-Gelman:2013}, which takes an ever
increasing number of steps until the direction of the simulation turns
around, then uses slice sampling to select a point on the simulated
trajectory.

We thought we were home free at this point.  But when we measured the
speed of some \BUGS examples versus \Stan, we were very disappointed.
The very first example model, Rats, ran more than an order of
magnitude faster in \JAGS than in \Stan.  Rats is a tough test case
because the conjugate priors and lack of posterior correlations make
it an ideal candidate for efficient Gibbs sampling.  But we thought
the efficiency of compilation might compensate for the lack of ideal
fit to the problem.  

We realized we were doing redundant calculations, so we wrote a
vectorized form of the normal distribution for multiple variates with
the same mean and scale, which sped things up a bit. At the same
time, we introduced some simple template metaprograms to remove the
calculation of constant terms in the log probability.  These both
improved speed, but not enough.  Finally, we figured out how to both
vectorize and partially evaluate the gradients of the densities using
a combination of expression templates and metaprogramming.  At this
point, we are within a factor of two or so of a hand-coded gradient
function.

Later, when we were trying to fit a time-series model, we found that
normalizing the data to unit sample mean and variance sped up the fits
by an order of magnitude.  Although \HMC and \NUTS are rotation
invariant (explaining why they can sample effectively from
multivariate densities with high correlations), they are not scale
invariant.  Gibbs sampling, on the other hand, is scale invariant, but
not rotation invariant.

We were still using a unit mass matrix in the simulated Hamiltonian
dynamics.  The last tweak to \Stan before version 1.0 was to estimate
a diagonal mass matrix during warmup; this has since been upgraded to
a full mass matrix in version 1.2.  Both these extensions go a bit
beyond the \NUTS paper on {\it arXiv}.  Using a mass matrix sped up
the unscaled data models by an order of magnitude, though it breaks
the nice theoretical property of rotation invariance.  The full mass
matrix estimation has rotational invariance as well, but scales less
well because of the need to invert the mass matrix once and then do matrix
multiplications every leapfrog step.

\section*{Stan 2}

It's been over a year since the initial release of Stan, and we have
been overjoyed by the quantity and quality of models people are
building with Stan.  We've also been a bit overwhelmed by the volume
of traffic on our user's list and issue tracker.

We've been particularly happy about all the feedback we've gotten
about installation issues as well as bugs in the code and
documentation.  We've been pleasantly surprised at the number of such
requests which have come with solutions in the form of a GitHub pull
request.  That certainly makes our life easy.

As the code base grew and as we became more familiar with it, we came
to realize that it required a major refactoring (see, for example,
\citep{FowlerEtAl:1999} for a nice discussion of refactoring).  So
while the outside hasn't changed dramatically in Stan 2, the inside is
almost totally different in terms of how the HMC samplers are
organized, how the output is analyzed, how the mathematics library is
organized, etc.  

We've also improved our optimization algorithm (BFGS) and its
parameterization.  We've added more compile-time and run-time error
checking for models.  We've added many new functions, including new
matrix functions and new distributions.  We've added some new
parameterizations and managed to vectorize all the univariate
distributions.  We've increased compatibility with a range of C++
compilers.  

We've also tried to fill out the manual to clarify things like array
and vector indexing, programming style, and the I/O and command-line
formats.  Most of these changes are direct results of user-reported
confusions.  So please let us know where we can be clearer or more
fully explain something.

Finally, we've fixed all the bugs which we know about.  It was keeping
up with the latter that really set the development time back,
including bugs that resulted in our having to add more error checking.


\section*{\Stan's Future}

We're not done. There's still an enormous amount of work to do to
improve \Stan.  Our to-do list is in the form of a Wiki on GitHub:
%
\begin{quote}
\url{https://github.com/stan-dev/stan/wiki/To-Do-List}
\end{quote}
%
We are gradually weaning ourselves off of the to-do list in favor of
the GitHub issue tracker (see the next section for a link).

Two major features are on the short-term horizon for us after Stan
2.  The first is a differential equation solver, which will allow
fitting parameters of ordinary differential equations as part of model
building (PKBUGS supplies this functionality for BUGS and it has been
rolled into OpenBUGS).  The second big project is Riemannian manifold
Hamiltonian Monte Carlo (RMHMC).  Both of these projects require us to
put the finishing touches on higher-order automatic differentiation.
We also have a number of smaller projects in the works, including more
improvements to the modeling language itself, such as a way to define
and reuse functions and general matrix and array index slicing.

\section*{You Can Help}

Please let us know if you have comments about this manual or
suggestions for \Stan.  We're especially interested in hearing about
models you've fit or had problems fitting with \Stan.  The best way to
communicate with the \Stan team about user issues is through the
following user's group.
%
\begin{quote}
\url{http://groups.google.com/group/stan-users}
\end{quote}
%
For reporting bugs or requesting features, \Stan's issue tracker is at
the following location.
%
\begin{quote}
\url{https://github.com/stan-dev/stan/issues}
\end{quote}

One of the main reasons \Stan is freedom-respecting, open-source
software%
%
\footnote{See \refappendix{licensing} for more information on Stan's
  licenses and the licenses of the software on which it depends.}
%
is that we love to collaborate.  We're interested in hearing
from you if you'd like to volunteer to get involved on the development
side.  We have all kinds of projects big and small that we haven't had
time to code ourselves.  For developer's issues, we have a separate
group.
%
\begin{quote}
\url{http://groups.google.com/group/stan-dev}
\end{quote}

To contact the project developers off the mailing lists, send email to
\begin{quote}
\href{mailto:stan@mc-stan.org}{\nolinkurl{stan@mc-stan.org}}
\end{quote}

\vspace*{12pt}
\mbox{ } \hfill {\it The \Stan Development Team}
\\
\mbox{ } \hfill \today
