\part{Modeling Language Reference}

\chapter{Execution of a \Stan Program}

\noindent 
This chapter provides a sketch of how a compiled \Stan model is
executed using sampling.  Optimization shares the same data reading
and initialization steps, but then does optimization rather than sampling.

This sketch is elaborated in the following chapters of this part,
which cover variable declarations, expressions, statements, and blocks
in more detail.


\section{Reading and Transforming Data}

The reading and transforming data steps are the same for sampling,
optimization and diagnostics.

\subsection{Read Data}

The first step of execution is to read data into memory.  
For the \Stan model executable, data is read from a file in the dump
format (see \refchapter{dump}).%
%
\footnote{The \Cpp code underlying \Stan is flexible enough to allow
  data to be read from memory or file.  Calls from \R, for instance,
  can be configured to read data from file or directly from \R's
  memory.}
All of the variables declared in the \code{data} block will be read.
If a variable cannot be read, the program will halt with a message
indicating which data variable is missing.

After each variable is read, if it has a declared constraint, the
constraint is validated.  For example, if a variable \code{N} is
declared as \code{int<lower=0>}, after \code{N} is read, it will be tested
to make sure it is greater than or equal to zero.  If a variable
violates its declared constraint, the program will halt with a warning
message indicating which variable contains an illegal value, the value
that was read, and the constraint that was declared.

\subsection{Define Transformed Data}

After data is read into the model, the transformed data variable
statements are executed in order to define the transformed data
variables.  As the statements execute, declared constraints on
variables are not enforced.

After the statements are executed, all declared constraints on
transformed data variables are validated.  If the validation fails,
execution halts and the variable's name, value and constraints are
displayed.

\section{Initialization}

Initialization is the same for sampling, optimization, and diagnosis

\subsection{User-Supplied Initial Values}

If there are user-supplied initial values for parameters, these are
read using the same input mechanism and same file format as data
reads.  Any constraints declared on the parameters are validated for
the initial values.  If a variable's value violates its declared
constraint, the program halts and a diagnostic message is printed.

After being read, initial values are transformed to unconstrained
values that will be used to initialize the sampler. 

\subsection{Random Initial Values}

If there are no user-supplied initial values, the default
initialization strategy is to initialize the unconstrained parameters
directly with values drawn uniformly from the interval $(-2,2)$.  The
bounds of this initialization can be changed but it is always
symmetric around 0. The value of 0 is special in that it represents
the median of the initialization.  An unconstrained value of 0
corresponds to different parameter values depending on the constraints
declared on the parameters.

An unconstrained real does not involve any transform, so an initial
value of 0 for the unconstrained parameters is also a value of 0 for
the constrained parameters.  

For parameters that are bounded below at 0, the initial value of 0 on
the unconstrained scale corresponds to $\exp(0) = 1$ on the
constrained scale.  A value of -2 corresponds to $\exp(-2) = .13$ and
a value of 2 corresponds to $\exp(2) = 7.4$.

For parameters bounded above and below, the initial value of 0 on the
unconstrained scale corresponds to a value at the midpoint of the
constraint interval.  For probability parameters, bounded below by 0
and above by 1, the transform is the inverse logit, so that an initial
unconstrained value of 0 corresonds to a constrained value of 0.5, -2
corresponds to 0.12 and 2 to 0.88.  Bounds other than 0 and 1 are
just scaled and translated.

Simplexes with initial values of 0 on the unconstrained basis
correspond to symmetric values on the constrained values (i.e., each
value is $1/K$ in a $K$-simplex).

Cholesky factors for positive-definite matrices are initialized to 1
on the diagonal and 0 elsewhere;  this is because the diagonal is log
transformed and the below-diagonal values are unconstrained.

The initial values for other parameters can be determined from the
transfrom that is applied.  The transforms are all described in full
detail in \refchapter{variable-transforms}.

\subsection{Zero Initial Values}

The initial values may all be set to 0 on the unconstrained scale.
This can be helpful for diagnosis, and may also be a good starting
point for sampling.  Once a model is running, multiple chains with
more diffuse starting points can help diagnose problems with
convergence; see \refsection{convergence} for more information on
convergence monitoring.





\section{Sampling}

Sampling is based on simulating the Hamiltonian of a particle with a
starting position equal to the current parameter values and an initial
momentum (kinetic energy) generated randomly.  The potential energy at
work on the particle is taken to be the negative log (unnormalized) total
probability function defined by the model.  In the usual approach to
implementing \HMC, the Hamiltonian dynamics of the particle is
simulated using the leapfrog integrator, which discretizes the smooth
path of the particle into a number of small time steps called leapfrog
steps.

\subsection{Leapfrog Steps}

For each leapfrog step, the negative log probability function and its
gradient need to be evaluated at the position corresponding to the
current parameter values (a more detailed sketch is provided in the
next section).  These are used to update the momentum based on the
gradient and the position based on the momentum.

For simple models, only a few leapfrog steps with large step sizes are
needed.  For models with complex posterior geometries, many small
leapfrog steps may be needed to accurately model the path of the
parameters.

If the user specifies the number of leapfrog steps (i.e., chooses to
use standard \HMC), that number of leapfrog steps are simulated.  If
the user has not specified the number of leapfrog steps, the No-U-Turn
sampler (\NUTS) will determine the number of leapfrog steps adaptively
\citep{Hoffman-Gelman:2011, Hoffman-Gelman:2013}.

\subsection{Log Probability and Gradient Calculation}

During each leapfrog step, the log probability function and its
gradient must be calculated.  This is where most of the time in the
\Stan algorithm is spent.  This log probability function, which is
used by the sampling algorithm, is defined over the unconstrained
parameters.

The first step of the calculation requires the inverse transform of
the unconstrained parameter values back to the constrained parameters
in terms of which the model is defined.  There is no error checking
required because the inverse transform is a total function on every point
in whose range satisfies the constraints.

Because the probability statements in the model are defined in terms
of constrained parameters, the log Jacobian of the inverse transform
must be added to the accumulated log probability.

Next, the transformed parameter statements are executed.  After they
complete, any constraints declared for the transformed parameters are
checked.  If the constraints are violated, the model will halt with a
diagnostic error message.

The final step in the log probability function calculation is to
execute the statements defined in the model block.  

As the log probability function executes, it accumulates an in-memory
representation of the expression tree used to calculate the log
probability.  This includes all of the transformed parameter
operations and all of the Jacobian adjustments.  This tree is then
used to evaluate the gradients by propagating partial derivatives
backward along the expression graph.  The gradient calculations
account for the majority of the cycles consumed by a \Stan program.

\subsection{Metropolis Accept/Reject}

A standard Metropolis accept/reject step is required to retain detailed
balance and ensure samples are marginally distributed according to the
probability function defined by the model.  This Metropolis adjustment
is based on comparing log probabilities, here defined by the
Hamiltonian, which is the sum of the potential (negative log
probability) and kinetic (squared momentum) energies.  In theory, the
Hamiltonian is invariant over the path of the particle and rejection
should never occur.  In practice, the probability of rejection is
determined by the accuracy of the leapfrog approximation to the true
trajectory of the parameters.

If step sizes are small, very few updates will be rejected, but many
steps will be required to move the same distance.  If step sizes are
large, more updates will be rejected, but fewer steps will be required
to move the same distance.  Thus a balance between effort and
rejection rate is required.  If the user has not specified a step
size, \Stan will tune the step size during warmup sampling to achieve
a desired rejection rate (thus balancing rejection versus number of
steps).

If the proposal is accepted, the parameters are updated to their new
values.  Otherwise, the sample is the current set of parameter values.


\section{Optimization}

Optimization runs very much like sampling in that it starts by reading
the data and then initializing parameters.  Unlike sampling, it
produces a deterministic output which requires no further analysis
other than to verify that the optimizer itself converged to a
posterior mode.  The output for optimization is also similar to that
for sampling.


\section{Model Diagnostics}

Model diagnostics are like sampling and optimization in that they
depend on a model's data being read and its parameters being
initialized.  The output, at least so far, is just to the command
line.  See \refsection{diagnostics} for information on the
available diagnostics (as of Stan 2.0, just gradients and log
probabilities at the initialization state).


\section{Output}

For each final sample (not counting samples during warmup or samples
that are thinned), there is an output stage of writing the samples.

\subsection{Generated Quantities} 

Before generating any output, the statements in the generated quantities 
block are executed.  This can be used for any forward simulation based
on parameters of the model.  Or it may be used to transform parameters
to an appropriate form for output.  

After the generated quantities statements execute, the constraints
declared on generated quantities variables are validated.   If these
constraints are violated, the program will terminate with a diagnostic message.

\subsection{Write}

The final step is to write the actual values.  The values of all
variables declared as parameters, transformed parameters, or generated
quantities are written.  Local variables are not written, nor is the
data or transformed data.  All values are written in their constrained
forms, that is the form that is used in the model definitions.

In the executable form of a \Stan models, parameters, transformed
parameters, and generated quantities are written to a file in
comma-separated value (\acronym{csv}) notation with a header defining
the names of the parameters (including indices for multivariate
parameters).%
\footnote{In the \R version of \Stan, the values may either be
written to a \acronym{csv} file or directly back to \R's memory.}

% PSEUDOCODE FOR STAN MODEL
% \begin{verbatim}
% read data into memory
% validate data constraints
% execute transformed data statements
% validate transformed data constraints
% read and inverse transform initial parameter values 
%     (OR randomly initialize)
% for each sample desired:
%     if warming up and adapting, adapt step size
%     randomly initialize momentum
%     for each leapfrog step:
%         calculate log probability and gradient
%         increment momentum by half step-size times gradient
%         increment parameters by step-size times momentum
%         calculate log probability and gradient
%         increment momentum by half step-size times gradient
%     if not rejected by Metropolis condition:
%         update parameters to new values
%         execute transformed parameters statement
%         validate transformed parameters constraints
%         write parameters, transformed parameters, generated quantities
% \end{verbatim}



\chapter{Data Types and Variable Declarations}\label{data-types.chapter}

\noindent 
This chapter covers the data types for expressions in \Stan.  Every
variable used in a \Stan program must have a declared data type.  Only
values of that type will be assignable to the variable (except for
temporary states of transformed data and transformed parameter
values).  This follows the convention of programming languages like
\Cpp, not the conventions of scripting languages like Python or
statistical languages such as \R or \BUGS.  

The motivation for strong, static typing is threefold.  
%
\begin{itemize}
\item Strong typing forces the programmer's intent to be declared with
  the variable, making programs easier to comprehend and hence easier
  to debug and maintain.
\item Strong typing allows programming errors relative to the
  declared intent to be caught sooner (at compile time) rather than
  later (at run time).  The \Stan compiler (see \refsection{stanc})
  will flag any type errors and indicate the offending expressions
  quickly when the program is compiled.
\item Constrained types will catch runtime data, initialization, and
  intermediate value errors as soon as they occur rather than allowing
  them to propagate and potentially pollute final results.
\end{itemize}
%
Strong typing disallows assigning the same variable to objects of
different types at different points in the program or in different
invocations of the program.

\section{Overview of Data Types}

\subsection{Basic Data Types}

The primitive \Stan data types are \code{real} for continuous scalar
quantities and \code{int} for integer values.  The compound data
types include \code{vector} (of real values), \code{row\_vector} (of
real values), and \code{matrix} (of real values).

\subsection{Constrained Data Types}

Integer or real types may be constrained with lower bounds, upper
bounds, or both.  There are four constrained vector data types,
\code{simplex} for unit simplexes, \code{unit\_vector} for unit-length
vectors, \code{ordered} for ordered vectors of scalars and
\code{positive\_ordered} for vectors of positive ordered scalars.
There are specialized matrix data types \code{corr\_matrix} and
\code{cov\_matrix} for correlation matrices (symmetric, positive
definite, unit diagonal) and covariance matrices (symmetric, positive
definite).  The type \code{cholesky\_factor\_cov} is for Cholesky
factors of covariance matrices (lower triangular, positive diagonal,
product with own transpose is a covariance matrix).

\subsection{Arrays}

\Stan supports arrays of arbitrary order of any of the basic data
types or constrained basic data types.  This includes
three-dimensional arrays of integers, one-dimensional arrays of
positive reals, four-dimensional arrays of simplexes, one-dimensional
arrays of row vectors, and so on.



\section{Primitive Numerical Data Types}

Unfortunately, the lovely mathematical abstraction of integers and
real numbers is only partially supported by finite-precision computer
arithmetic.  

\subsection{Integers}\label{int-data-type.section}

Stan uses 32-bit (4-byte) integers for all of its integer
representations.  The maximum value that can be represented
as an integer is $2^{31}-1$; the minimum value is $-(2^{31})$.

When integers overflow, their values wrap.  Thus it is up to the \Stan
programmer to make sure the integer values in their programs stay in
range.  In particular, every intermediate expression must have an
integer value that is in range.

Integer arithmetic works in the expected way for addition,
subtraction, and multiplication, but rounds the result of division
(see \refsection{int-arithmetic} for more information).

\subsection{Reals}\label{real-data-type.section}

\Stan uses 64-bit (8-byte) floating point representations of real
numbers.  \Stan roughly%
%
\footnote{\Stan compiles integers to \code{int} and reals to
  \code{double} types in \Cpp.  Precise details of rounding will depend
  on the compiler and hardware architecture on which the code is run.}
%
follows the {\sc ieee} 754 standard for floating-point computation.
The range of a 64-bit number is roughly $\pm 2^{1022}$, which is
slightly larger than $\pm 10^{307}$.  It is a good idea to stay well
away from such extreme values in \Stan models as they are prone to
cause overflow.

64-bit floating point representations have roughly 15 decimal digits
of accuracy.  But when they are combined, the result often has less
accuracy.  In some cases, the difference in accuracy between two
operands and their result is large.

There are three special real values used to represent (1) error
conditions, (2) positive infinity, and (3) negative infinity.  The
error value is referred to as ``not a number.''

\subsection{Promoting Integers to Reals}

\Stan automatically promotes integer values to real values if
necessary, but does not automatically demote real values to integers.
For very large integers, this will cause a rounding error to fewer
significant digits in the floating point representation than in the
integer representation.

Unlike in \Cpp, real values are never demoted to integers.  Therefore,
real values may only be assigned to real variables.  Integer values
may be assigned to either integer variables or real variables.
Internally, the integer representation is cast to a floating-point
representation.  This operation is not without overhead and should
thus be avoided where possible.


\section{Univariate Data Types and Variable Declarations}

All variables used in a \Stan program must have an explicitly declared
data type.  The form of a declaration includes the type and the name
of a variable.  This section covers univariate types, the next section
vector and matrix types, and the following section array types.

\subsection{Unconstrained Integer}

Unconstrained integers are declared using the \code{int} keyword.
For example, the variable \code{N} is declared to be an integer as follows.
%
\begin{quote}
\begin{Verbatim} 
int N;
\end{Verbatim}
\end{quote}
% 

\subsection{Constrained Integer}

Integer data types may be constrained to allow values only in a
specified interval by providing a lower bound, an upper bound, or
both.  For instance, to declare \code{N} to be a positive integer, use
the following.
%
\begin{quote}
\begin{Verbatim}
int<lower=1> N;
\end{Verbatim}
\end{quote}
%
This illustrates that the bounds are inclusive for integers.

To declare an integer variable \code{cond} to take only binary values,
that is zero or one, a lower and upper bound must be provided, as in
the following example.
%
\begin{quote}
\begin{Verbatim} 
int<lower=0,upper=1> cond;
\end{Verbatim}
\end{quote}


\subsection{Unconstrained Real}

Unconstrained real variables are declared using the keyword
\code{real}, The following example declares \code{theta} to be an
unconstrained continuous value.
%
\begin{quote}
\begin{Verbatim}
real theta;
\end{Verbatim}
\end{quote}
%

\subsection{Constrained Real}

Real variables may be bounded using the same syntax as integers.  In
theory (that is, with arbitrary-precision arithmetic), the bounds on
real values would be exclusive.  Unfortunately, finite-precision
arithmetic rounding errors will often lead to values on the
boundaries, so they are allowed in \Stan.
 
The variable \code{sigma} may be declared to be non-negative as follows.
%
\begin{quote}
\begin{Verbatim}
real<lower=0> sigma;
\end{Verbatim}
\end{quote}
%
The following declares the variable \code{x} to be less than or equal
to $-1$.
%
\begin{quote}
\begin{Verbatim} 
real<upper=-1> x;
\end{Verbatim}
\end{quote}
% 
To ensure \code{rho} takes on values between $-1$ and $1$, use the
following declaration.
%
\begin{quote}
\begin{Verbatim}
real<lower=-1,upper=1> rho;
\end{Verbatim}
\end{quote}
%

\subsubsection{Infinite Constraints}

Lower bounds that are negative infinity or upper bounds that are
positive infinity are ignored.  Stan provides constants
\code{positive\_infinity()} and \code{negative\_infinity()} which may
be used for this purpose, or they may be read as data in the dump
format.  


\subsection{Expressions as Bounds}

Bounds for integer or real variables may be arbitrary expressions.
The only requirement is that they only include variables that have
been defined before the declaration.  If the bounds themselves are
parameters, the behind-the-scenes variable transform accounts for them
in the log Jacobian.  

For example, it is acceptable to have the
following declarations.
%
\begin{quote}
\begin{Verbatim}
data { 
 real lb;
}
parameters {
   real<lower=lb> phi;
}
\end{Verbatim}
\end{quote}
%
This declares a real-valued parameter \code{phi} to take values
greater than the value of the real-valued data variable \code{lb}.
Constraints may be complex expressions, but must be of type \code{int}
for integer variables and of type \code{real} for real variables
(including constraints on vectors, row vectors, and matrices).
Variables used in constraints can be any variable that has been
defined at the point the constraint is used.  For instance,
\begin{quote}
\begin{Verbatim}
data { 
   int<lower=1> N;
   real y[N];
}
parameters {
   real<lower=fmin(y),upper=fmax(y)> phi;
}
\end{Verbatim}
\end{quote}
%
This declares a positive integer data variable \code{N}, an array
\code{y} of real-valued data of length \code{N}, and then a parameter
ranging between the minimum and maximum value of \code{y}.  The
functions \code{fmin()} and \code{fmax()} are minimum and maximum
functions for floating point quantities.


\section{Vector and Matrix Data Types}

\subsection{Values}

Vectors, row vectors, and matrices contain real values.  Arrays, on
the other hand, may contain any kind of value, including integers and
structured values like vectors.


\subsection{Indexing}

Vectors and matrices, as well as arrays, are indexed starting from one
in \Stan.  This follows the convention in statistics and linear
algebra as well as their implementations in the statistical software
packages \R, \MATLAB, \BUGS, and \JAGS.  General computer programming
languages, on the other hand, such as \Cpp and Python, index arrays
starting from zero.

\subsection{Vectors}

Vectors in \Stan are column vectors; see the next subsection for
information on row vectors.  Vectors are declared with a size (i.e., a
dimensionality).  For example, a 3-dimensional vector is declared with
the keyword \code{vector}, as follows.
%
\begin{quote}
\begin{Verbatim}
vector[3] u;
\end{Verbatim}
\end{quote}
%
Vectors may also be declared with constraints, as in the following
declaration of a 3-vector of non-negative values.
%
\begin{quote}
\begin{Verbatim}
vector<lower=0>[3] u;
\end{Verbatim}
\end{quote}
%

\subsection{Unit Simplexes}

A unit simplex is a vector with non-negative values whose entries sum
to 1.  For instance, $(0.2,0.3,0.4,0.1)^{\top}$ is a unit 4-simplex.
Unit simplexes are most often used as parameters in categorical
or multinomial distributions, and they are also the sampled variate in
a Dirichlet distribution.  Simplexes are declared with their full
dimensionality.  For instance, \code{theta} is declared to
be a unit $5$-simplex by
%
\begin{quote}
\begin{Verbatim} 
simplex[5] theta;
\end{Verbatim}
\end{quote}
% 

Unit simplexes are implemented as vectors and may be assigned to other
vectors and vice-versa.  Simplex variables, like other constrained
variables, are validated to ensure they contain simplex values; for
simplexes, this is only done up to a statically specified accuracy
threshold $\epsilon$ to account for errors arising from floating-point
imprecision.

\subsection{Unit Vectors}

A unit vector is a vector with a norm of one.  For instance,
$(0.5,0.5,0.5,0.5)^{\top}$ is a unit 4-vector.
Unit vectors are sometimes used in directional statistics.
Unit vectors are declared with their full
dimensionality.  For instance, \code{theta} is declared to
be a unit $5$-vector by
%
\begin{quote}
\begin{Verbatim} 
unit_vector[5] theta;
\end{Verbatim}
\end{quote}
% 
Unit vectors are implemented as vectors and may be assigned to other
vectors and vice-versa.  Unit vector variables, like other constrained
variables, are validated to ensure that they are indeed unit length; for
unit vectors, this is only done up to a statically specified accuracy
threshold $\epsilon$ to account for errors arising from floating-point
imprecision.

\subsection{Ordered Vectors}

An ordered vector type in \Stan represents a vector whose entries are
sorted in ascending order.  For instance, $(-1.3,2.7,2.71)^{\top}$ is
an ordered 3-vector.  Ordered vectors are most often employed as cut
points in ordered logistic regression models (see
\refsection{ordered-logistic}).

The variable \code{c} is declared as an ordered 5-vector by
%
\begin{quote}
\begin{Verbatim}
ordered[5] c;
\end{Verbatim}
\end{quote}
%
After their declaration, ordered vectors, like unit simplexes, may be 
assigned to other vectors and other vectors may be assigned to them. 
Constraints will be checked after executing the block in which the 
variables were declared.  


\subsection{Positive, Ordered Vectors}

There is also a positive, ordered vector type which operates similarly
to ordered vectors, but all entries are constrained to be positive.
For instance, $(2,3.7,4,12.9)$ is a positive, ordered 4-vector.

The variable \code{d} is declared as a positive, ordered 5-vector by
%
\begin{quote}
\begin{Verbatim}
positive_ordered[5] d;
\end{Verbatim}
\end{quote}
%
Like ordered vectors, after their declaration positive ordered vectors
assigned to other vectors and other vectors may be assigned to them. 
Constraints will be checked after executing the block in which the 
variables were declared.  

\subsection{Row Vectors}

Row vectors are declared with the keyword \code{row\_vector}.
Like (column) vectors, they are declared with a size.  For example,
a 1093-dimensional row vector \code{u} would be declared as
%
\begin{quote}
\begin{Verbatim}
row_vector[1093] u;
\end{Verbatim}
\end{quote}
%
Constraints are declared as for vectors, as in the following example
of a 10-vector with values between -1 and 1.
\begin{quote}
\begin{Verbatim}
row_vector<lower=-1,upper=1>[10] u;
\end{Verbatim}
\end{quote}
%

Row vectors may not be assigned to column vectors, nor may column
vectors be assigned to row vectors.  If assignments are required, they
may be accommodated through the transposition operator.

\subsection{Matrices}

Matrices are declared with the keyword \code{matrix} along with a
number of rows and number of columns.  For example, 
%
\begin{quote}
\begin{Verbatim}  
matrix[3,3] A;  
matrix[M,N] B;
\end{Verbatim}
\end{quote}
%  
declares \code{A} to be a $3 \times 3$ matrix and \code{B} to be a $M
\times N$ matrix.  For the second declaration to be well formed, the
variables \code{M} and \code{N} must be declared as integers in either
the data or transformed data block and before the matrix declaration.

Matrices may also be declared with constraints, as in this ($3 \times $4)
matrix of non-positive values.
%
\begin{quote}
\begin{Verbatim}  
matrix<upper=0>[3,4] B;
\end{Verbatim}
\end{quote}
%  

\subsubsection{Assigning to Rows of a Matrix}

Rows of a matrix can be assigned by indexing the left-hand side of an
assignment statement. For example, this is possible.
%
\begin{quote}
\begin{Verbatim}
matrix[M,N] a;
row_vector[N] b;
...
a[1] <- b;
\end{Verbatim}
\end{quote}
%
This copies the values from row vector \code{b} to \code{a[1]}, which
is the first row of the matrix \code{a}.  If the number of columns in
\code{a} is not the same as the size of \code{b}, a run-time error is
raised;  the number of rows of \code{a} is \code{N}, which is also the
size of \code{b}.

Assignment works by copying values in Stan.  That means any subsequent
assignment to \code{a[1]} does not affect \code{b}, nor does an
assignment to \code{b} affect \code{a}.


\subsection{Correlation Matrices}

Matrix variables may be constrained to represent correlation matrices.
A matrix is a correlation matrix if it is symmetric and positive
definite, has entries between $-1$ and $1$, and has a unit diagonal.
Because correlation matrices are square, only one dimension needs
to be declared.  For example,
%
\begin{quote}
\begin{Verbatim} 
corr_matrix[3] Sigma;
\end{Verbatim}
\end{quote}
% 
declares \code{Sigma} to be a $3 \times 3$ correlation matrix.

Correlation matrices may be assigned to other matrices, including
unconstrained matrices, if their dimensions match, and vice-versa.

\subsection{Covariance Matrices}

Matrix variables may be constrained to represent covariance matrices.
A matrix is a covariance matrix if it is symmetric and positive
definite.  Like correlation matrices, covariance matrices only need a
single dimension in their declaration.  For instance,
%
\begin{quote}
\begin{Verbatim} 
cov_matrix[K] Omega;
\end{Verbatim}
\end{quote}
% 
declares \code{Omega} to be a $K \times K$ covariance matrix, where
$K$ is the value of the data variable \code{K}.  

\subsection{Cholesky Factors of Covariance Matrices}

Matrix variables may be constrained to represent the Cholesky factors
of a covariance matrix.  This is often more convenient or more
efficient than representing covariance matrices directly.  

A Cholesky factor $L$ is an $M \times N$ lower-triangular matrix (if
$m < n$ then $L[m,n] =0$) with a positive diagonal ($L[k,k] = 0$) and
$M \geq N$.  If $L$ is a Cholesky factor, then $\Sigma = L \, L^{\top}$
is a covariance matrix.  Furthermore, every covariance matrix has a
Cholesky factorization.

The typical case of a square Cholesky factor may be declared with a
single dimension,
%
\begin{quote}
\begin{Verbatim}
cholesky_factor_cov[4] L;
\end{Verbatim}
\end{quote}
%
In general, two dimensions may be declared, with the above being equal to 
\code{cholesky\_factor\_cov[4,4]}.  The
type \code{cholesky\_factor\_cov[M,N]} may be used for the general 
$M \times N$.

\subsection{Assigning Constrained Variables}

Constrained variables of all types may be assigned to other variables
of the same unconstrained type and vice-versa.  For instance, a
variable declared to be \code{real<lower=0,upper=1>} could be assigned
to a variable declared as \code{real} and vice-versa.  Similarly, a
variable declared as \code{matrix[3,3]} may be assigned to a variable
declared as \code{cov\_matrix[3]} or
\code{cholesky\_factor\_cov[3]}, and vice-versa.

Checks are carried out at the end of each relevant block of statements
to ensure constraints are enforced.  This includes run-time size
checks.  The \Stan compiler isn't able to catch the fact that an
attempt may be made to assign a matrix of one dimensionality to a
matrix of mismatching dimensionality.  


\subsection{Expressions as Size Declarations}

Variables may be declared with sizes given by expressions.  Such
expressions are constrained to only contain data or transformed data
variables.  This ensures that all sizes are determined once the data
is read in and transformed data variables defined by their statements.
For example, the following is legal.
%
\begin{quote}
\begin{Verbatim}
data {
  int<lower=0> N_observed;    int<lower=0> N_missing;
  ...
transformed parameters {
  vector[N_observed + N_missing] y;
  ...
\end{Verbatim}
\end{quote}

\subsection{Accessing Vector and Matrix Elements}

If \code{v} is a column vector or row vector, then \code{v[2]} is the
second element in the vector.  If \code{m} is a matrix, then
\code{m[2,3]} is the value in the second row and third column.

Providing a matrix with a single index returns the specified row.  For
instance, if \code{m} is a matrix, then \code{m[2]} is the second row.
This allows \Stan blocks such as
%
\begin{quote}
\begin{Verbatim} 
matrix[M,N] m;    
row_vector[N] v;    
real x;
...
v <- m[2];   
x <- v[3];   // x == m[2][3] == m[2,3]
\end{Verbatim}
\end{quote}
% 
The type of \code{m[2]} is \code{row\_vector} because it is the second
row of \code{m}.  Thus it is possible to write \code{m[2][3]} instead
of \code{m[2,3]} to access the third element in the second row.  When
given a choice, the form \code{m[2,3]} is preferred.%
%
\footnote{As of \Stan version 1.0, the form
  \code{m[2,3]} is more efficient because it does not require the
  creation and use of an intermediate expression template for
  \code{m[2]}.  In later versions, explicit calls to \code{m[2][3]}
  may be optimized to be as efficient as \code{m[2,3]} by the \Stan
  compiler.\label{array-index-style.footnote}}


\section{Array Data Types}

\Stan supports arrays of arbitrary dimension.  An array's elements may
be any of the basic data types, that is univariate integers,
univariate reals, vectors, row vectors matrices, including all of the
constrained forms.

\subsection{Declaring Array Variables}

Arrays are declared by enclosing the dimensions in square brackets
following the name of the variable.

The variable \code{n} is declared as an array of five integers as follows.
%
\begin{quote}
\begin{Verbatim}  
int n[5];
\end{Verbatim}
\end{quote}
% 
A two-dimensional array of real values with three rows and four columns is
declared with the following.
%
\begin{quote}
\begin{Verbatim}  
real a[3,4];
\end{Verbatim}
\end{quote}
% 
A three-dimensional array \code{z} of positive reals with five rows, four
columns, and two shelves can be declared as follows.
%
\begin{quote}
\begin{Verbatim} 
real<lower=0> z[5,4,2];
\end{Verbatim}
\end{quote}
%

Arrays may also be declared to contain vectors.  For example,
%
\begin{quote}
\begin{Verbatim}  
vector[7] mu[3];
\end{Verbatim}
\end{quote}
% 
declares \code{mu} to be a 3-dimensional array of 7-vectors.  
Arrays may also contain matrices.  The example
%
\begin{quote}
\begin{Verbatim} 
matrix[7,2] mu[15,12];
\end{Verbatim}
\end{quote}
%
declares a $15 \times 12$-dimensional array of $7 \times 2$ matrices.
Any of the constrained types may also be used in arrays, as in the
declaration
%
\begin{quote}
\begin{Verbatim}  
cholesky_factor_cov[5,6] mu[2,3,4];
\end{Verbatim}
\end{quote}
% 
of a $2 \times 3 \times 4$ array of $5 \times 6$ Cholesky factors of
covariance matrices.

\subsection{Accessing Array Elements and Subarrays}

If \code{x} is a 1-dimensional array of length 5, then \code{x[1]} is
the first element in the array and \code{x[5]} is the last.  For a $3
\times 4$ array \code{y} of two dimensions, \code{y[1,1]} is the first
element and \code{y[3,4]} the last element.  For a three-dimensional
array \code{z}, the first element is \code{z[1,1,1]}, and so on.

Subarrays of arrays may be accessed by providing fewer than the full
number of indexes.  For example, suppose \code{y} is a two-dimensional
array with three rows and four columns.  Then \code{y[3]} is
one-dimensional array of length four.  This means that \code{y[3][1]}
may be used instead of \code{y[3,1]} to access the value of the first
column of the third row of \code{y}.  The form \code{y[3,1]} is the
preferred form (see \refnote{array-index-style} in this chapter).

Subarrays may be manipulated and assigned just like any other
variables.  Similar to the behavior of matrices, \Stan allows blocks
such as 
%
\begin{quote}
\begin{Verbatim} 
real w[9,10,11];
real x[10,11];
real y[11];
real z;
...
x <- w[5];
y <- x[4];  // y == w[5][4] == w[5,4]
z <- y[3];  // z == w[5][4][3] == w[5,4,3]
\end{Verbatim}
\end{quote}
%

\subsection{Assigning}

---fixme---

\subsection{Arrays of Matrices and Vectors}\label{arrays-of-matrices.section}

Arrays of vectors and matrices are accessed in the same way as arrays
of doubles.  Consider the following vector and scalar declarations.
%
\begin{quote}
\begin{Verbatim}
vector[5] a[4,3];
vector[5] b[4];
vector[5] c;
real x;
\end{Verbatim}
\end{quote}
%
With these declarations, the following assignments are legal.
%
\begin{quote}
\begin{Verbatim}
b <- a[1];      // result is array of vectors
c <- a[1,3];    // result is vector
c <- b[3];      //   same result as above
x <- a[1,3,5];  // result is scalar
x <- b[3,5];    //   same result as above
x <- c[5];      //   same result as above
\end{Verbatim}
\end{quote}
%
Row vectors and other derived vector types (simplex and ordered)
behave the same way in terms of indexing.

Consider the following matrix, vector and scalar declarations.
%
\begin{quote}
\begin{Verbatim}
matrix[6,5] d[3,4];
matrix[6,5] e[4];
matrix[6,5] f;
row_vector[5] g;
real x;
\end{Verbatim}
\end{quote}
%
With these declarations, the following definitions are legal.
%
\begin{quote}
\begin{Verbatim}
e <- d[1];        // result is array of matrices
f <- d[1,3];      // result is matrix
f <- e[3];        //   same result as above
g <- d[1,3,2];    // result is row vector
g <- e[3,2];      //   same result as above
g <- f[2];        //   same result as above
x <- d[1,3,5,2];  // result is scalar
x <- e[3,5,2];    //   same result as above
x <- f[5,2];      //   same result as above
x <- g[2];        //   same result as above
\end{Verbatim}
\end{quote}
%
As shown, the result \code{f[2]} of supplying a single index to a
matrix is the indexed row, here row 2 of matrix \code{f}.


\subsection{Partial Array Assignment}

Subarrays of arrays may be assigned by indexing on the left-hand side
of an assignment statement.  For example, the following is legal.
%
\begin{quote}
\begin{Verbatim}
real x[I,J,K];
real y[J,K];
real z[K];
...
x[1] <- y;
x[1,1] <- z;
\end{Verbatim}
\end{quote}
%
The sizes must match.  Here, \code{x[1]} is a \code{J} by \code{K}
array, as is is \code{y}.  

Partial array assignment also works for arrays of matrices, vectors,
and row vectors.  








\subsection{Mixing Array, Vector, and Matrix Types}

Arrays, row vectors, column vectors and matrices are not
interchangeable in Stan.  Thus a variable of any one of these
fundamental types is not assignable to any of the others, nor may it
be used as an argument where the other is required (use as arguments
follows the assignment rules).



\subsubsection{Mixing Vectors and Arrays}

For example, vectors cannot be assigned to arrays or vice-versa.
%
\begin{quote}
\begin{Verbatim}
real a[4];
vector b[4];
row_vector c[4];
...
a <- b; // illegal assignment of vector to array
b <- a; // illegal assignment of array to vector
a <- c; // illegal assignment of row vector to array
c <- a; // illegal assignment of array to row vector
\end{Verbatim}
\end{quote}

\subsubsection{Mixing Row and Column Vectors}

It is not even legal to assign row vectors to column vectors or vice
versa.  
\begin{quote}
\begin{Verbatim}
vector b[4];
row_vector c[4];
...
b <- c; // illegal assignment of row vector to column vector
c <- b; // illegal assignment of column vector to row vector
\end{Verbatim}
\end{quote}
%

\subsubsection{Mixing Matrices and Arrays}

The same holds for matrices, where 2-dimensional arrays may not be
assigned to matrices or vice-versa.

\begin{quote}
\begin{Verbatim}
real a[3,4];
matrix[3,4] b;
...
a <- b;  // illegal assignment of matrix to array
b <- a;  // illegal assignment of array to matrix
\end{Verbatim}
\end{quote}
%

\subsubsection{Mixing Matrices and Vectors}

A $1 \times N$ matrix cannot be assigned a row vector or
vice versa.  
%
\begin{quote}
\begin{Verbatim}
matrix[1,4] a;
row_vector[4] b;
...
a <- b;  // illegal assignment of row vector to matrix
b <- a;  // illegal assignment of matrix to row vector
\end{Verbatim}
\end{quote}
%
Similarly, an $M \times 1$ matrix may not be assigned to a column vector.
%
\begin{quote}
\begin{Verbatim}
matrix[4,1] a;
vector[4] b;
...
a <- b;  // illegal assignment of column vector to matrix
b <- a;  // illegal assignment of matrix to column vector
\end{Verbatim}
\end{quote}



\section{Variable Types vs.\ Constraints and Sizes}

The type information associated with a variable only contains the
underlying type and dimensionality of the variable.  

\subsection{Type Information Excludes Sizes}

The size associated with a given variable is not part of its data
type.  For example, declaring a variable using
\begin{quote}
\begin{Verbatim}
real a[3];
\end{Verbatim}
\end{quote}
%
declares the variable \code{a} to be an array.  The fact that it was
declared to have size 3 is part of its declaration, but not part of
its underlying type.  

\subsubsection{When are Sizes Checked?}

Sizes are determined dynamically (at run time) and thus cannot be
type-checked statically when the program is compiled.  As a result,
any conformance error on size will raise a run-time error.  For
example, trying to assign an array of size 5 to an array of size 6
will cause a run-time error.  Similarly, multiplying an $N
\times M$ by a $J \times K$ matrix will raise a run-time error if $M
\neq J$.

\subsection{Type Information Excludes Constraints}

Like sizes, constraints are not treated as part of a variable's type
in Stan when it comes to the compile-time check of operations it may
participate in.  Anywhere Stan accepts a matrix as an argument, it
will syntactically accept a correlation matrix or covariance matrix or
Cholesky factor.  Thus a covariance matrix may be assigned to a matrix
and vice-versa.

Similarly, a bounded real may be assigned to an unconstrained real and
vice-versa.  

\subsubsection{When are Function Argument Constraints Checked?}

For arguments to functions, constraints are sometimes, but not always
checked when the function is called.  Exclusions include \Cpp standard
library functions.  All probabiliy functions and cumulative
distribution functions check that their arguments are appropriate at
run time as the function is called.  

\subsubsection{When are Declared Variable Constraints Checked?}

For data variables, constraints are checked after the variable is read
from a data file or other source.  For transformed data variables, the
check is done after the statements in the transformed data block have
executed.  Thus it is legal for intermediate values of variables to
not satisfy declared constraints.

For parameters, constraints are enforced by the transform applied and
do not need to be checked.  For transformed parameters, the check is
done after the statements in the transformed parameter block have
executed.  

For generated quantities, constraints are enforced after the
statements in the generated quantities block have executed.


\subsection{Type Naming Notation}

In order to refer to data types, it is convenient to have a way to
refer to them.  The type naming notation outlined in this section is
not part of the \Stan programming language, but rather a convention
adopted in this document to enable a concise description of a type.

Because size information is not part of a data type, data
types will be written without size information.  For instance,
\code{real[]} is the type of one-dimensional array of reals and
\code{matrix} is the type of matrices.  The three-dimensional integer
array type is written as \code{int[\, , \, ,]}, indicating the number slots
available for indexing.  Similarly, \code{vector[,]} is the type of a
two-dimensional array of vectors.


\chapter{Expressions}

\noindent 
An expression is the basic syntactic unit in a \Stan program that
denotes a value.  Every expression in a well-formed \Stan program has
a type that is determined statically (at compile time).  If an
expressions type cannot be determined statically, the \Stan compiler
(see \refsection{stanc}) will report the location of the problem.

This chapter covers the syntax, typing, and usage of the various forms
of expressions in \Stan. 

\section{Numeric Literals}

The simplest form of expression is a literal that denotes a primitive
numerical value.   

\subsection{Integer Literals}

Integer literals represent integers of type \code{int}.  Integer
literals are written in base 10 without any separators.  Integer
literals may contain a single negative sign.  (The expression
\code{{-}-1} is interpreted as the negation of the literal \code{-1}.)

The following list contains well-formed integer literals.
%
\begin{quote}
\code{0}, \ \code{1}, \ \code{-1}, \ \code{256}, 
\ \code{-127098}, \ \code{24567898765}
\end{quote}
%
Integer literals must have values that fall within the bounds for
integer values (see \refsection{int-data-type}).

Integer literals may not contain decimal points (\code{.}).  Thus the
expressions \code{1.} and \code{1.0} are of type \code{real} and may
not be used where a value of type \code{int} is required.

\subsection{Real Literals}

A number written with a period or with scientific notation is assigned
to a the continuous numeric type \code{real}.  Real literals are
written in base 10 with a period (\code{.}) as a separator.  Examples
of well-formed real literals include the following.
%
\begin{quote}
\code{0.0}, \ \code{1.0}, \ \code{3.14}, \ \code{-217.9387}, \ 
\code{2.7e3}, \ \code{-2E-5}
\end{quote}
%
The notation \code{e} or \code{E} followed by a positive or negative
integer denotes a power of 10 to multiply.  For instance, \code{2.7e3}
denotes $2.7 \times 10^3$ and \code{-2E-5} denotes $-2 \times
10^{-5}$.


\section{Variables}

A variable by itself is a well-formed expression of the same type as
the variable.  Variables in \Stan consist of \ASCII strings containing
only the basic lower-case and upper-case Roman letters, digits, and
the underscore (\code{\_}) character.  Variables must start with a
letter (\code{a--z} and \code{A--Z}) and may not end with two underscores
(\code{\_\_}).

Examples of legal variable identifiers are as follows.
%
\begin{quote}
\code{a}, 
\ \code{a3}, 
\ \code{a\_3},
\ \code{Sigma}, 
\ \code{my\_cpp\_style\_variable},
\ \code{myCamelCaseVariable}
\end{quote}
%
Unlike in \R and \BUGS, variable identifiers in \Stan may not contain
a period character.  

\subsection{Reserved Names}

Stan reserves many strings for internal use and these may not be used
as the name of a variable.  An attempt to name a variable after an
internal string results in the \code{stanc} translator halting with an
error message indicating which reserved name was used and its location
in the model code.

\subsubsection{Model Name}

The name of the model cannot be used as a variable within the model.
This is usually not a problem because the default in \code{bin/stanc}
is to append \code{\_model} to the name of the file containing the
model specification.  For example, if the model is in file
\code{foo.stan}, it would not be legal to have a variable named
\code{foo\_model} when using the default model name through
\code{bin/stanc}.  With user-specified model names, variables cannot
match the model.

\subsubsection{Reserved Words from Stan Language}

The following list contains reserved words for \Stan's programming
language.  Not all of these features are implemented in Stan yet, but
the tokens are reserved for future use.
%
\begin{quote}
\code{for},
\code{in},
\code{while},
\code{repeat},
\code{until},
\code{if},
\code{then},
\code{else},
\code{true}, 
\code{false}
\end{quote}
%
Variables should not be named after types, either, and thus may not be
any of the following.
%
\begin{quote}
\code{int},
\code{real},
\code{vector},
\code{simplex},
\code{unit\_vector},
\code{ordered},
\code{positive\_ordered},
\code{row\_vector},
\code{matrix},
\code{cholesky\_factor\_cov},
\code{corr\_matrix},
\code{cov\_matrix}.
\end{quote}
%
Variable names will {\it not}\ conflict with the following block identifiers,
%
\begin{quote}
\code{model},
\code{data},
\code{parameters},
\code{quantities},
\code{transformed}, 
\code{generated},
\end{quote}
%

\subsubsection{Reserved Names from Stan Implementation}

Some variable names are reserved because they are used within
Stan's \Cpp implementation.  These are
%
\begin{quote}
var
fvar
\end{quote}
%

\subsubsection{Reserved Function and Distribution Names}

Variable names will conflict with the names of predefined functions
other than constants.  Thus a variable may not be named \code{logit}
or \code{add}, but it may be named \code{pi} or \code{e}.

Variable names will also conflict with the names of distributions
suffixed with \code{\_log}, \code{\_cdf}, \code{\_cdf\_log},
and \code{\_ccdf\_log}, such as \code{normal\_cdf\_log}.

Using any of these variable names causes the \code{stanc} translator
to halt and report the name and location of the variable causing the
conflict.


\subsubsection{Reserved Names from C++}

Finally, variable names, including the names of models, should not
conflict with any of the C++ keywords.
%
\begin{quote}
\code{alignas},
\code{alignof},
\code{and},
\code{and\_eq},
\code{asm},
\code{auto},
\code{bitand},
\code{bitor},
\code{bool},
\code{break},
\code{case},
\code{catch},
\code{char},
\code{char16\_t},
\code{char32\_t},
\code{class},
\code{compl},
\code{const},
\code{constexpr},
\code{const\_cast},
\code{continue},
\code{decltype},
\code{default},
\code{delete},
\code{do},
\code{double},
\code{dynamic\_cast},
\code{else},
\code{enum},
\code{explicit},
\code{export},
\code{extern},
\code{false},
\code{float},
\code{for},
\code{friend},
\code{goto},
\code{if},
\code{inline},
\code{int},
\code{long},
\code{mutable},
\code{namespace},
\code{new},
\code{noexcept},
\code{not},
\code{not\_eq},
\code{nullptr},
\code{operator},
\code{or},
\code{or\_eq},
\code{private},
\code{protected},
\code{public},
\code{register},
\code{reinterpret\_cast},
\code{return},
\code{short},
\code{signed},
\code{sizeof},
\code{static},
\code{static\_assert},
\code{static\_cast},
\code{struct},
\code{switch},
\code{template},
\code{this},
\code{thread\_local},
\code{throw},
\code{true},
\code{try},
\code{typedef},
\code{typeid},
\code{typename},
\code{union},
\code{unsigned},
\code{using},
\code{virtual},
\code{void},
\code{volatile},
\code{wchar\_t},
\code{while},
\code{xor},
\code{xor\_eq}
\end{quote}

\subsection{Legal Characters}

The legal variable characters have the same \ASCII code points in the
range 0--127 as in Unicode.
%
\begin{center}
\begin{tabular}{cc}
Characters  & \ASCII (Unicode) Code Points
\\ \hline
\code{a -- z} & \code{{}~97 -- 122}
\\
\code{A -- Z} & \code{{}~65 -- {}~90}
\\
\code{0 -- 9} & \code{{}~48 -- {}~57}\
\\
\code{\_} & \code{95}
\end{tabular}
\end{center}
%
Although not the most expressive character set, \ASCII is the most
portable and least prone to corruption through improper character
encodings or decodings.

\subsubsection{Comments Allow ASCII-Compatible Encoding}

Within comments, Stan can work with any ASCII-compatible character
encoding, such as ASCII itself, UTF-8, or Latin1.  It is up to user
shells and editors to display them properly.  


\section{Parentheses for Grouping}

Any expression wrapped in parentheses is also an expression. Like in
\Cpp, but unlike in \R, only the round parentheses, \code{(} and
\code{)}, are allowed.  The square brackets \code{[} and \code{]} are
reserved for array indexing and the curly braces \code{\{} and
\code{\}} for grouping statements.

With parentheses it is possible to explicitly group subexpressions
with operators.  Without parentheses, the expression \code{1 + 2 * 3}
has a subexpression \code{2 * 3} and evaluates to 7.  With
parentheses, this grouping may be made explicit with the expression
\code{1 + (2 * 3)}.  More importantly, the expression \code{(1 + 2) *
  3} has \code{1 + 2} as a subexpression and evaluates to 9.


\section{Arithmetic and Matrix Expressions}

For integer and real-valued expressions, \Stan supports the basic
binary arithmetic operations of addition (\code{+}), subtraction
(\code{-}), multiplication (\code{*}) and division (\code{/}) in the
usual ways.  \Stan also supports the unary operation of negation for
integer and real-valued expressions.  For example, assuming \code{n}
and \code{m} are integer variables and \code{x} and \code{y} real
variables, the following expressions are legal.
\begin{quote}
\code{3.0 + 0.14}, 
\ \ \code{-15},
\ \ \code{2 * 3 + 1}, 
\ \ \code{(x - y) / 2.0},
\\
\ \ \code{(n * (n + 1)) / 2},
\ \ \code{x / n}
\end{quote}
%
The negation, addition, subtraction, and multiplication operations are
extended to matrices, vectors, and row vectors.  The transpose
operation, written using an apostrophe (\code{'}) is also supported
for vectors, row vectors, and matrices.  Return types for matrix
operations are the smallest types that can be statically guaranteed to
contain the result.  The full set of allowable input types and
corresponding return types is detailed in
\refchapter{matrix-operations}.

For example, if \code{y} and \code{mu} are variables of type
\code{vector} and \code{Sigma} is a variable of type \code{matrix},
then
%
\begin{quote}
\code{(y - mu)' * Sigma * (y - mu)}
\end{quote}
%
is a well-formed expression of type \code{real}.  The type of the
complete expression is inferred working outward from the
subexpressions.  The subexpression(s) \code{y - mu} are of type
\code{vector} because the variables \code{y} and \code{mu} are of type
\code{vector}.  The transpose of this expression, the subexpression
\code{(y - mu)'} is of type \code{row\_vector}.  Multiplication is
left associative and transpose has higher precedence than
multiplication, so the above expression is equivalent to the following
well-formed, fully specified form.
%
\begin{quote}
\code{(((y - mu)') * Sigma) * (y - mu)}
\end{quote}
%
The type of subexpression \code{(y - mu)' * Sigma} is inferred to be
\code{row\_vector}, being the result of multiplying a row vector by a
matrix.  The whole expression's type is thus the type of a row vector
multiplied by a (column) vector, which produces a \code{real} value.



\subsection{Operator Precedence and Associativity}\label{operator-precedence.section}

The precedence and associativity of operators, as well as built-in
syntax such as array indexing and function application is given in
tabular form in \reffigure{operator-precedence}.
%
\begin{figure}
\begin{center}
\begin{tabular}{c|ccl|l}
{\it Op.} & {\it Prec.} & {\it Assoc.} & {\it
  Placement} & {\it Description}
\\ \hline \hline
\code{||} & 9 & left & binary infix & logical or
\\ \hline
\Verb|&&| & 8 & left & binary infix & logical and
\\ \hline
\Verb|==| & 7 & left & binary infix & equality
\\
\Verb|!=| & 7 & left & binary infix & inequality
\\ \hline
\Verb|<| & 6 & left & binary infix & less than
\\
\Verb|<=| & 6 & left & binary infix & less than or equal
\\
\Verb|>| & 6 & left & binary infix & greater than 
\\
\Verb|>=| & 6 & left & binary infix & greater than or equal
\\ \hline
\code{+} & 5 & left & binary infix & addition
\\
\code{-} & 5 & left & binary infix & subtraction
\\ \hline
\code{*} & 4 & left & binary infix & multiplication
\\
\code{/} & 4 & left & binary infix & (right) division
\\ \hline
\Verb|\| & 3 & left & binary infix & left division
\\ \hline
\code{.*} & 2 & left & binary infix & elementwise multiplication
\\
\code{./} & 2 & left & binary infix & elementwise division
\\ \hline
\code{!} & 1 & n/a & unary prefix & logical negation
\\
\code{-} & 1 & n/a & unary prefix & negation
\\ 
\code{+} & 1 & n/a & unary prefix & promotion (no-op in \Stan)
\\ \hline
\code{'} & 0 & n/a & unary postfix & transposition
\\ \hline \hline
\code{()} & 0 & n/a & prefix, wrap & function application
\\
\code{[]} & 0 & left & prefix, wrap & array, matrix indexing
\end{tabular}
\end{center}
\caption{\it Stan's unary and binary operators, with their
  precedences, associativities, place in an expression, and a
  description.  The last two lines list the precedence of function
  application and array, matrix, and vector indexing. The operators are
  listed in order of precedence, from least tightly binding to most
  tightly binding.  The full set of legal arguments and corresponding
  result types are provided in the function documentation in
  \refpart{built-in-functions} prefaced with \code{operator} (i.e.,
  \code{operator*(int,int):int} indicates the application of the
  multiplication operator to two integers, which returns an integer).
  Parentheses may be used to group expressions explicitly rather than
  relying on precedence and
  associativity.}\label{operator-precedence.figure}
\end{figure}
%
Other expression-forming operations, such as function application and
subscripting bind more tightly than any of the arithmetic operations.  

The precedence and associativity determine how expressions are
interpreted.  Because addition is left associative, the expression
\code{a+b+c} is interpreted as \code{(a+b)+c}.  Similarly,
\code{a/b*c} is interpreted as \code{(a/b)*c}.  

Because multiplication has higher precedence than addition, the
expression \code{a*b+c} is interpreted as \code{(a*b)+c} and the
expression \code{a+b*c} is interpreted as \code{a+(b*c)}.  Similarly,
\code{2*x+3*-y} is interpreted as \code{(2*x)+(3*(-y))}.

Transposition binds tighter than all other operations, so that
\code{-u'} is interpreted as \code{-(u')}, \code{u*v'} as
\code{u*(v')}, and \code{u'*v} as \code{(u')*v}.

\section{Subscripting}

\Stan arrays, matrices, vectors, and row vectors are all accessed
using the same array-like notation.  For instance, if \code{x} is a
variable of type \code{real[]} (a one-dimensional array of reals)
then \code{x[1]} is the value of the first element of the
array.  

Subscripting has higher precedence than any of the arithmetic
operations.  For example, \code{alpha*x[1]} is equivalent to
\code{alpha*(x[1])}.  

Multiple subscripts may be provided within a single pair of square
brackets.  If \code{x} is of type \code{real[~,~]}, a two-dimensional
array, then \code{x[2,501]} is of type \code{real}.

\subsection{Accessing Subarrays}

The subscripting operator also returns subarrays of arrays.  For
example, if \code{x} is of type \code{real[~,~,~]}, then \code{x[2]}
is of type \code{real[~,~]}, and \code{x[2,3]} is of type
\code{real[]}.  As a result, the expressions \code{x[2,3]} and
\code{x[2][3]} have the same meaning.  

\subsection{Accessing Matrix Rows}

If \code{Sigma} is a variable of type \code{matrix}, then
\code{Sigma[1]} denotes the first row of \code{Sigma} and has the
type \code{row\_vector}.  

\subsection{Mixing Array and Vector/Matrix Indexes}

\Stan supports mixed indexing of arrays and their vector, row vector
or matrix values.  For example, if \code{m} is of type
\code{matrix[,]}, a two-dimensional array of matrices, then
\code{m[1]} refers to the first row of the array, which is a
one-dimensional array of matrices.  More than one index may be used,
so that \code{m[1,2]} is of type \code{matrix} and denotes the matrix
in the first row and second column of the array.  Continuing to add
indices, \code{m[1,2,3]} is of type \code{row\_vector} and denotes
the third row of the matrix denoted by \code{m[1,2]}.  Finally,
\code{m[1,2,3,4]} is of type \code{real} and denotes the value in the
third row and fourth column of the matrix that is found at the first
row and second column of the array \code{m}.

\section{Function Application}

\Stan provides a broad-range of built in mathematical and statistical
functions, which are documented in \refpart{built-in-functions}.

Expressions in \Stan may consist of the name of function followed by a
sequence of zero or more argument expressions.  For instance,
\code{log(2.0)} is the expression of type \code{real} denoting the
result of applying the natural logarithm to the value of the real
literal \code{2.0}.

Syntactically, function application has higher precedence than any of
the other operators, so that \code{y + log(x)} is interpreted as
\code{y + (log(x))}.

\subsection{Type Signatures and Result Type Inference}

Each function has a type signature which determines the allowable type
of its arguments and its return type.  For instance, the function
signature for the logarithm function can be expressed as
%
\begin{quote}
\code{real log(real);}
\end{quote}
%
and the signature for the \code{multiply\_log} function is
%
\begin{quote}
\code{real multiply\_log(real,real);}
\end{quote}
%
A function is uniquely determined by its name and its sequence of
argument types.  For instance, the following two functions are
different functions.
%
\begin{quote}
\code{real mean(real[]);}
\\
\code{real mean(vector);}
\end{quote}
%
The first applies to a one-dimensional array of real values and the
second to a vector.

The identity conditions for functions explicitly forbids having two
functions with the same name and argument types but different return
types.  This restriction also makes it possible to infer the type of a
function expression compositionally by only examining the type of its
subexpressions. 

\subsection{Constants}

Constants in \Stan are nothing more than nullary (no-argument)
functions.  For instance, the mathematical constants $\pi$ and $e$ are
represented as nullary functions named \code{pi()} and \code{e()}.
See \refsection{built-in-constants} for a list of built-in constants.

\subsection{Type Promotion and Function Resolution}\label{type-promotion.section}

Because of integer to real type promotion, rules must be established
for which function is called given a sequence of argument types.  The
scheme employed by \Stan is the same as that used by \Cpp, which
resolves a function call to the function requiring the minimum number
of type promotions.  

For example, consider a situation in which the following two function
signatures have been registered for \code{foo}.
%
\begin{quote}
\code{real foo(real,real);}
\\
\code{int foo(int,int);}
\end{quote}
%
The use of \code{foo} in the expression \code{foo(1.0,1.0)} resolves
to \code{foo(real,real)}, and thus the expression \code{foo(1.0,1.0)}
itself is assigned a type of \code{real}.  

Because integers may be promoted to real values, the expression
\code{foo(1,1)} could potentially match either \code{foo(real,real)}
or \code{foo(int,int)}.  The former requires two type promotions and
the latter requires none, so \code{foo(1,1)} is resolved to function
\code{foo(int,int)} and is thus assigned the type \code{int}.

The expression \code{foo(1,1.0)} has argument types \code{(int,real)}
and thus does not explicitly match either function signature.  By
promoting the integer expression \code{1} to type \code{real}, it is
able to match \code{foo(real,real)}, and hence the type of the
function expression \code{foo(1,1.0)} is \code{real}.

In some cases (though not for any built-in \Stan functions), a
situation may arise in which the function referred to by an
expression remains ambiguous.  For example, consider a situation in
which there are exactly two functions named \code{bar} with the
following signatures.
%
\begin{quote}
\code{real bar(real,int);}
\\
\code{real bar(int,real);}
\end{quote}
%
With these signatures, the expression \code{bar(1.0,1)} and
\code{bar(1,1.0)} resolve to the first and second of the above
functions, respectively.  The expression \code{bar(1.0,1.0)} is
illegal because real values may not be demoted to integers.  The
expression \code{bar(1,1)} is illegal for a different reason.  If the
first argument is promoted to a real value, it matches the first
signature, whereas if the second argument is promoted to a real value,
it matches the second signature.  The problem is that these both
require one promotion, so the function name \code{bar} is ambiguous.
If there is not a unique function requiring fewer promotions than all
others, as with \code{bar(1,1)} given the two declarations above, 
the \Stan compiler will flag the expression as illegal. 

\subsection{Random-Number Generating Functions}

For most of the distributions supported by Stan, there is a
corresponding random-number generating function.  These random number
generators are named by the distribution with the suffix \code{\_rng}.
For example, a univariate normal random number can be generated by
\code{normal\_rng(0,1)};  only the parameters of the distribution,
here a location (0) and scale (1) are specified because the variate is
generated.  

\subsubsection{Random-Number Generators Restricted to Generated Quantities Block}

The use of random-number generating functions is restricted to the
generated quantities block; attempts to use them elsewhere will result
in a parsing error with a diagnostic message.  

This allows the random number generating functions to be used for
simulation in general, and for Bayesian posterior predictive checking
in particular. 

\subsubsection{Posterior Predictive Checking}

Posterior predictive checks typically use the parameters of the model
to generate simulated data (at the individual and optionally at the
group level for hierarchical models), which can then be compared
informally using plots and formally by means of test statistics, to
the actual data in order to assess the suitability of the model; see
\citep[Chapter~6]{GelmanCarlinSternRubin:2003} for more information on
posterior predictive checks.

\section{Type Inference}

Stan is strongly statically typed, meaning that the implementation
type of an expression can be resolved at compile time.

\subsection{Implementation Types}

The primitive implementation types for Stan are 
%
\code{int},
\code{real},
\code{vector},
\code{row\_vector}, and
\code{matrix}.
%
Every basic declared type corresponds to a primitive type;  see
\reffigure{primitive-type} for the mapping from types to their
primitive types.
%
\begin{figure}
\begin{center}
\begin{tabular}{c|c}
{\it Type} & {\it Primitive Type} \\ \hline \hline
\code{int} & \code{int} \\
\code{real} & \code{real} \\[6pt]
\code{matrix} & \code{matrix} \\
\code{cov\_matrix} & \code{matrix} \\
\code{corr\_matrix} & \code{matrix} \\
\code{cholesky\_factor\_cov} & \code{matrix} \\
\end{tabular}
\hspace{0.4in}
\begin{tabular}{c|c}
{\it Type} & {\it Primitive Type} \\ \hline \hline
\code{vector} & \code{vector} \\
\code{simplex} & \code{vector} \\
\code{unit\_vector} & \code{vector} \\
\code{ordered} & \code{vector} \\
\code{positive\_ordered} & \code{vector} \\[6pt]
\code{row\_vector} & \code{row\_vector}
\end{tabular}
\end{center}
\caption{\it The table shows the variable declaration types of Stan
  and their corresponding primitive implementation type.  Stan
  functions, operators and probability functions have argument and
  result types declared in terms of primitive types.
}\label{primitive-type.figure}
\end{figure}
%
A full implementation type consists of a primitive implementation type
and an integer array dimensionality greater than or equal to zero.
These will be written to emphasize their array-like nature.  For
example, \code{int[]} has an array dimensionality of 1, \code{int} an
array dimensionality of 0, and \code{int[,,]} an array dimensionality
of 3. The implementation type \code{matrix[,,]} has a total of five
dimensions and takes up to five indices, three from the array and two
from the matrix.  

Recall that the array dimensions come before the matrix or vector
dimensions in an expression such as the following declaration of a
three-dimensional array of matrices.
%
\begin{quote}
\begin{Verbatim}
matrix[M,N] a[I,J,K];
\end{Verbatim}
\end{quote}
%
The matrix \code{a} is indexed as \code{a[i,j,k,m,n]} with the array
indices first, followed by the matrix indices, with \code{a[i,j,k]}
being a matrix and \code{a[i,j,k,m]} being a row vector.

\subsection{Type Inference Rules}

Stan's type inference rules define the implementation type of an
expression based on a background set of variable declarations.  The
rules work bottom up from primitive literal and variable expressions
to complex expressions.

\subsubsection{Literals}

An integer literal expression such as \code{42} is of type \code{int}.
Real literals such as \code{42.0} are of type \code{real}.

\subsubsection{Variables}

The type of a variable declared locally or in a previous block is
determined by its declaration.  The type of a loop variable is
\code{int}.  

There is always a unique declaration for each variable because Stan
prohibits the redeclaration of an already-declared variables.%
%
\footnote{Languages such as \Cpp and R allow the declaration of a
  variable of a given name in a narrower scope to hide (take
  precedence over for evaluation) a variable defined in a containing
  scope.  Stan will have to introduce this behavior eventually for
  user-defined functions written in Stan.}

\subsubsection{Indexing}

If \code{x} is an expression of total dimensionality greater than or
equal to $N$, then the type of expression \code{e[i1,...,iN]} is the
same as that of \code{e[i1]...[iN]}, so it suffices to define the type
of a singly-indexed function.  Suppose \code{e} is an expression and
\code{i} is an expression of primitive type \code{int}.  Then
%
\begin{itemize}
\item if \code{e} is an expression of array dimensionality $K > 0$,
  then \code{e[i]} has array dimensionality $K-1$ and the same
  primitive implementation type as \code{e},
%
\item if \code{e} has implementation type \code{vector} or
  \code{row\_vector} of array dimensionality 0, then \code{e[i]} has
  implementation type \code{real}, and
%
\item if \code{e} has implementation type \code{matrix}, then
  \code{e[i]} has type \code{row\_vector}.
\end{itemize}

\subsubsection{Function Application}

If \code{f} is the name of a function and \code{e1,...,eN} are
expressions for $N \geq 0$, then \code{f(e1,...,eN)} is an expression
whose type is determined by the return type in the function signature
for \code{f} given \code{e1} through \code{eN}.  Recall that a
function signature is a declaration of the argument types and the
result type.  

In looking up functions, binary operators like \code{real~*~real} are
defined as \code{operator*(real,real)} in the documentation and index.

In matching a function definition, arguments of type \code{int} may be
promoted to type \code{real} if necessary (see \refsection{type-promotion} for an
exact specification of Stan's integer-to-real type-promotion rule).

In general, matrix operations return the lowest inferrable type.  For
example, \code{row\_vector~*~vector} returns a value of type
\code{real}, which is declared in the function documentation and index
as \code{real~operator*(row\_vector,vector)}.

% \subsection{Extending \Stan with User-Defined Functions}

% \Stan enables users to add their own functions by defining appropriate
% \Cpp functions and registering their signatures with the \Stan
% compiler.  Details are provided in \refappendix{user-defined-functions}.


\section{Chain Rule and Derivatives}

Derivatives of the log probability function defined by a model are
used in several ways by Stan.  The Hamiltonian Monte Carlo samplers,
including NUTS, use gradients to guide updates.  The BFGS optimizers
also use gradients to guide search for posterior modes.  

\subsection{Errors Due to Chain Rule}

Unlike evaluations in pure mathematics, evaluation of derivatives in
Stan is done by applying the chain rule on an expression-by-expression
basis, evaluating using floating-point arithmetic.  As a result,
models such as the following are problematic for inference involving
derivatives.
%
\begin{quote}
\begin{Verbatim}
parameters {
  real x;
}
model {
  x ~ normal(sqrt(x - x), 1);
}
\end{Verbatim}
\end{quote}
%
Algebraically, the sampling statement in the model could be reduced to
%
\begin{quote}
\begin{Verbatim}
  x ~ normal(0, 1);
\end{Verbatim}
\end{quote}
%
and it would seem the model should produce unit normal samples for
\code{x}.  But rather than cancelling, the expression \code{sqrt(x -
  x)} causes a problem for derivatives.  The cause is the mechanistic
evaluation of the chain rule, 
%
\begin{eqnarray*}
\frac{d}{dx} \sqrt{x - x}
& = &
\frac{1}{2 \sqrt{x - x}} \times \frac{d}{dx} (x - x)
\\[4pt]
& = & 
\frac{1}{0} \times (1 - 1)
\\[4pt]
& = & 
\infty \times 0
\\[4pt]
& = & \mbox{NaN}.
\end{eqnarray*}
%
Rather than the $x - x$ cancelling out, it introduces a 0 into the
numerator and denominator of the chain-rule evaluation.

The only way to avoid this kind problem is to be careful to do the
necessary algebraic reductions as part of the model and not introduce
expressions like \code{sqrt(x - x)} for which the chain rule produces
not-a-number values.  

\subsection{Diagnosing Problems with Derivatives}

The best way to diagnose whether something is going wrong with the
derivatives is to use the test-gradient option to the sampler or
optimizer inputs; this option is availabe in both Stan and RStan
(though it may be slow, because it relies on finite differences to
make a comparison to the built-in automatic differentiation).

For example, compiling the above model to an executable
\code{sqrt-x-minus-x}, the test can be run as
%
\begin{Verbatim}[fontshape=sl,fontsize=\small]
> ./sqrt-x-minus-x diagnose test=gradient
\end{Verbatim}
\begin{Verbatim}[fontsize=\small]
...
TEST GRADIENT MODE

 Log probability=-0.393734

 param idx           value           model     finite diff           error
         0       -0.887393             nan               0             nan
\end{Verbatim}
%
Even though finite differences calculates the right gradient of 0,
automatic differentiation follows the chain rule and produces a
not-a-number output.



\chapter{Statements}

\noindent 
The blocks of a \Stan program (see \refchapter{blocks}) are made up of
variable declarations and statements.  Unlike programs in \BUGS, the
declarations and statements making up a \Stan program are executed in
the order in which they are written.  Variables must be defined to
have some value (as well as declared to have some type) before they
are used --- if they do not, the behavior is undefined.

Like \BUGS, \Stan has two kinds of atomic statements, assignment
statements and sampling statements.  Also like \BUGS, statements may
be grouped into sequences and into for-each loops.  In addition, \Stan
allows local variables to be declared in blocks and also allows an
empty statement consisting only of a semicolon.

\section{Assignment Statement}

An assignment statement consists of a variable (possibly multivariate
with indexing information) and an expression.  Executing an
assignment statement evaluates the expression on the right-hand side
and assigns it to the (indexed) variable on the left-hand side.  An
example of a simple assignment is
%
\begin{quote}
\code{n <- 0;}
\end{quote}
%
Executing this statement assigns the value of the expression \code{0},
which is the integer zero, to the variable \code{n}.  For an assignment
to be well formed, the type of the expression on the right-hand side
should be compatible with the type of the (indexed) variable on the
left-hand side.  For the above example, because \code{0} is an
expression of type \code{int}, the variable \code{n} must be declared
as being of type \code{int} or of type \code{real}.  If the variable
is of type \code{real}, the integer zero is promoted to a
floating-point zero and assigned to the variable.  After the
assignment statement executes, the variable \code{n} will have the
value zero (either as an integer or a floating-point value, depending on
its type).

Syntactically, every assignment statement must be followed by a
semicolon.  Otherwise, whitespace between the tokens does not matter
(the tokens here being the left-hand-side (indexed) variable, the
assignment operator, the right-hand-side expression and the
semicolon).

Because the right-hand side is evaluated first, it is possible to
increment a variable in \Stan just as in \Cpp and other programming
languages by writing
%
\begin{quote}
\code{n <- n + 1;}
\end{quote}
%
Such self assignments are not allowed in \BUGS, because they induce a
cycle into the directed graphical model.

The left-hand side of an assignment may contain indices for array,
matrix, or vector data structures.  For instance, if \code{Sigma} is
of type \code{matrix}, then
%
\begin{quote}
\code{Sigma[1,1] <- 1.0;}
\end{quote}
%
sets the value in the first column of the first row of \code{Sigma} to one.

Assignments can involve complex objects of any type.  If \code{Sigma}
and \code{Omega} are matrices and \code{sigma} is a vector, then the
following assignment statement, in which the expression and variable
are both of type \code{matrix}, is well formed.
%
\begin{quote}
\begin{Verbatim}
Sigma
  <- diag_matrix(sigma)
     * Omega 
     * diag_matrix(sigma);
\end{Verbatim}
\end{quote}
%
This example also illustrates the preferred form of splitting a
complex assignment statement and its expression across lines.

Assignments to slices of larger multi-variate data structures are
supported by \Stan.  For example, \code{a} is an array of type
\code{real[~,~]} and \code{b} is an array of type \code{real[]}, then
the following two statements are both well-formed.
%
\begin{quote}
\begin{Verbatim}
a[3] <- b;
b <- a[4];
\end{Verbatim}
\end{quote}
%
Similarly, if \code{x} is a variable declared to have type
\code{row\_vector} and \code{Y} is a variable declared as type
\code{matrix}, then the following sequence of statements to swap the
first two rows of \code{Y} is well formed.
%
\begin{quote}
\begin{Verbatim}
x <- Y[1];
Y[1] <- Y[2];
Y[2] <- x;
\end{Verbatim}
\end{quote}
%

In \R, if \code{x} is a matrix or two-dimensional array, its first row
is \code{x[1,]} and its first column is \code{x[,1]}.  As of version
2.0, this notation is not supported by \Stan.  There are functions to
access rows and columns of matrices, but general array slicing is not
supported.  Similarly, \Stan 2.0 does not support providing an array of
indices as an argument to create a piecemeal subarray of a larger
array.

\section{Log Probability Increment Statement}

The basis of Stan's execution is the evaluation of a log probability
function for a given set of parameters.  Data and transformed data are
fixed before log probability is involved.  Statements in the
transformed parameter block and model block can have an effect on the
log probability function defined by a model.

The total log probability is initialized to zero.  Then statements in
the transformed parameter block and model block may add to it.  The
most direct way this is done is through the log probability increment
statement, which is of the following form.
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(-0.5 * y * y);
\end{Verbatim}
\end{quote}
%
In this example, the unnormalized log probability of a unit normal
variable $y$ is adeed to the total log probability.  In the general
case, the argument can be any expression.  Writing this model as
\code{-0.5 * y * y} is more efficient than the equivalent \code{y * y
  / 2} because multiplication is more efficient than division and the
negation is rolled into the constant \code{-0.5}.

An entire Stan model can be implemented this way.  For instance, the
following model will draw a single variable according to a unit normal
probability.  
%
\begin{quote}
\begin{Verbatim}
parameters {
  real y;
}
model {
  increment_log_prob(-0.5 * y * y);
}
\end{Verbatim}
\end{quote}
%
This model defines a log probability function 
%
\[
\log p(y) = - \, \frac{y^2}{2} - \log Z
\]
%
where $Z$ is a normalizing constant that does not depend on $y$.  The
constant $Z$ is conventionally written this way because on the linear
scale,
\[
p(y) = \frac{1}{Z} \exp\left(-\frac{y^2}{2}\right).
\]
which is typically written without reference to $Z$ as
\[
p(y) \propto \exp\left(-\frac{y^2}{2}\right).
\]

Stan only requires models to be defined up to a constant that does not
depend on the parameters.  This is convenient because often the
normalizing constant $Z$ is either time-consuming to compute or
intractable to evaluate.

\subsection{Log Probability Variable \code{lp\_\_}}

Before version 2.0 of Stan, rather than writing
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(u);
\end{Verbatim}
\end{quote}
%
it was necessary to manipulate a special variable \code{lp\_\_}
as follows
%
\begin{quote}
\begin{Verbatim}
lp__ <- lp__ + u;
\end{Verbatim}
\end{quote}
%
The special variable \code{lp\_\_} refers to the log probability value
that will be returned by Stan's log probability function.  

\subsubsection{Deprecation of \code{lp\_\_}}

As of Stan version 2.0, the use of \code{lp\_\_} is deprecated.  It
will be removed altogether in a future release, but until that time,
programs still work with \code{lp\_\_}.  A deprecation warning is now
printed every time \code{lp\_\_} is used.


\section{Sampling Statements}\label{sampling-statements.section}

Like \BUGS and \JAGS, \Stan supports probability statements in
sampling notation, such as
%
\begin{quote}
\begin{Verbatim}
y ~ normal(mu,sigma);
\end{Verbatim}
\end{quote}
%
The name ``sampling statement'' is meant to be suggestive, not
interpreted literally.  Conceptually, the variable \code{y}, which may
be an unknown parameter or known, modeled data, is being declared
to have the distribution indicated by the right-hand side of the
sampling statement.

Executing such a statement does not perform any sampling.  In \Stan, a
sampling statement is merely a notational convenience.  The above
sampling statement could be expressed as a direct increment on the
total log probability as
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(normal_log(y,mu,sigma));
\end{Verbatim}
\end{quote}
%
See \refsection{incr-vs-sample} for a full explanation.

In general, a sampling statement of the form
%
\begin{quote}
\begin{Verbatim}
ex0 ~ dist(ex1,...,exN);
\end{Verbatim}
\end{quote}
%
involving subexpressions \code{ex0} through \code{exN} (including the
case where \code{N} is zero) will be well formed if and only if the
corresponding assignment statement is well-formed,
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(dist_log(ex0,ex1,...,exN));
\end{Verbatim}
\end{quote}
%
This will be well formed if and only if
\code{dist\_log(ex0,ex1,...,exN)} is a well-formed function expression
of type \code{real}.

\subsection{Log Probability Increment vs.\ Sampling Statement}\label{incr-vs-sample.section}

Although both lead to the same sampling behavior in Stan, there is one
critical difference between using the sampling statement, as in
%
\begin{quote}
\begin{Verbatim}
y ~ normal(mu,sigma);
\end{Verbatim}
\end{quote}
%
and explicitly incrementing the log probability function, as in
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(normal_log(y,mu,sigma));
\end{Verbatim}
\end{quote}
%
The sampling statement drops all the terms in the log probability
function that are constant, whereas the explicit call to
\code{normal\_log} adds all of the terms in the definition of the log
normal probability function, including all of the constant normalizing
terms.  Therefore, the explicit increment form can be used to recreate
the exact log probability values for the model.  Otherwise, the
sampling statement form will be faster if any of the input expressions,
\code{y}, \code{mu}, or \code{sigma}, involve only constants, data
variables or transformed data variables. 


\subsection{User-Transformed Variables}

The left-hand side of a sampling statement may be a complex
expression.  For instance, it is legal syntactically to write
%
\begin{quote}
\begin{Verbatim}
data {
  real<lower=0> y;
}
...
model {
  log(y) ~ normal(mu,sigma);
}
\end{Verbatim}
\end{quote}
%
Unfortunately, this is not enough to properly model \code{y} as having
a lognormal distribution.  The log Jacobian of the transform must be
added to the log probability accumulator to account for the
differential change in scale (see \refsection{change-of-variables} for
full definitions).  For the case above, the following adjustment will
account for the log transform.%
%
\footnote{Because $\log | \frac{d}{dy} \log y | = \log | 1/y | = - \log
  |y|$;  see \refsection{change-of-variables}.}
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(- log(fabs(y)));
\end{Verbatim}
\end{quote}
%

\subsection{Truncated Distributions}\label{truncated-distributions.section}

A density function $p(x)$ may be truncated to an interval $(a,b)$ to
define a new density $p_{(a,b)}\!(x)$ by setting
%
\[ 
p_{\!(a,b)\!}(x) = \frac{p(x)}
                  {\int_a^b p(x') \, dx'}.
\] 
Given a probability function $p_X(x)$ for a random variable $X$, its
cumulative distribution function (cdf) $F_X(x)$ is defined to be the
probability that $X \leq a$.  For continuous random variables, this is
\[
F_X(x) = \int_{-\infty}^{x} p_X(x') \, dx',
\]
whereas for discrete variables, it's
\[
F_X(x) = \sum_{n \leq x} p_X(x).
\]
%
The complementary cumulative distribution function (ccdf) is $1 -
F_X(x)$.  

Cumulative distribution functions are useful for defining truncated
distributions, because
\[
\int_a^b p(x') \, dx' = F(b) - F(a),
\]
so that
\[ 
p_{\!(a,b)\!}(x) = \frac{p_X(x)}
                  {F(b) - F(a)}.
\] 
On the log scale,
\[ 
\log p_{\!(a,b)\!}(x) 
\ = \ 
\log p_X(x)
- \log \left( F(b) - F(a) \right).
\]
The denominator is more stably computed using Stan's
log\_diff\_exp operation as
\begin{eqnarray*}
\log \left( F(b) - F(a) \right)
& = & \log \left( \exp(\log F(b)) - \exp(\log F(a)) \right)
\\[6pt]
& = & \mbox{log\_diff\_exp}\!\left( \log F(b), \log F(a) \right).
\end{eqnarray*}



As in \BUGS and \JAGS, \Stan allows probability functions to be
truncated.  For example, a truncated unit normal distribution
restricted to $(-0.5, 2.1)$ is encoded as follows.
%
\begin{quote}
\begin{Verbatim}
y ~ normal(0,1) T[-0.5, 2.1];
\end{Verbatim}
\end{quote}
% 
Truncated distributions are translated as an addition summation for
the accumulated log probability.  For instance, this example has the
same translation (up to arithmetic precision issues; see below) as
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(normal_log(y,0,1));
increment_log_prob(-(log(normal_cdf(2.1,0,1)
                         - normal_cdf(-0.5,0,1))));
\end{Verbatim}
\end{quote}
%
The function \code{normal\_cdf} represents the cumulative normal
distribution function.  For example, \code{normal\_cdf(2.1,0,1)} evaluates to 
\[
\int_{-\infty}^{2.1} \mbox{\sf Normal}(x|0,1) \, dx,
\]
%
which is the probability a unit normal variable takes on values less
than 2.1, or about 0.98.  

Arithmetic precision is handled by working on the log scale and using
Stan's log scale cdf implementations.  The logarithm of the cdf for
the normal distribution is \code{normal\_cdf\_log} and the ccdf is
\code{normal\_ccdf\_log}.  Stan's translation for the denominator
introduced by truncation is equivalent to
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(-log_diff_exp(normal_cdf_log(2.1,0,1),
                                 normal_cdf_log(-0.5,0,1)));
\end{Verbatim}
\end{quote}
%

As with constrained variable declarations, truncation can be one
sided.  The density $p(x)$ can be truncated below by $a$ to define a
density $p_{(a,)}(x)$ with support $(a,\infty)$ by setting
%
\[
p_{(a,)}(x) = \frac{p(x)}
                 {\int_a^{\infty} p(x') \, dx'}.
\]
For example, the unit normal distribution truncated below at -0.5 would
be represented as
%
\begin{quote}
\begin{Verbatim} 
y ~ normal(0,1) T[-0.5,];
\end{Verbatim}
\end{quote}
% 
The truncation has the same effect as the following direct update to
the accumulated log probability (see \refsection{incr-vs-sample} for
more information about the relation between incrementing the total log
probability and sampling).
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(normal_log(y, 0, 1));
increment_log_prob(-(1 - log(normal_cdf(-0.5, 0, 1))));
\end{Verbatim}
\end{quote}
%
The denominator is actually implemented with the more efficient and
stable version
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(-normal_ccdf_log(-0.5, 0, 1));
\end{Verbatim}
\end{quote}

The density $p(x)$ can be truncated above by $b$ to define a density
$p_{(,b)}(x)$ with support $(-\infty,b)$ by setting
\[
p_{(,b)}(x) = \frac{p(x)}
                    {\int_{-\infty}^b p(x') \, dx'}.
\]
For example, the unit normal distribution truncated above at 2.1 would
be represented as
%
\begin{quote}
\begin{Verbatim} 
y ~ normal(0,1) T[,2.1];
\end{Verbatim}
\end{quote}
% 
The truncation has the same effect as the following direct update to
the accumulated log probability.
%
\begin{quote}
\begin{Verbatim}
increment_log_prob(normal_log(y, 0, 1));
increment_log_prob(-normal_cdf_log(2.1, 0, 1));
\end{Verbatim}
\end{quote}

In all cases, the truncation is only well formed if the
appropriate log cumulative distribution functions are defined.%
%
\footnote{Although most distributions have cdfs and ccdfs implemented,
  some cumulative distribution functions and their gradients present
  computational challenges because they lack simple, analytic forms.}
%
\refpart{discrete-prob-functions} and
\refpart{continuous-prob-functions} document the available discrete
and continuous cumulative distribution functions.  Most distributions
have cdf and ccdf functions.

For continuous distributions, truncation points must be expressions of
type \code{int} or \code{real}.  For discrete distributions, truncation
points must be expressions of type \code{int}.  

For a truncated sampling statement, if the value sampled is not within
the bounds specified by the truncation expression, the result is zero
probability and the entire statement adds $-\infty$ to the total log
probability, which in turn results in the sample being rejected; see
\refsection{truncation-constraints} for programming strategies to keep
all values within bounds.

\subsubsection{Vectorizing Truncated Distributions}

Stan does not (yet) support vectorization of distribution functions
with truncation.

% \subsection{User-Defined Distributions}

% Users may add their own probability functions and cumulative
% distribution by adding functions of the appropriate type.  If the
% appropriate cumulative distribution is available to match a
% probability function, truncated versions of that probability function
% may be used. 

% See \refappendix{user-defined-functions} for details on how to add
% user-defined functions to \Stan.


\section{For Loops}

Suppose
\code{N} is a variable of type \code{int}, \code{y} is a
one-dimensional array of type \code{real[]}, and \code{mu} and
\code{sigma} are variables of type \code{real}.  Furthermore, suppose
that \code{n} has not been defined as a variable. Then the following
is a well-formed for-loop statement.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) {
  y[n] ~ normal(mu,sigma);
}
\end{Verbatim}
\end{quote}
%
The loop variable is \code{n}, the loop bounds are the values in the
range \code{1:N}, and the body is the statement following the
loop bounds.  

\subsection{Loop Variable Typing and Scope}

The bounds in a for loop must be integers.  Unlike in \R, the loop is
always interpreted as an upward counting loop.  The range \code{L:H}
will cause the loop to execute the loop with the loop variable taking
on all integer values greater than or equal to \code{L} and less than
or equal to \code{H}.  For example, the loop \code{for (n in 2:5)}
will cause the body of the for loop to be executed with \code{n} equal
to 2, 3, 4, and 5, in order.  The variable and bound \code{for (n in
  5:2)} will not execute anything because there are no integers
greater than or equal to 5 and less than or equal to 2.

\subsection{Order Sensitivity and Repeated Variables}

Unlike in \BUGS, \Stan allows variables to be reassigned.  For
example, the variable \code{theta} in the following program is
reassigned in each iteration of the loop.
%
\begin{quote}
\begin{Verbatim} 
for (n in 1:N) {
  theta <- inv_logit(alpha + x[n] * beta);
  y[n] ~ bernoulli(theta);
}
\end{Verbatim}
\end{quote}
% 
Such reassignment is not permitted in \BUGS.  In \BUGS, for loops are
declarative, defining plates in directed graphical model notation,
which can be thought of as repeated substructures in the graphical
model.  Therefore, it is illegal in \BUGS or \JAGS to have a for loop
that repeatedly reassigns a value to a variable.%
%
\footnote{A programming idiom in \BUGS code simulates 
a local variable by replacing \code{theta} in the above example with
\code{theta[n]}, effectively creating \code{N} different variables,
\code{theta[1]}, \ldots, \code{theta[N]}.  Of course, this is not a
hack if the value of \code{theta[n]} is required for all \code{n}.}

In \Stan, assignments are executed in the order they are encountered.
As a consequence, the following \Stan program has a very different
interpretation than the previous one.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) {
  y[n] ~ bernoulli(theta);
  theta <- inv_logit(alpha + x[n] * beta);
}
\end{Verbatim}
\end{quote}
%
In this program, \code{theta} is assigned after it is used in the
probability statement.  This presupposes it was defined before the
first loop iteration (otherwise behavior is undefined), and then each
loop uses the assignment from the previous iteration. 

\Stan loops may be used to accumulate values.  Thus it is possible to
sum the values of an array directly using code such as the
following.
%
\begin{quote}
\begin{Verbatim}
total <- 0.0;
for (n in 1:N) 
  total <- total + x[n];
\end{Verbatim}
\end{quote}
%
After the for loop is executed, the variable \code{total} will hold
the sum of the elements in the array \code{x}.  This example was
purely pedagogical; it is easier and more efficient to write
%
\begin{quote}
\begin{Verbatim}
total <- sum(x);
\end{Verbatim}
\end{quote}

A variable inside (or outside) a loop may even be reassigned multiple
times, as in the following legal code.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:100) {
  y <- y + y * epsilon;
  epsilon <- epsilon / 2.0;
  y <- y + y * epsilon;
}
\end{Verbatim}
\end{quote}

\section{Conditional Statements}

Stan supports full conditional statements using
the same if-then-else syntax as \Cpp.  The general format is
%
\begin{quote}
\begin{Verbatim}
if (condition1)
  statement1
else if (condition2)
  statement2
...
else if (conditionN-1)
  statementN-1
else
  statementN
\end{Verbatim}
\end{quote}
%
There must be a single leading \code{if} clause, which may be followed
by any number of \code{else if} clauses, all of which may be
optionally followed by an \code{else} clause.  Each condition must be
a real or integer value, with non-zero values interpreted as true and
the zero value as false.

The entire sequence of if-then-else clauses forms a single conditional
statement for evaluation.  The conditions are evaluated in order
until one of the conditions evaluates to a non-zero value, at which
point its corresponding statement is executed and the conditional
statement finishes execution.  If none of the conditions evaluates to
a non-zero value and there is a final else clause, its statement is
executed.  

\section{While Statements}

Stan supports standard while loops using the same syntax as \Cpp.  The
general format is as follows.
%
\begin{quote}
\begin{Verbatim}
while (condition)
  body
\end{Verbatim}
\end{quote}
%
The condition must be an integer or real expression and the body can
be any statement (or sequence of statements in curly braces).  

Evaluation of a while loop starts by evaluating the condition.  If the
condition evaluates to a false (zero) value, the execution of the loop
terminates and control moves to the position after the loop.  If the
loop's condition evaluates to a true (non-zero) value, the body statement is
executed, then the whole loop is executed again.  Thus the loop is
continually executed as long as the condition evaluates to a true value.


\section{Statement Blocks and Local Variable Declarations}

Just as parentheses may be used to group expressions, curly brackets
may be used to group a sequence of zero or more statements into a
statement block.  At the beginning of each block, local variables may be
declared that are scoped over the rest of the statements in the block.

\subsection{Blocks in For Loops}

Blocks are often used to group a sequence of statements together to be
used in the body of a for loop.  Because the body of a for loop can be
any statement, for loops with bodies consisting of a single statement
can be written as follows.
%
\begin{quote}
\begin{Verbatim} 
for (n in 1:N) 
  y[n] ~ normal(mu,sigma);
\end{Verbatim}
\end{quote}
% 
To put multiple statements inside the body of a for loop, a block is
used, as in the following example.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) {
  lambda[n] ~ gamma(alpha,beta);
  y[n] ~ poisson(lambda[n]);
}
\end{Verbatim}
\end{quote}
%
The open curly bracket (\code{\{}) is the first character of the block
and the close curly bracket (\code{\}}) is the last character.

Because whitespace is ignored in \Stan, the following program will
not compile.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) 
  y[n] ~ normal(mu,sigma);
  z[n] ~ normal(mu,sigma); // ERROR!
\end{Verbatim}
\end{quote}
%
The problem is that the body of the for loop is taken to be the
statement directly following it, which is 
\Verb|y[n] ~ normal(mu,sigma)|.  This leaves the probability statement for
\code{z[n]} hanging, as is clear from the following equivalent
program.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) {
  y[n] ~ normal(mu,sigma);
}
z[n] ~ normal(mu,sigma); // ERROR!
\end{Verbatim}
\end{quote}
%
Neither of these programs will compile. If the loop variable \code{n}
was defined before the for loop, the for-loop declaration will raise
an error.  If the loop variable \code{n} was not defined before the
for loop, then the use of the expression \code{z[n]} will raise an
error. 

\subsection{Local Variable Declarations}

A for loop has a statement as a body.  It is often convenient in
writing programs to be able to define a local variable that will be
used temporarily and then forgotten.  For instance, the for loop
example of repeated assignment should use a local variable for maximum
clarity and efficiency, as in the following example.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) {
  real theta;
  theta <- inv_logit(alpha + x[n] * beta);
  y[n] ~ bernoulli(theta);
}
\end{Verbatim}
\end{quote}
%
The local variable \code{theta} is declared here inside the for loop.
The scope of a local variable is just the block in which it is
defined.  Thus \code{theta} is available for use inside the for loop,
but not outside of it.  As in other situations, \Stan does not allow
variable hiding.  So it is illegal to declare a local variable
\code{theta} if the variable theta is already defined in the scope of
the for loop.  For instance, the following is not legal.
%
\begin{quote}
\begin{Verbatim}
for (m in 1:M) {
  real theta;
  for (n in 1:N) {
    real theta; // ERROR!
    theta <- inv_logit(alpha + x[m,n] * beta);
    y[m,n] ~ bernoulli(theta);
...
\end{Verbatim}
\end{quote}
%
The compiler will flag the second declaration of \code{theta} with a
message that it is already defined.

\subsection{No Constraints on Local Variables}

Local variables may not have constraints on their declaration.  The
only types that may be used are
%
\begin{quote}
\code{int}, \code{real}, 
\code{vector[K]}, \code{row\_vector[K]}, and \code{matrix[M,N]}.
\end{quote}

\subsection{Blocks within Blocks}

A block is itself a statement, so anywhere a sequence of statements is
allowed, one or more of the statements may be a block.  For instance,
in a for loop, it is legal to have the following
%
\begin{quote}
\begin{Verbatim}
for (m in 1:M) {
  { 
     int n;  
     n <- 2 * m; 
     sum <- sum + n 
  }
  for (n in 1:N) 
    sum <- sum + x[m,n];
}
\end{Verbatim}
\end{quote}
%
The variable declaration \code{int n;} is the first element of an
embedded block and so has scope within that block.  The for loop
defines its own local block implicitly over the statement following it
in which the loop variable is defined.  As far as \Stan is concerned,
these two uses of \code{n} are unrelated.  

\section{Print Statements}\label{print-statements.section}

Stan provides print statements that can print literal strings and the
values of expressions.  Print statements accept any number of
arguments.  Consider the following for-each statement with a print
statement in its body.
%
\begin{quote}
\begin{Verbatim}
for (n in 1:N) { print("loop iteration: ", n); ... }
\end{Verbatim}
\end{quote}
%
The print statement will execute every time the body of the loop does.
Each time the loop body is executed, it will print the string ``loop iteration:
'' (with the trailing space), followed by the value of the expression
\code{n}, followed by a new line.

\subsection{Print Content}

The text printed by a print statement varies based on its content.  A
literal (i.e., quoted) string in a print statement always prints
exactly that string (without the quotes).  Expressions in print
statements result in the value of the expression being printed.
But how the value of the expression is formatted will depend on its type.

Printing a simple \code{real} or \code{int} typed variable always
prints the variable's value.%
%
\footnote{The adjoint component is always zero during execution for
  the algorithmic differentiation variables used to implement
  parameters, transformed parameters, and local variables in the model.}
%
For array, vector, and matrix variables, the print format uses
brackets.  For example, a 3-vector will print as
%
\begin{quote}
\begin{Verbatim}
[1,2,3]
\end{Verbatim}
\end{quote}
%
and a $2 \times 3$-matrix as 
%
\begin{quote}
\begin{Verbatim}
[[1,2,3],[4,5,6]]
\end{Verbatim}
\end{quote}
%

Printing a more readable version of arrays or matrices can be done
with loops.  An example is the print statement in the following
transformed data block.
%
\begin{quote}
\begin{Verbatim}
transformed data {
  matrix[2,2] u;  
  u[1,1] <- 1.0;  u[1,2] <- 4.0;    
  u[2,1] <- 9.0;  u[2,2] <- 16.0;
  for (n in 1:2)
    print("u[", n, "] = ", u[n]);
}
\end{Verbatim}
\end{quote}
%
This print statement executes twice, printing the following two lines
of output.
%
\begin{quote}
\begin{Verbatim}
u[1] = [1,4]
u[2] = [9,16]
\end{Verbatim}
\end{quote}



\subsection{Print Frequency}

Printing for a print statement happens every time it is executed.  The
\code{transformed data} block is executed once per chain, the
\code{transformed parameter} and \code{model} blocks once per leapfrog
step, and the \code{generated quantities} block once per iteration.

\subsection{String Literals}

String literals begin and end with a double quote character
(\Verb|"|).  The characters between the double quote characters may be
the space character or any visible ASCII character, with the exception
of the backslash character (\Verb|\|) and double quote character
(\Verb|"|).  The full list of visible ASCII characters is as follows.
%
\begin{quote}
\begin{Verbatim}
a b c d e f g h i j k l m n o p q r s t u v w x y z
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
0 1 2 3 4 5 6 7 8 9 0 ~ @ # $ % ^ & * _ ' ` - + = { 
} [ ] ( ) < > | / ! ? . , ; :
\end{Verbatim}
\end{quote}

\subsection{Debug by \code{print}}

Because Stan is an imperative language, print statements can be very
useful for debugging.  They can be used to display the values of
variables or expressions at various points in the execution of a
program.  They are particularly useful for spotting problematic
not-a-number of infinite values, both of which will be printed.




\chapter{Program Blocks}\label{blocks.chapter}

\noindent
A \Stan program is organized into a sequence of named blocks, the
bodies of which consist of variable declarations, followed in the case
of some blocks with statements.  

\section{Comments}\label{comments.section}

Stan supports \Cpp-style line-based and bracketed comments.  Comments
may be used anywhere whitespace is allowed in a Stan program.

\subsection{Line-Based Comments}

In line-based comments, any text on a line following
two forward slashes (\Verb|//|) or the pound sign (\Verb|#|)
is ignored (along with the slashes or pound sign).  

\subsection{Bracketed Comments}

For bracketed comments, any text between a forward-slash and asterisk
pair (\Verb|/*|) and an asterisk and forward-slash pair (\Verb|*/|) is
ignored.

\subsection{Character Encoding}

Comments may be in ASCII, UTF-8, Latin1, or any other character
encoding that is byte-wise compatible with ASCII.  This excludes
encodings like UTF-16, Big5, etc.%
%
\footnote{The issue is that they will separate the characters in
  \code{*/} and \code{*/}.}


\section{Overview of \Stan's Program Blocks}

The full set of named program blocks is exemplified in the following
skeletal \Stan program.
%
\begin{quote}
\begin{Verbatim} 
data { 
  ... declarations ...
}
transformed data { 
   ... declarations ... statements ... 
}
parameters { 
   ... declarations ... 
}
transformed parameters { 
   ... declarations ... statements ...
}
model { 
   ... declarations ... statements ...
}
generated quantities {
   ... declarations ... statements ...
}
\end{Verbatim}
\end{quote}
%

\subsection{Optionality and Ordering}

All of the blocks other than the \code{model} block are optional.  The
blocks that occur must occur in the order presented in the skeletal
program above.  Within each block, both declarations and statements
are optional, subject to the restriction that the declarations come
before the statements.

\subsection{Variable Scope}

The variables declared in each block have scope over all subsequent
statements.  Thus a variable declared in the transformed data block
may be used in the model block.  But a variable declared in the
generated quantities block may not be used in any earlier block,
including the model block.

\subsection{Automatic Variable Definitions}

The variables declared in the \code{data} and \code{parameters} block
are treated differently than other variables in that they are
automatically defined by the context in which they are used.  This is
why there are no statements allowed in the data or parameters block.

The variables in the \code{data} block are read from an external input
source such as a file or a designated \R data structure.  The
variables in the \code{parameters} block are read from the sampler's
current parameter values (either standard \HMC or \NUTS).  The initial
values may be provided through an external input source, which is also
typically a file or a designated \R data structure.  In each case, the
parameters are instantiated to the values for which the model defines
a log probability function.

\subsection{Transformed Variables}

The \code{transformed data} and \code{transformed parameters} block
behave similarly to each other.  Both allow new variables to be
declared and then defined through a sequence of statements.  Because
variables scope over every statement that follows them, transformed
data variables may be defined in terms of the data variables.

Before generating any samples, data variables are read in, then the
transformed data variables are declared and the associated statements
executed to define them.  This means the statements in the transformed
data block are only ever evaluated once.%
%
\footnote{If the \Cpp code is configured for concurrent threads, the
  data and transformed data blocks can be executed once and reused for
  multiple chains.}
%
Transformed parameters work the same way, being defined in terms of
the parameters, transformed data, and data variables.  The difference
is the frequency of evaluation.  Parameters are read in and (inverse)
transformed to constrained representations on their natural scales
once per log probability and gradient evaluation.  This means the
inverse transforms and their log absolute Jacobian determinants are
evaluated once per leapfrog step.  Transformed parameters are then
declared and their defining statements executed once per leapfrog
step.

\subsection{Generated Quantities}

The generated quantity variables are defined once per sample after all
the leapfrog steps have been completed.  These may be random
quantities, so the block must be rerun even if the Metropolis
adjustment of \HMC or \NUTS rejects the update proposal.  


\subsection{Variable Read, Write, and Definition Summary}

A table summarizing the point at which variables are read, written, and
defined is given in \reffigure{block-actions}.
%
\begin{figure}
\begin{center}
\begin{tabular}{l|c|l}
{\it Block} & {\it Stmt} & {\it Action / Period} 
\\\hline\hline
\code{data} & no & read / chain  
\\
\code{transformed data} & yes & evaluate / chain  
\\ \hline
\code{parameters} & no & inv.\ transform, Jacobian / leapfrog  \\
& & inv.\ transform, write / sample 
\\[3pt]
\code{transformed parameters} & yes & evaluate / leapfrog \\
& & write / sample 
\\\hline
\code{model} & yes & evaluate / leapfrog step 
\\\hline
\code{generated quantities} & yes & eval / sample \\
& & write / sample
\\\hline\hline
\code{\slshape (initialization)} & n/a & read, transform / chain
\end{tabular}
\end{center}
\caption{\it The read, write, transform, and evaluate actions and
  periodicities listed in the last column correspond to the \Stan
  program blocks in the first column.  The middle column indicates
  whether the block allows statements.  The last row indicates that 
  parameter initialization requires a read and transform operation
  applied once per chain.}%
\label{block-actions.figure}
\end{figure}
%
\begin{figure}[t]
\begin{center}
\begin{tabular}{ccc|l}
{\it Params} & {\it Log Prob} & {\it Print} & {\it Declare In}
\\\hline \hline
$+$ & $+$ & $+$ & \code{transformed parameters}
\\
$+$ & $+$ & $-$ & {\it local in}\ \code{model}
\\
$+$ & $-$ & $-$ & {\it local in}\ \code{generated quantities}
\\
$+$ & $-$ & $+$ & \code{generated quantities}
\\
$-$ & $-$ & $+$ & \code{generated quantities}$^*$
\\
$-$ & $\pm$ & $-$ & {\it local in}\ \code{transformed data}
\\
$-$ & $+$ & $+$ & \code{transformed data} {\it and}\ \code{generated quantities}$^*$
\end{tabular}
\end{center}
\caption{\it This table indicates where variables that are not basic
  data or parameters should be declared, based on whether it is
  defined in terms of parameters, whether it is used in the log
  probability function defined in the model block, and whether it is
  printed.  The two lines marked with asterisks ($*$) should not be
  used as there is no need to print a variable every iteration that
  does not depend on the value of any parameters (for information on
  how to print these if necessary, see \refnote{print-var-no-param}
  in this chapter).}%
\label{variable-flowchart.figure}
\end{figure}
%
Another way to look at the variables is in terms of their function.
To decide which variable to use, consult the charts in 
\reffigure{variable-flowchart}.  The last line has no
corresponding location, as there is no need to print a variable
every iteration that does not depend on parameters.%
%
\footnote{It is possible to print a variable every iteration that does
  not depend on parameters --- just define it (or redefine it if it is
  transformed data) in the \code{generated quantities} block.\label{print-var-no-param.footnote}}
%
The rest of this chapter provides full details on when and how the
variables and statements in each block are executed.
%


\section{Statistical Variable Taxonomy}

%
\begin{figure}
\begin{center}
\begin{tabular}{l|l}
{\it Variable Kind} & {\it Declaration Block}
\\ \hline\hline
% constants & \code{transformed data}
% \\ \hline
unmodeled data & \code{data}, \code{transformed data}
\\ 
modeled data & \code{data}, \code{transformed data}
\\ \hline
missing data & \code{parameters}, \code{transformed parameters}
\\
modeled parameters & \code{parameters}, \code{transformed parameters}
\\
unmodeled parameters & \code{data}, \code{transformed data}
\\[2pt] \hline
generated quantities & \code{transformed data}, \code{transformed parameters}, 
\\ 
& \code{generated quantities}
\\ \hline\hline
loop indices & loop statement
\\ 
\end{tabular}
\end{center}
\caption{\it Variables of the kind indicated in the left column must
 be declared in one of the blocks declared in the right
 column.}\label{variable-kinds.figure}
\end{figure}
%
\cite[p.~366]{GelmanHill:2007} provides a taxonomy of the kinds of
variables used in Bayesian models.  \reffigure{variable-kinds} contains
Gelman and Hill's taxonomy along with a missing-data kind along with
the corresponding locations of declarations and definitions in \Stan.

Constants can be built into a model as literals, data variables, or
as transformed data variables.  If specified as variables, their
definition must be included in data files.  If they are specified as
transformed data variables, they cannot be used to specify the sizes
of elements in the \code{data} block.

The following program illustrates various variables kinds, listing the
kind of each variable next to its declaration.
%
\begin{quote}
\begin{Verbatim}[fontsize=\small]
data {
  int<lower=0> N;           // unmodeled data
  real y[N];                // modeled data
  real mu_mu;               // config. unmodeled param
  real<lower=0> sigma_mu;   // config. unmodeled param
}
transformed data {
  real<lower=0> alpha;      // const. unmodeled param
  real<lower=0> beta;       // const. unmodeled param
  alpha <- 0.1;       
  beta <- 0.1;
} 
parameters {
  real mu_y;                // modeled param
  real<lower=0> tau_y;      // modeled param
} 
transformed parameters {
  real<lower=0> sigma_y;    // derived quantity (param)
  sigma_y <- pow(tau_y,-0.5);
}
model {
  tau_y ~ gamma(alpha,beta);
  mu_y ~ normal(mu_mu,sigma_mu);
  for (n in 1:N)
    y[n] ~ normal(mu_y,sigma_y);
}
generated quantities {
  real variance_y;       // derived quantity (transform)
  variance_y <- sigma_y * sigma_y; 
}
\end{Verbatim}
\end{quote}
%  from generated_quantities
%  for (n in 1:N)
%    y_variance[n] <- sigma_y rand_normal(mu_y,sigma_y);
%
In this example, \code{y[N]} is a modeled data vector.  Although it is
specified in the \code{data} block, and thus must have a known value
before the program may be run, it is modeled as if it were generated
randomly as described by the model.  

The variable \code{N} is a typical example of unmodeled data.  It is
used to indicate a size that is not part of the model itself.

The other variables declared in the data and transformed data block are
examples of unmodeled parameters, also known as hyperparameters.
Unmodeled parameters are parameters to probability densities that are
not themselves modeled probabilistically.  In \Stan, unmodeled
parameters that appear in the \code{data} block may be specified on a
per-model execution basis as part of the data read.  In the above
model, \code{mu\_mu} and \code{sigma\_mu} are configurable unmodeled
parameters.  

Unmodeled parameters that are hard coded in the model must be declared
in the \code{transformed data} block.  For example, the unmodeled
parameters \code{alpha} and \code{beta} are both hard coded to the
value 0.1.  To allow such variables to be configurable based on data
supplied to the program at run time, they must be declared in the
\code{data} block, like the variables \code{mu\_mu} and
\code{sigma\_mu}.

This program declares two modeled parameters, \code{mu} and
\code{tau\_y}.  These are the location and precision used in the normal
model of the values in \code{y}.  The heart of the model will be
sampling the values of these parameters from their posterior
distribution.

The modeled parameter \code{tau\_y} is transformed from a precision to
a scale parameter and assigned to the variable \code{sigma\_y} in the
\code{transformed parameters} block. Thus the variable \code{sigma\_y}
is considered a derived quantity --- its value is entirely determined
by the values of other variables.  

The \code{generated quantities} block defines a value
\code{variance\_y}, which is defined as a transform of the scale or
deviation parameter \code{sigma\_y}.  It is defined in the generated
quantities block because it is not used in the model.  Making it
a generated quantity allows it to be monitored for convergence (being
a non-linear transform, it will have different autocorrelation and
hence convergence properties than the deviation itself).  

In later versions of Stan which have random number generators for
the distributions, the \code{generated quantities} block will be
usable to generate replicated data for model checking.

Finally, the variable \code{n} is used as a loop index in the
\code{model} block.  


\section{Program Block: \code{data}}

The rest of this chapter will lay out the details of each block in
order, starting with the \code{data} block in this section.

\subsection{Variable Reads and Transformations}

The \code{data} block is for the declaration of variables that are
read in as data.  With the current model executable, each Markov chain
of samples will be executed in a different process, and each such
process will read the data exactly once.%
%
\footnote{With multiple threads, or even running chains sequentially
  in a single thread, data could be read only once per set of
  chains. \Stan was designed to be thread safe and future versions 
  will provide a multithreading option for Markov chains.\label{thread.footnote}}
%

Data variables are not transformed in any way.  The format for data
files is provided in \refchapter{dump}.

\subsection{Statements}

The \code{data} block does not allow statements.

\subsection{Variable Constraint Checking}

Each variable's value is validated against its declaration as it is
read.  For example, if a variable \code{sigma} is declared as
\code{real<lower=0>}, then trying to assign it a negative value will raise
an error.  As a result, data type errors will be caught as early as
possible.  Similarly, attempts to provide data of the wrong size for a
compound data structure will also raise an error.


\section{Program Block: \code{transformed data}}

The \code{transformed data} block is for declaring and defining
variables that do not need to be changed when running the program.  

\subsection{Variable Reads and Transformations}

For the \code{transformed data} block, variables are all declared in
the variable declarations and defined in the statements.  There is no
reading from external sources and no transformations performed.

Variables declared in the \code{data} block may be used to declare
transformed variables.

\subsection{Statements}

The statements in a \code{transformed data} block are used to define
(provide values for) variables declared in the \code{transformed data}
block. Assignments are only allowed to variables declared in the
\code{transformed data} block.

These statements are executed once, in order, right after the data is
read into the data variables.  This means they are executed once per
chain (though see \refnote{thread} in this chapter).

Variables declared in the \code{data} block may be used in statements
in the \code{transformed data} block.

\subsubsection{Restriction on Operations in \code{transformed data}}

The statements in the transformed data block are designed to be
executed once and have a deterministic result.  Therefore, log
probability is not accumulated and sampling statements may not be
used.  Random number generating functions are also prohibited.

\subsection{Variable Constraint Checking}

Any constraints on variables declared in the \code{transformed data}
block are checked after the statements are executed.  If any defined
variable violates its constraints, \Stan will halt with a diagnostic
error message.


\section{Program Block: \code{parameters}}

The variables declared in the \code{parameters} program block
correspond directly to the variables being sampled by \Stan's samplers
(\HMC and \NUTS).  From a user's perspective, the parameters in the
program block \emph{are} the parameters being sampled by \Stan.  

Variables declared as parameters cannot be directly assigned values.
So there is no block of statements in the \code{parameters} program
block.  Variable quantities derived from parameters may be declared in
the \code{transformed parameters} or \code{generated quantities} blocks,
or may be defined as local variables in any statement blocks following
their declaration.

There is a substantial amount of computation involved for parameter
variables in a \Stan program at each leapfrog step within the
\HMC or \NUTS samplers, and a bit more computation along with writes
involved for saving the parameter values corresponding to a sample.

\subsection{Constraining Inverse Transform}

Stan's two samplers, standard Hamiltonian Monte Carlo (\HMC) and the
adaptive No-U-Turn sampler (\NUTS), are most easily (and often most
effectively) implemented over a multivariate probability density that
has support on all of $\reals^n$.  To do this, the parameters
defined in the \code{parameters} block must be transformed so they are
unconstrained. 

In practice, the samplers keep an unconstrained parameter vector in
memory representing the current state of the sampler.  The model
defined by the compiled \Stan program defines an (unnormalized) log
probability function over the unconstrained parameters.  In order to
do this, the log probability function must apply the inverse transform
to the unconstrained parameters to calculate the constrained
parameters defined in \Stan's \code{parameters} program block.  The
log Jacobian of the inverse transform is then added to the accumulated
log probability function.  This then allows the \Stan model to be
defined in terms of the constrained parameters.

In some cases, the number of parameters is reduced in the
unconstrained space.  For instance, a $K$-simplex only requires $K-1$
unconstrained parameters, and a $K$-correlation matrix only requires
${K \choose 2}$ unconstrained parameters.  This means that the
probability function defined by the compiled \Stan program may have
fewer parameters than it would appear from looking at the declarations
in the \code{parameters} program block. 

The probability function on the unconstrained parameters is defined in
such a way that the order of the parameters in the vector corresponds
to the order of the variables defined in the \code{parameters} program
block.  The details of the specific transformations are provided in
\refchapter{variable-transforms}.

\subsection{Gradient Calculation}

Hamiltonian Monte Carlo requires the gradient of the (unnormalized)
log probability function with respect to the unconstrained parameters
to be evaluated during every leapfrog step.  There may be one leapfrog
step per sample or hundreds, with more being required for models with
complex posterior distribution geometries.

Gradients are calculated behind the scenes using \Stan's algorithmic
differentiation library.  The time to compute the gradient does not
depend directly on the number of parameters, only on the number of
subexpressions in the calculation of the log probability.  This
includes the expressions added from the transforms' Jacobians.  

The amount of work done by the sampler does depend on the number of
unconstrained parameters, but this is usually dwarfed by the gradient
calculations.

\subsection{Writing Samples}

In the basic \Stan compiled program, the values of variables are
written to a file for each sample.  The constrained versions of the
variables are written, again in the order they are defined in the
\code{parameters} block.  In order to do this, the transformed
parameter, model, and generated quantities statements must be
executed.  


\section{Program Block: \code{transformed parameters}}

The \code{transformed parameters} program block consists of optional
variable declarations followed by statements.  After the statements
are executed, the constraints on the transformed parameters are
validated.  Any variable declared as a transformed parameter is part
of the output produced for samples.

Any variable that is defined wholly in terms of data or transformed
data should be declared and defined in the transformed data block.
Defining such quantities in the transformed parameters block is legal,
but much less efficient than defining them as transformed data.

\section{Program Block: \code{model}}

The \code{model} program block consists of optional variable
declarations followed by statements.  The variables in the model block
are local variables and are not written as part of the output.  

Local variables may not be defined with constraints because there is
no well-defined way to have them be both flexible and easy to
validate.

The statements in the model block typically define the model.  This is
the block in which probability (sampling notation) statements are
allowed.  These are typically used when programming in the \BUGS idiom
to define the probability model.  


\section{Program Block: \code{generated quantities}}

The \code{generated quantities} program block is rather different than
the other blocks.  Nothing in the generated quantities block affects
the sampled parameter values.  The block is executed only after a
sample has been generated.  

Among the applications of posterior inference that can be coded in the
generated quantities block are
%
\begin{itemize}
\item forward sampling to generate simulated data for model testing,
\item generating predictions for new data,
\item calculating posterior event probabilities, including multiple comparisons,
  sign tests, etc.,
\item calculating posterior expectations,
\item transforming parameters for reporting,
\item applying full Bayesian decision theory,
\item calculating log likelihoods, deviances, etc.\ for model comparison.
\end{itemize}
%
Forward samples, event probabilities and statistics may all be
calculated directly using plug-in estimates.  \Stan automatically
provides full Bayesian inference by producing samples from the
posterior distribution of any calculated event probabilities,
predictions, or statistics.  See \refchapter{bayesian} for more
information on Bayesian inference.

Within the generated quantities block, the values of all other variables
declared in earlier program blocks (other than local variables) are
available for use in the generated quantities block.

It is more efficient to define a variable in the generated quantities
block instead of the transformed parameters block.  Therefore, if a
quantity does not play a role in the model, it should be defined in
the generated quantities block.  

After the generated quantities statements are executed, the constraints
on the declared generated quantity variables are validated.

All variables declared as generated quantities are printed as part of
the output. 

\chapter{Modeling Language Syntax}

\noindent
This chapter defines the basic syntax of the \Stan modeling language
using a Backus-Naur form (\BNF) grammar plus extra-grammatical
constraints on function typing and operator precedence and
associativity.


\section{BNF Grammars}

\subsection{Syntactic Conventions}

In the following \BNF grammars, literal strings are indicated in
single quotes (\code{'}).  Grammar non-terminals are unquoted strings.
A prefix question mark (\code{?})  indicates optionality.  A postfixed
Kleene star (\code{*}) indicates zero or more occurrences.

\subsection{Programs}

{\small
\begin{Verbatim}[fontsize=\small]
program ::= ?data ?tdata ?params ?tparams model ?generated

data ::= 'data' var_decls
tdata ::= 'transformed data' var_decls_statements
params ::= 'parameters' var_decls
tparams ::= 'transformed parameters' var_decls_statements
model ::= 'model' statement
generated ::= 'generated quantities' var_decls_statements

var_decls ::= '{' var_decl* '}'
var_decls_statements ::= '{' var_decl* statement* '}'
\end{Verbatim}
}

\subsection{Variable Declarations}

{
\small
\begin{Verbatim}[fontsize=\small]
var_decl ::= var_type variable ?dims

var_type ::= 'int' range_constraint
           | 'real' range_constraint
           | 'vector' range_constraint '[' expression ']'
           | 'ordered' '[' expression ']'
           | 'positive_ordered' '[' expression ']'
           | 'simplex' '[' expression ']'
           | 'unit_vector' '[' expression ']'
           | 'row_vector' range_constraint '[' expression ']'
           | 'matrix' range_constraint '[' expression ',' expression ']'
           | 'cholesky_factor_cov' '[' expression ?(',' expression) ']'
           | 'corr_matrix' '[' expression ']'
           | 'cov_matrix' '[' expression ']'

range_constraint ::= ?('<' range '>')

range ::= 'lower' '=' expression ',' 'upper' = expression
        | 'lower' '=' expression
        | 'upper' '=' expression

dims ::= '['  expression (',' expression)*  ']'

variable ::= identifier

identifier ::= [a-zA-Z] [a-zA-Z0-9_]*
\end{Verbatim}
}

\subsection{Expressions}

{
\small
\begin{Verbatim}[fontsize=\small]
expression ::= numeric_literal
             | variable
             | expression infixOp expression
             | prefixOp expression
             | expression postfixOp
             | expression '[' expressions ']'
             | function_literal '(' ?expressions ')'
             | '(' expression ')'

expressions ::= expression
              | expression ',' expressions

numeric_literal ::= int_literal | real_literal

integer_literal ::= [0-9]*

real_literal ::= [0-9]* ?('.' [0-9]*) ?exp_literal
                  
exp_literal ::= ('e' | 'E') integer_literal

function_literal ::= identifier
\end{Verbatim}
}

\subsection{Statements}

{
\small
\begin{Verbatim}[fontsize=\small]
statement 
::= lhs '<-' expression ';'
  | expression '~' identifier '(' ?expressions ')' ?truncation ';'
  | 'if' '(' expression ')' statement
    ('else' 'if' '(' expression ')' statement)*
    ?('else' statement)
  | 'while' '(' expression ')' statement
  | 'for' '(' identifier 'in' expression ':' expression ')' statement
  | '{' var_decl* statement+ '}'
  | 'print' '(' (expression | string_literal)* ')'
  | ';'

string_literal ::= '"' char* '"'

truncation ::= 'T' '[' ?expression ',' ?expression ']'

lhs ::= identifier
      | identifier '[' expressions ']'

\end{Verbatim}
%
}

\section{Extra-Grammatical Constraints}

\subsection{Type Constraints}

A well-formed \Stan program
must satisfy the type constraints imposed by functions and
distributions.  For example, the binomial distribution requires an
integer total count parameter and integer variate and when truncated
would require integer truncation points.  If these constraints are
violated, the program will be rejected during parsing with an error
message indicating the location of the problem.  For information on
argument types, see \refpart{built-in-functions}.

\subsection{Operator Precedence and Associativity}

In the \Stan grammar provided in this chapter, the expression \code{1
  + 2 * 3} has two parses.  As described in
\refsection{operator-precedence}, \Stan disambiguates between the
meaning $1 + (2 \times 3)$ and the meaning $(1 + 2) \times 3$ based on
operator precedences and associativities. 

\subsection{Forms of Numbers}

Integer literals longer than one digit may not start with 0 and real
literals cannot consist of only a period or only an exponent.

\subsection{Conditional Arguments}

Both the conditional if-then-else statement and while-loop statement
require the expression denoting the condition to be a primitive type,
integer or real.
