.. _linear:


===========================
Example: Linear Map Choose QoIs (volume / bin_size / large)
===========================

.. warning::

    The documentation for this example is NOT up to date. However, the code in
    the ``examples/`` directory is up to date.

This example generates uniform random samples in the unit hypercube and
corresponding QoIs (data) generated by a linear map Q.  We then calculate the
gradients using an RBF scheme and use the gradient information to choose the
optimal set of 2 (3, 4, ... Lambda_dim) QoIs to use in the inverse problem.

Every real world problem requires special attention regarding how we choose
*optimal QoIs*.  This set of examples (examples/sensitivity/linear) covers
some of the more common scenarios using easy to understand linear maps.

In this *volume_binsize_large* example we choose *optimal QoIs* to be the set of 
QoIs of size Lambda_dim that produces the smallest support of the inverse 
solution, assuming we define the uncertainty in our data to be fixed, i.e.,
independent of the range of data maesured for each QoI (bin_size).

Import the necessary modules::


    import numpy as np
    import bet.sensitivity.gradients as grad
    import bet.sensitivity.chooseQoIs as cQoI
    import bet.calculateP.simpleFunP as simpleFunP
    import bet.calculateP.calculateP as calculateP
    import bet.postProcess.postTools as postTools
    import bet.Comm as comm

Let Lambda be a 5 dimensional hypercube::

    Lambda_dim = 10
    Data_dim = 100
    num_samples = 1E5
    num_centers = 10

Let the map Q be a random matrix of size (Data_dim, Lambda_dim)::

    np.random.seed(0)
    Q = np.random.random([Data_dim, Lambda_dim])

Choose random samples in parameter space to solve the model::

    samples = np.random.random([num_samples, Lambda_dim])
    data = Q.dot(samples.transpose()).transpose()

Calculate the gradient vectors at some subset of the samples.  Here the 
*normalize* argument is set to *False* because we are using *bin_size* to
determine the uncertainty in our data::

    G = grad.calculate_gradients_rbf(samples, data,
        centers=samples[:num_centers, :], normalize=False)

With these gradient vectors, we are now ready to choose an optimal set of
QoIs to use in the inverse problem, based on minimizing the support of the
inverse solution (volume).  The most robust method for this is
:meth:~bet.sensitivity.chooseQoIs.chooseOptQoIs_large which returns the
best set of 2, 3, 4 ... until Lambda_dim.  This method returns a list of
matrices.  Each matrix has 10 rows, the first column representing the
expected inverse volume ratio, and the rest of the columns the corresponding
QoI indices.

Here we set some arguments to speed up the search for optimal QoIs::

    best_sets = cQoI.chooseOptQoIs_large(G, max_qois_return=5,
        num_optsets_return=2, inner_prod_tol=0.9, cond_tol=1E2, volume=True)

We see here the expected volume ratios are small.  This number represents the
expected volume of the inverse image of a unit hypercube in the data space.
With the bin_size definition of the uncertainty in the data, here we expect to
see inverse solutions that have a smaller support (expected volume ratio < 1)
than the original volume of the hypercube in the data space.

This interpretation of the expected volume ratios is only valid for inverting
from a data space that has the same dimensions as the paramter space.  When
inverting into a higher dimensional space, this expected volume ratio is the
expected volume of the cross section of the inverse solution.

At this point we have determined the optimal set of QoIs to use in the inverse
problem.  Now we compare the support of the inverse solution using
different sets of these QoIs.  We set Q_ref to correspond to the center of
the parameter space.  We choose the set of QoIs to consider::

    QoI_indices = [0, 7]

Restrict the data to have just QoI_indices::

    data = data[:, QoI_indices]
    Q_ref = Q[QoI_indices, :].dot(0.5 * np.ones(Lambda_dim))

bin_ratio defines the uncertainty in our data::

    bin_ratio = 0.25

Find the simple function approximation::

    (P,  lam_vol, io_ptr) = calculateP.prob(samples=samples, data=data,
        rho_D_M=d_distr_prob, d_distr_samples=d_distr_samples)

Sort samples by highest probability density and find how many samples lie in
the support of the inverse solution.  With the Monte Carlo assumption, this
also tells us the approximate volume of this support::

    percentile = 1.0
    (num_samples, P_high, samples_high, lam_vol_high, data_high) =\
        postTools.sample_highest_prob(top_percentile=percentile, P_samples=P,
        samples=samples, lam_vol=lam_vol,data = data,sort=True)

Print the number of samples that make up the highest percentile percent
samples and ratio of the volume of the parameter domain they take up
::

    if comm.rank == 0:
        print (num_samples, np.sum(lam_vol_high))
