======
CPrior
======

.. image::  https://travis-ci.com/guillermo-navas-palencia/cprior.svg?branch=master
   :target: https://travis-ci.com/guillermo-navas-palencia/cprior

.. image::  https://codecov.io/gh/guillermo-navas-palencia/cprior/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/guillermo-navas-palencia/cprior

.. image::  https://img.shields.io/github/license/guillermo-navas-palencia/cprior
   :target: https://img.shields.io/github/license/guillermo-navas-palencia/cprior

.. image:: https://img.shields.io/pypi/v/cprior?color=blue
   :target: https://img.shields.io/pypi/v/cprior?color=blue 

**CPrior** has functionalities to perform Bayesian statistics and running A/B and multivariate testing. CPrior includes several conjugate prior distributions and has been developed focusing on performance, implementing many closed-forms in terms of special functions to obtain fast and accurate results, avoiding Monte Carlo methods whenever possible.

**Website**: http://gnpalencia.org/cprior/


Installation
------------

To install the current release of CPrior on Linux/Windows:

.. code-block:: bash

    pip install cprior

For different OS and/or custom installations, see http://gnpalencia.org/cprior/getting_started.html.

Dependencies
""""""""""""

CPrior has been tested with CPython 3.5, 3.6 and 3.7. It requires:

* mpmath 1.0.0 or later. Website: http://mpmath.org/
* numpy 1.15.0 or later. Website: https://www.numpy.org/
* scipy 1.0.0 or later. Website: https://scipy.org/scipylib/
* pytest
* coverage

Testing
"""""""
Run all unit tests

.. code-block:: bash

   python setup.py test

Examples
--------

Example: run experiment
"""""""""""""""""""""""

A Bayesian multivariate test with control and 3 variants. Data follows a
Bernoulli distribution with distinct success probability.

1. Generate control and variant models and build experiment. Select stopping rule and threshold (epsilon).

.. code-block:: python
   
   from scipy import stats
   from cprior.models.bernoulli import BernoulliModel
   from cprior.models.bernoulli import BernoulliMVTest
   from cprior.experiment.base import Experiment
   
   modelA = BernoulliModel(name="control", alpha=1, beta=1)
   modelB = BernoulliModel(name="variation", alpha=1, beta=1)
   modelC = BernoulliModel(name="variation", alpha=1, beta=1)
   modelD = BernoulliModel(name="variation", alpha=1, beta=1)

   mvtest = BernoulliMVTest({"A": modelA, "B": modelB, "C": modelC, "D": modelD})

   experiment = Experiment(name="CTR", test=mvtest, stopping_rule="probability_vs_all",
                           epsilon=0.99, min_n_samples=1000, max_n_samples=None)

2. See experiment description

.. code-block:: python

   experiment.describe()
   
.. code-block:: txt

   =====================================================
     Experiment: CTR
   =====================================================
       Bayesian model:                bernoulli-beta
       Number of variants:                         4

       Options:
         stopping rule            probability_vs_all
         epsilon                             0.99000
         min_n_samples                          1000
         max_n_samples                       not set

       Priors:

            alpha  beta
         A      1     1
         B      1     1
         C      1     1
         D      1     1
     -------------------------------------------------
   
3. Generate or pass new data and update models until a clear winner is found. The stopping rule will be updated after a new update.

.. code-block:: python

   with experiment as e:
       while not e.termination:
           data_A = stats.bernoulli(p=0.0223).rvs(size=25)
           data_B = stats.bernoulli(p=0.1128).rvs(size=15)
           data_C = stats.bernoulli(p=0.0751).rvs(size=35)
           data_D = stats.bernoulli(p=0.0280).rvs(size=15)

           e.run_update(**{"A": data_A, "B": data_B, "C": data_C, "D": data_D})

       print(e.termination, e.status)

.. code-block:: txt
   
   True winner B

4. Reporting: experiment summary

.. code-block:: python

   experiment.summary()

.. image:: img/bernoulli_summary.png
  :width: 400

5. Reporting: visualize stopping rule metric over time (updates)

.. code-block:: python

   experiment.plot_metric()

.. image:: img/bernoulli_plot_metric.png

6. Reporting: visualize statistics over time (updates)

.. code-block:: python

   experiment.plot_stats()

.. image:: img/bernoulli_plot_stats.png


Example: basic A/B test
"""""""""""""""""""""""

A Bayesian A/B test with data following a Bernoulli distribution with two
distinct success probability. This example is a simple use case for
CRO (conversion rate) or CTR (click-through rate) testing.

.. code-block:: python

   from scipy import stats

   from cprior.models import BernoulliModel
   from cprior.models import BernoulliABTest

   modelA = BernoulliModel()
   modelB = BernoulliModel()

   test = BernoulliABTest(modelA=modelA, modelB=modelB)

   data_A = stats.bernoulli(p=0.10).rvs(size=1500, random_state=42)
   data_B = stats.bernoulli(p=0.11).rvs(size=1600, random_state=42)

   test.update_A(data_A)
   test.update_B(data_B)

   # Compute P[A > B] and P[B > A]
   print("P[A > B] = {:.4f}".format(test.probability(variant="A")))
   print("P[B > A] = {:.4f}".format(test.probability(variant="B")))

   # Compute posterior expected loss given a variant
   print("E[max(B - A, 0)] = {:.4f}".format(test.expected_loss(variant="A")))
   print("E[max(A - B, 0)] = {:.4f}".format(test.expected_loss(variant="B")))

The output should be the following:

.. code-block::

   P[A > B] = 0.1024
   P[B > A] = 0.8976
   E[max(B - A, 0)] = 0.0147
   E[max(A - B, 0)] = 0.0005