###############
Code comparison
###############

Here we document comparisons between the analysis produced using CWInPy compared with the `LALSuite
<https://git.ligo.org/lscsoft/lalsuite>`_ code ``lalapps_pulsar_parameter_estimation_nested``, which
is described in Pitkin et al., `arXiv:1705.08978v1 <https://arxiv.org/abs/1705.08978v1>`_.

The codes will be run on identical data and in configurations that are as close as possible. Links
to the various comparisons are given below:

.. toctree::
   :maxdepth: 1

   single-detector-noise-only
   single-detector-software-injection-linear
   single-detector-software-injection-circular
   multi-detector-noise-only
   multi-detector-software-injection-linear
   multi-detector-software-injection-circular
   single-detector-noise-only-two-harmonics
   single-detector-software-injection-two-harmonics
   single-detector-O1-data
   single-detector-O1-data-restricted-prior
   single-detector-O1-data-hardware-injection
   multi-detector-O1-data
   multi-detector-O1-data-hardware-injection

Posterior credible interval checks
==================================

Another test of the code is to check that the posterior credible intervals resulting from any
analysis correctly ascribe probability, i.e., that they are "well calibrated" [1]_. To do this one
can create a set of simulated signal with true parameters drawn from a particular prior, and then
use the code to sample the posterior probability distribution over the parameter space for each
parameter using the *same* prior. Using the one-dimensional marginalised posteriors for each
parameter, and each simulation, one can then find the credible interval in which the known true
signal parameter lies. If the credible intervals are correct you would expect, e.g., to find the
true parameter in the 1% credible interval for 1% of the simulations, the true parameter within the
50% credible interval for 50% of the simulations, etc. A plot of the credible interval versus the
percentage of true signal values found within that credible interval is known within the
gravitational-wave community colloquially as a "PP plot" (see, e.g., Section VC of [2]_).

These tests have been performed in the output of ``cwinpy_knope`` by generating a set of simulated
signals (using the :class:`cwinpy.knope.testing.KnopePPPlotsDAG`) to be analysed. After all the
individual simulations have been analysed a PP plot is generated using :meth:`cwinpy.knope.testing.
generate_pp_plots` (which itself uses functions from `bilby
<https://lscsoft.docs.ligo.org/bilby/index.html>`_ and `bilby_pipe
<https://lscsoft.docs.ligo.org/bilby_pipe/index.html>`_).

Single harmonic signal
----------------------

We produce PP plots for the case of a signal from the :math:`l=m=2` mass quadrupole of a pulsar,
where emission would be at twice the rotation frequency and defined by the parameters :math:`h_0`,
:math:`\phi_0`, :math:`\iota` and :math:`\psi`. A Python file to run such an analysis for 250
simulated signals using :class:`~cwinpy.knope.testing.KnopePPPlotsDAG` is shown below_. This also
shows the priors used for the generation of signal parameters and their recovery.

.. note::

   When drawing parameters from the :math:`h_0` prior a maximum cut-off (``maxamp`` in the code)
   that is lower than the upper range of the prior is used. This is to ensure that posteriors are
   not truncated by the upper end of the prior in this case. However, this should not bias the
   recovered credible intervals due to the :math:`h_0` prior being uniform and extended well
   above the ``maxamp`` value.

.. thumbnail:: ppplot_2f.png
   :width: 600px
   :align: center

The distributions of signal-to-noise ratios for these simulations is:

.. thumbnail:: snrs_2f.png
   :width: 400px
   :align: center

Dual harmonic signal
--------------------

We produce PP plots for the case of a signal from the :math:`l=2, m=1,2` mass quadrupole of a
pulsar, where emission would be at both once and twice the rotation frequency and defined by the
parameters :math:`C_{12}`, :math:`C_{22}`, :math:`\Phi_{12}`, :math:`\Phi_{22}`, :math:`\iota` and
:math:`\psi`. A Python file to run such an analysis for 250 simulated signals using
:class:`~cwinpy.knope.testing.KnopePPPlotsDAG` is shown below_. This also shows the priors used for
the generation of signal parameters and their recovery.

.. note::

   When drawing parameters from the :math:`C_{12}` and :math:`C_{22}` priors a maximum cut-off
   (``maxamp`` in the code) that is lower than the upper range of the prior is used. This is to
   ensure that posteriors are not truncated by the upper end of the prior in this case. However,
   this should not bias the recovered credible intervals due to the :math:`C_{12}` and
   :math:`C_{22}` priors being uniform and extended well above the ``maxamp`` value.

.. thumbnail:: ppplot_1f2f.png
   :width: 600px
   :align: center

The distributions of signal-to-noise ratios for these simulations is:

.. thumbnail:: snrs_1f2f.png
   :width: 400px
   :align: center

.. _below:

.. literalinclude:: scripts/pptest_one_harmonic.py
   :language: python
   :lines: 1-29,33-42,45-48,50-51

.. literalinclude:: scripts/pptest_two_harmonics.py
   :language: python
   :lines: 1-45,49-58,61-64,66-67

Comparison References
---------------------

.. [1] `A. P. Dawid
   <https://www.tandfonline.com/doi/abs/10.1080/01621459.1982.10477856>`_, *Journal of the
   American Statistical Association*, **77**, 379 (1982).
.. [2] `J. Veitch et al.
   <https://ui.adsabs.harvard.edu/abs/2015PhRvD..91d2003V/abstract>`_,
   *PRD*, **91**, 042003 (2015)
