.. currentmodule:: yaw

Concepts
========

The two most important design concepts of `yet_another_wizz` are
:ref:`caching<caching>` and spatial :ref:`patches<patches>`. Caching the input
catalog data is important to improve the code performance and memory management,
patches are important to compute uncertainties and covariances.

**Choosing appropriate patches can have significant performance impacts.**
The best choice for the number of patches depends on the data samples. The
optimum performance is achieved with a small number of patches that can be
optimally distributed on the available CPUs. When measuring correlations, each
CPU needs to load two patches simultaneously into memory (together with some
overhead from building a binary search tree).

.. note::
    The number of patches should be large enough to get a robust covariance
    estimate from spatial resampling.

.. figure:: /_static/patches.png
    :figwidth: 100%
    :alt: Example patches

    Example of spatial patches generated by `yet_another_wizz` for a data set
    with stellar masks.


.. _caching:

Caching
-------

Caching is used to speed up parallel computations and reduce the memory usage
of the code. Every input data set is automatically cached when creating a new
:obj:`~yaw.Catalog` object, which are used for the correlation function
measurements.

Therefore, every catalog instance is tied to a cache directory on disk, where
the data is stored in :obj:`~yaw.catalog.patch.Patch` es and read back on demand.
Therefore it is beneficial to choose a cache location on a fast device, such as
an SSD, RAID system, or file systems optimised for parallel processing.

.. tip::

    On UNIX-like systems, using a RAM disk maybe be an fast option if the
    system has sufficient spare memory.

When created, an input catalog is split into :ref:`patches<patches>`, which
represent smaller chunks of the input data. The cache directory of a catalog
typically as the following structure::

    [cache_directory]/
       ├╴ patch_ids.bin     # list of patch IDs
       ├╴ patch_0/          # first patch with ID=0
       │    ├╴ data.bin     # chunk of the catalog data
       │    └╴ meta.yml     # meta data describing the chunk
       ├╴ patch_1/
       │    └╴ ...
       ├╴ patch_2/
       │  ...
       └╴ patch_N/



.. _patches:

Patches
-------

Spatial patches serve two purposes in `yet_another_wizz`:

1) Dividing the an input catalog in spatial regions that can be used for
   jackknife resampling to estimate uncertainties when measuring correlations.
2) Dividing the an input catalog in smaller chunks that can be loaded on-demand
   to reduce the memory footprint of the application.

Both should be taken into account when designing the patch layout of the input
catalogs.

.. caution::
    It is crucial to use exactly the same patches for all input catalogs when
    measuring correlations with :func:`~yaw.autocorrelate` or
    :func:`~yaw.crosscorrelate`. This ensures that the patches correctly
    identified and properly aligned when running the pair counting.


A patch is defined by a center point coordinate and a radius, the minimum
enclosing circle given the center point. There are three ways to define a set
of patches for a catalog (see e.g. :obj:`Catalog.from_file`):

1) Providing a list of center coordinates for each patch:

   Data points are assigned to the nearest patch center, similar to a Voronoi
   diagram. This method is useful to ensure that multiple catalogs share the
   same patch centers, as is required for correlation measurements.

2) Providing patch indices alongside the input data:

   The patch indices are used to assign data points to patches. It is assumed
   that the set of patch indices is contiguous and counting from 0.

3) Automatically generating the patches:

   A sparse sample of the input catalog is read and `treecorr`_ is used to
   compute a spcified number of patch centers. Afterwards, use the patch centers
   and assign objects to the nearest center as above.

.. note::
    Patch centers created with treecorr are not deterministic and will differ
    every time they are constructed.


.. _treecorr: https://rmjarvis.github.io/TreeCorr/_build/html/patches.html
