Temporary files and disk usage
===============================================================================

This page aims to characterize intermediate files created by the hipscat-import 
catalog creation process. Most users are going to be ok with setting the ``tmp_dir``
and not thinking much more about it.

This page will mostly be useful for more contrained use cases, and when users have some
understanding when to use local vs remote object storage. This can also be a guide for
how to allocate resources for an import process, when the files will be large.

Overview
-------------------------------------------------------------------------------

In catalog creation, we currently have 3 different kinds of intermediate files:

- dask worker space (``dask_tmp``)

  - this is something dask needs. we don't really control this too much, nor should we.
  - when dask workers run out of memory, they spill their state to disk to be resumed by another worker.

- sharded parquet files (``tmp_dir``)

  - we split the input files by their destination pixel into smaller parquet files. 
    these small files are then concatenated into the final partitioned parquet files.

- intermediate resume files (``resume_tmp``)

  - log that some tasks are complete and their results. useful in the case of externally-killed jobs.

If you don't provide any of these arguments, we'll create an ``intermediate`` directory
under the ``output_path`` to shove temporary files. We also try to make reasonable
guesses if you only provide a subset of the arguments. If you want to control each
set of temporary files, then you should specify all three arguments.

There are also a few types of storage, and I'm going to give them arbitrary names:

- COLD

  - cloud object store

    - long-lived, potentially slow reads, potentially expensive reads and at-rest

  - big long term disk

    - something like a raid array. long-lived, but ideally faster and cheaper than object store.

- WARM

  - small regular local disk
  - something like a home directory, which might not have much room but is long-term

- HOT

  - short-term local disk

    - when running in cloud or HPC, this is the disk space that is associated with 
      the running instance. it generally has less space than your larger data allocations, 
      only lives for the duration of the job.
    - or when running on a regular linux system, the /tmp/ directory.

Thinking about the handful of environments we have run on, we have different needs for 
different kinds of intermediate files:

==================================  ============  =========== ==============
environment style                   ``dask_tmp``  ``tmp_dir`` ``resume_tmp``
==================================  ============  =========== ==============
vanilla linux / developer laptop    COLD          COLD        COLD
github actions / unit tests         HOT           HOT         HOT
HPC (e.g. PSC)                      HOT           COLD        WARM
Cloud (e.g. AWS)                    HOT           COLD        WARM
==================================  ============  =========== ==============

Some more explanation:

- AFAIK, dask worker data is never used between executions of the same job, 
  so the files need not be long-lived. they can always live in the HOT zone.
- the sharded parquet files have the potential to exceed the disk limits of either 
  HOT or WARM, and so should go where you have the most disk space available to you. 
  in addition, you want them to stick around between jobs if your job is killed, so they 
  shouldn't be on short-term local disk, even if you have 2T of it.
- resume files are going to be smaller than the sharded parquet files, and there are 
  going to be many times fewer of them. because they're small, you don't want to write 
  them to cloud object storage. however, you want them to stick around between jobs 
  (because that's the whole point of them).
- if you're ok not being able to resume your job, then you should totally have the 
  option to use HOT storage for sharded parquet and resume files, even in PSC/AWS scenarios. 
  you do you.

What's happening when
-------------------------------------------------------------------------------

The hipscat-import catalog creation process generates a lot of temporary files. Some find this 
surprising, so we try to provide a narrative of what's happening and why.

Planning stage
...............................................................................

At this stage, generally the only file that is written out is ``<resume_tmp>/input_paths.txt``
and contains the paths of all the input files. This is to make sure that resumed instances
of the job are using the same input files.

The final output directory has been created, but is empty until "Reducing".

Mapping stage
...............................................................................

In this stage, we're reading each input file and building a map of how many objects are in 
each high order pixel. For each input file, once finished, we will write a binary file with 
the numpy array representing the number of objects in each pixel. 

.. tip::
    For ``highest_healpix_order=10``, this binary file is 96M. If you know your data will be 
    partitioned at a lower order (e.g. order 7), using the lower order in the arguments 
    can improve runtime and disk usage of the pipeline.

Binning stage
...............................................................................

In this stage, we're reading each of the numpy binary files from above, combining into a 
single histogram, and doing Some Math on it. 
We write out one new histogram file, and remove the intermediate histograms.

Splitting stage
...............................................................................

Here's where it starts to get serious.

In this stage, we re-read *all* the original input files. 
We have calculated our desired partitioning, and so we split each input file into shards 
based on their destination pixel. These are written out as parquet files. 

Say we have 1,000 input files and 4,000 destination pixels. We chunk the reads of input 
files to avoid out of memory issues, so say there are 1-30 chunks per file. We will 
have anywhere from 4,000 to 120,000,000 parquet shard files. 
There are levers to tune this, but since these are just intermediate files, 
it's likely not useful, unless you plan to run the import dozens of times over similar input data. 

If you're interested in tuning these levers, please reach out! We'd love to help! :doc:`/guide/contact`.

The total size on disk of the sharded parquet files is likely going to be within 
1-25% of the final catalog size on disk. You might expect it to be the same as the 
final catalog size, since it's the same data and it's all parquet. 
There are some subtleties that could cause a discrepancy:

- Additional file overhead of parquet metadata (more small files means more overhead)
- Different compression rates for small sets of points vs larger sets

Reducing stage
...............................................................................
In this stage, we're taking the sharded parquet files from the previous stage and combining 
them into a single parquet file per destination pixel.

For the example, we will have 4,000 tasks, each of which will concatenate the shard files 
for a single pixel into 4,000 final files. 
As the final files are written, the parquet shards are removed. 
This is when storage shifts from intermediate files to the real output files.

Finishing stage
...............................................................................

Here, we will write out a few additional final files (e.g. ``catalog_info.json``, ``_metadata``).
Additionally, we will clean up any straggling intermediate resume files. 
This includes all text log files, and the summed histogram file. 
After this stage, we should have zero intermediate files.

Approximate file sizes
-------------------------------------------------------------------------------

We discuss the size of intermediate files for a sample hipcat-import pipeline, 
using the publicly available TIC catalog (:doc:`public/tic`).

We have 90 input files, all gzipped csvs. They range from 24M to 10G, and total 389G.

Total directory size (all intermediate files) and size in final output directory after each stage:

============= ====== ======================= =========== =========================
Stage         Intermediate                   Output
------------- ------------------------------ -------------------------------------
Stage         Size   Description             Size        Description
============= ====== ======================= =========== =========================
Planning      7.8 K  a single text log       0           it's created, but empty.
Mapping       8.5 G  all numpy histograms    0
Binning       97 M   one histogram, few logs 0
Splitting     535 G  246,227 files.
                     246,220 sharded parquet 0
Reducing      97 M   one histogram, few logs 439 G       3,768 parquet files
Finishing     0                              439 G       adds root-level metadata
============= ====== ======================= =========== =========================

Input vs output size
-------------------------------------------------------------------------------

As alluded to in the `Splitting` section above, the total on-disk size of the 
final catalog can be very different from the on-disk size of the input files.

In our internal testing, we converted a number of different kinds of catalogs, 
and share some of the results with you, to give some suggestion of the disk requirements
you may face when converting your own catalogs to hipscat format.

============= =============== =========== =============== =========================
Catalog	      Input size (-h) Input size  Hipscatted size Ratio
============= =============== =========== =============== =========================
allwise       1.2T             1196115700       310184460   0.26 (a lot smaller)
neowise	      3.9T             4177447284      4263269112   1.02 (about the same)
tic           389G              407367196       463070176   1.14 (a little bigger)
sdss (object) 425G              445204544       255775912   0.57 (a little smaller)
zubercal      8.4T             8991524224     11629945932   1.29 (a little bigger)
============= =============== =========== =============== =========================

Notes:

- allwise, neowise, and tic were all originally compressed CSV files.
- sdss was originally a series of fits files
- zubercal was originally 500k parquet files, and is reduced in the example to 
  around 70k hipscat parquet files.
