# Output Structure Documentation

This document describes the structure of the output generated by the distributed downloader. The output consists of
several folders and files organized as follows (names correspond to those specified in the configuration file, see
also [example_output](./example_output/)):

## Directory Structure Overview

- [urls_folder](#urls-folder) — Contains partitioned URLs for downloading
- [logs_folder](#logs-folder) — Contains logs generated by the downloader
- [images_folder](#images-folder) — Contains downloaded images and metadata
- [schedules_folder](#schedules-folder) — Contains scheduling information for the downloader
- [profiles_table](#profiles-table) — Table with download speed information for each server
- [ignored_table](#ignored-table) — CSV file listing servers to be ignored by the downloader
- [inner_checkpoint_file](#inner-checkpoint-file) — File storing the current state of the downloader
- [tools_folder](#tools-folder) — Contains output files from pipeline tools

## Detailed Structure

### URLs Folder

This structure is created during the `initialization` step of the pipeline. URLs are first partitioned by server name (
obtained via `urlparse(url).netloc`) and then grouped into batches of `downloader_parameters/batch_size` size.

Each batch is stored as a Parquet file containing the following required columns (additional columns are ignored by the
downloader but may be present):

- `uuid` — Unique internal identifier of the dataset
- `identifier` — URL to the image
- `source_id` — Identification of the image for accessing relevant information from the source
- `license` (optional) — License of the image
- `source` (optional) — Source of the image for licensing purposes
- `title` (optional) — Title of the image for licensing purposes

### Logs Folder

The logs folder contains all logs generated by the downloader pipeline. This section describes the most important files;
other files are typically empty or contain minimal content.

#### Initialization Logs

- `initialization.log` — Log file from the Python script executing the initialization step, containing information about
  performed operations
- `initialization.err` — Log file from the underlying Spark job running the initialization step, containing all
  Spark-generated logs

#### Profiler Logs

The profiler does not generate logs, though `profiler.*` files exist but remain empty.

#### Downloader Logs

This step generates extensive logs organized into separate folders.

**Folder Structure:**

- Folders are created for each batch of workers launched
- The current batch is named `current`
- Previous batches are numbered sequentially (`0001`, `0002`, etc.)

**Within each batch folder:**

- `schedule_creation.err` — Log file for the schedule creation step, containing job IDs of submitted worker "layers" for
  each batch
- Numbered folders (`0000` and forward) — Represent server batches corresponding to the schedule

**Within each server batch folder:**

- Numbered folders (`0000` and forward) — One per "layer" of workers launched for the batch

**Within each worker layer folder:**

- `MPI_downloader_verifier.log` — Verification step log file indicating download completion status
- `MPI_multimedia_downloader.log` — Local schedule creation log file showing the created schedule
- `MPI_multimedia_downloader-X.log` — Main downloader log files (where X is the worker number) containing all downloader
  logs, including errors and warnings

### Images Folder

Downloaded data is stored in the configured `images_folder`, partitioned by `server_name` and `partition_id`. Each
partition contains two Parquet files:

#### successes.parquet

- `uuid`: string — Internal unique identifier for the downloaded dataset
- `source_id`: string — ID provided by the source (e.g., `gbifID`)
- `identifier`: string — Source URL of the image
- `is_license_full`: boolean — True when `license`, `source`, and `title` all contain non-null values
- `license`: string — License information
- `source`: string — Source information
- `title`: string — Title information
- `hashsum_original`: string — MD5 hash of the original image data
- `hashsum_resized`: string — MD5 hash of the resized image data
- `original_size`: [height, width] — Dimensions of the original image
- `resized_size`: [height, width] — Dimensions after resizing
- `image`: bytes — Binary image data

#### errors.parquet

- `uuid`: string — Internal unique identifier for the downloaded dataset
- `identifier`: string — URL of the image
- `retry_count`: integer — Number of download attempts
- `error_code`: integer — HTTP or other error code
- `error_msg`: string — Detailed error message

### Schedules Folder

Contains scheduling information for the downloader, organized similarly to the logs folder with `current` and numbered
folders (`0001`, `0002`, etc.).

**Within each schedule folder:**

- Numbered folders (`0000` and forward) corresponding to server batches

**Within each server batch folder:**

- `0000.csv` and forward — Individual schedules for each worker "layer"
- `_config.csv` — Configuration table for each server in the schedule with columns:
    - `index` — Pandas index
    - `server_name` — Name of the server (extracted from URL)
    - `total_batches` — Total number of batches remaining for this server
    - `process_per_node` — Number of processes per node for this server
    - `nodes` — Number of nodes assigned to this server

> [!NOTE]
> The dual parameters `process_per_node` and `nodes` were designed for scenarios with long write times compared to
> download times. `nodes` represents separate worker groups operating independently, while `process_per_node` indicates
> downloaders within a node that process one download at a time. However, since download time typically exceeds write
> time
> by 10x, `process_per_node` is usually set to 1.

- `_jobs_ids.csv` — Job IDs of workers launched for this batch:
    - `job_id` — Worker job ID
    - `is_verification` — Boolean indicating verification job (True) or downloader job (False)

- `_verification.csv` — Identifiers of completed batches (server_name + partition_id):
    - `server_name` — Name of the server (extracted from URL)
    - `partition_id` — Partition ID of the downloaded batch
    - `status` — Batch status ("Completed" or "Failed", typically "Completed")

### Profiles Table

Contains download speed information for each server (images per second). Currently uses uniform speed settings across
all servers rather than actual profiling.

**Schema:**

- `server_name` — Server name (extracted from URL)
- `total_batches` — Total number of batches for this server
- `success_count` — Number of successful downloads (currently 0, reserved for future use)
- `error_count` — Number of errors (currently 0, reserved for future use)
- `rate_limit` — Download rate limit in images per second

### Ignored Table

CSV file listing servers that should be ignored by the downloader. Not created by default but can be manually created.

**Required columns:**

- `server_name` — Name of the server to ignore (extracted from URL)

> [!NOTE]
> Changes to the ignored table require restarting the download batch for the "big" scheduler to recognize them.

### Inner Checkpoint File

Stores the current state of the downloader, tracking pipeline stage progress. While typically not requiring manual
intervention, it can be consulted to verify download completion status.

### Tools Folder

Contains output files from pipeline tools. Each tool creates its own subfolder matching the tool name.
