# Passiflora Leaf Datasets Design

## Status

Draft

## Authors

- Koji Noshita

## Last Updated

2026-02-19

## Overview

### Problem Statement

The ktch project requires a biologically rich, real-world dataset to support tutorials on outline-based morphometrics (EFA/PCA) and landmark-based morphometrics (GPA/PCA). The Passiflora leaf data from Chitwood & Otoni (2016) is an ideal candidate: it covers 40 species with dramatically different leaf shapes, provides both outline and landmark data, and is publicly available.

A subset of this data is already distributed as `image_passiflora_leaves` (since v0.7.0). However, the current version contains only 7 images from 2 species, which is insufficient for full morphometric workflows. The coordinate datasets (`outline_passiflora_leaves` and `landmark_passiflora_leaves`) do not yet exist in ktch.

The data preparation work happens in a separate project. This document serves as the authoritative specification for that work, defining the content, format, and integration requirements for all three Passiflora datasets.

### Goals

- Define the content and format for `image_passiflora_leaves` (updated version with 8-10 species)
- Specify the planned `outline_passiflora_leaves` dataset
- Specify the planned `landmark_passiflora_leaves` dataset
- Define how the three datasets relate to each other and how they can be joined
- Provide enough detail for data preparation to proceed without referring back to the ktch codebase

### Non-Goals

- Implementation of the loader functions (already partially implemented in `ktch/datasets/_base.py`)
- Data extraction algorithms (outline tracing, landmark digitization)
- The tutorial notebooks themselves

## Background

Source data: GigaDB repository DOI:10.5524/100251 (Chitwood & Otoni, 2016). The repository contains 555 high-resolution TIFF scan images covering more than 3,300 leaves from 40 Passiflora species. Each scan contains multiple leaves photographed adaxial face down on an A4 sheet. Leaves are numbered sequentially from shoot tip (1) to base.

The repository also includes `Leaf_outlines.zip`, which contains 3,294 pre-extracted outline mask images (one per leaf). A separate GitHub repository (https://github.com/DanChitwood/PassifloraLeaves) provides landmark data for 3,319 leaves. The difference of 25 leaves likely reflects cases where automated outline extraction was not successful.

The data accompanies the paper "Morphometric analysis of Passiflora leaves: the relationship between landmarks of the vasculature and elliptical Fourier descriptors of the blade." Both outline (EFA) and landmark (vasculature, sinuses, lobes) analyses are presented.

All other ktch coordinate datasets (trilobite cephala, mosquito wings, bottles, leaf bending) include all specimens and tutorials use complete datasets for full workflows. The Passiflora datasets should follow this convention for the coordinate data.

Image data is a special case: even after resizing, 555 images at ~2 MB each would produce a ~1 GB archive, which is impractical to serve as a dataset download. A representative subset is selected instead.

## Proposed Solution: 3-Dataset Architecture

Three separate datasets, each derived from the same GigaDB source:

```
image_passiflora_leaves     -- Representative scan images for tutorials
outline_passiflora_leaves   -- Full outline coordinates (all specimens)
landmark_passiflora_leaves  -- Full landmark coordinates (all specimens)
```

The separation is motivated by:

- Image data and coordinate data have fundamentally different storage and delivery requirements.
- Coordinate data is lightweight (hundreds of KB) and can be bundled inside the package, matching the pattern of all existing coordinate datasets.
- Image data is large (tens of MB) and must be fetched on demand via pooch from Cloudflare R2.
- Tutorials can chain naturally: "extract outlines from images" leads to "analyze the full outline dataset with EFA/PCA."

## Dataset Specifications

### image_passiflora_leaves

#### Purpose

Provides scan images for tutorials demonstrating outline extraction and landmark digitization. Not intended to cover all specimens—representative coverage across morphological diversity is the goal.

#### Content

- Species: 8-10 (covering the range from simple to deeply lobed leaf shapes)
- Individuals: 1 plant per species
- Images: approximately 24-40 PNG files (3-4 scan images per individual)
- Image resolution: 1748 x 1268 pixels (resized from original TIFF scans)
- Format: PNG (lossless; preserves edge quality needed for outline extraction)
- Estimated zip size: 50-85 MB
- Delivery: remote via pooch from Cloudflare R2

#### Zip Structure

```
image_passiflora_leaves/
├── metadata.csv
└── images/
    ├── Pact1_1_17.png
    ├── Pact1_18_20.png
    ├── Pedu1_1_15.png
    └── ...
```

#### metadata.csv

`image_id` is the filename stem without extension. The actual image file is `images/{image_id}.png`.

Index column and column definitions:

| Column | Description | Example |
|--------|-------------|---------|
| `image_id` (index) | Filename stem (no extension) | `Pact1_1_17` |
| `abbreviation` | Species abbreviation from original dataset | `Pact` |
| `genus` | Genus name | `Passiflora` |
| `species` | Species epithet | `actinia` |
| `individual_id` | Plant identifier | `Pact1` |
| `leaf_start` | First leaf number in the scan (tip side) | `1` |
| `leaf_end` | Last leaf number in the scan (base side) | `17` |

Example rows:

```csv
image_id,abbreviation,genus,species,individual_id,leaf_start,leaf_end
Pact1_1_17,Pact,Passiflora,actinia,Pact1,1,17
Pact1_18_20,Pact,Passiflora,actinia,Pact1,18,20
Pedu1_1_15,Pedu,Passiflora,edulis,Pedu1,1,15
Pedu1_16_22,Pedu,Passiflora,edulis,Pedu1,16,22
```

#### Source File Naming Convention

Original TIFF files follow the pattern: `{species_abbrev}{plant_number}_{leaf_start}_{leaf_end}.tif`

Example: `Pact1_1_17.tif` = P. actinia, plant 1, leaves 1-17 (leaf 1 is closest to shoot tip).

PNG output files preserve the same base name, changing only the extension.

#### Species Selection Criteria

Select 8-10 species satisfying all of the following:

- Cover the range of leaf shape complexity (simple symmetric shapes through deeply lobed or asymmetric forms)
- Include variation in image quality and background complexity (to make tutorials realistic)
- Include the 2 species already present in the current v0.7.0 dataset
- Prioritize individuals where vascular structure is clearly visible (useful for landmark digitization demonstrations)
- 8 species is acceptable if morphological diversity is sufficiently covered; up to 10 if needed for better coverage

#### Image Processing Pipeline

1. Download TIFF scans for selected individuals from GigaDB (DOI:10.5524/100251)
2. Resize to 1748 x 1268 pixels
3. Convert to PNG format
4. Preserve original filename base (change `.tif` to `.png` only)
5. Package as `image_passiflora_leaves.zip` with the directory structure above

#### Size Budget

- Target: 50-100 MB for the zip archive
- Baseline measurement: 7 images = approximately 15 MB (~2.1 MB per image average)
- Projection: 24-40 images x ~2.1 MB per image = 50-84 MB
- If the archive exceeds 100 MB: increase PNG compression level (lossless, no quality impact) or crop excess white background

#### Versioning

- v0.7.0: 7 images, 2 species (current)
- Next version: 8-10 species, approximately 24-40 images

The versioning strategy (whether to replace v0.7.0 or register a new version entry in `_registry.py`) is to be decided when the new data is ready.

### outline_passiflora_leaves

Status: planned.

#### Content

- 3,294 leaf specimens with pre-extracted outlines available
- 2D outline coordinates (closed contours)
- Lightweight CSV; expected size: hundreds of KB to low single-digit MB
- Delivery: bundled in the ktch package (same as `outline_mosquito_wings` and `outline_leaf_bending`)

#### Sample Selection

The final specimen set should be validated against the original paper (Chitwood & Otoni, 2017, GigaScience 6(1)). `Leaf_outlines.zip` contains 3,294 outline masks, while the GitHub repository provides landmark data for 3,319 leaves. The 25-leaf difference suggests some leaves could not be outline-extracted. Cross-reference with the paper's analysis to confirm inclusion criteria.

#### Files

Following the existing convention under `ktch/datasets/data/`:

```
outline_passiflora_leaves/
├── data_outline_passiflora_leaves.csv
└── meta_outline_passiflora_leaves.csv
```

`data_outline_passiflora_leaves.csv`: MultiIndex (leaf_id, coord_id), columns (x, y).

`meta_outline_passiflora_leaves.csv`: One row per leaf. Columns defined in the Inter-Dataset Relationships section below.

#### Loader

```python
load_outline_passiflora_leaves(*, as_frame=False) -> Bunch
```

Returns: `coords`, `meta`, `DESCR`, `filename` — matching the pattern of `load_outline_mosquito_wings`.

### landmark_passiflora_leaves

Status: planned.

#### Content

- 3,319 leaf specimens with landmark data (from GitHub repository)
- 15 homologous landmarks per specimen (vasculature, sinuses, lobes), 2D
- Raw (pre-Procrustes) landmark coordinates preferred; GPA should be performed on the ktch side
- Lightweight CSV
- Delivery: bundled in the ktch package (same as `landmark_mosquito_wings` and `landmark_trilobite_cephala`)

#### Sample Selection

The specimen set should be validated against the original paper's landmark analysis. The landmark analysis may use a different subset than the outline analysis (e.g., different filtering criteria or a different set of species). Cross-reference with the paper's supplementary data to confirm which specimens have landmark annotations and which were included in the published GPA analysis.

#### Files

```
landmark_passiflora_leaves/
├── data_landmark_passiflora_leaves.csv
└── meta_landmark_passiflora_leaves.csv
```

`data_landmark_passiflora_leaves.csv`: MultiIndex (leaf_id, landmark_id), columns (x, y).

`meta_landmark_passiflora_leaves.csv`: One row per leaf. Columns defined in the Inter-Dataset Relationships section below.

#### Loader

```python
load_landmark_passiflora_leaves(*, as_frame=False) -> Bunch
```

Returns: `coords`, `meta`, `DESCR`, `filename` — matching the pattern of `load_landmark_mosquito_wings`.

## Inter-Dataset Relationships

### Row Granularity

| Dataset | Row unit | Example |
|---------|----------|---------|
| `image_passiflora_leaves` | Scan image (contains multiple leaves) | `Pact1_1_17` |
| `outline_passiflora_leaves` | Individual leaf | Pact1, leaf 5 |
| `landmark_passiflora_leaves` | Individual leaf | Pact1, leaf 5 |

### Join Keys

`individual_id` is the shared key connecting all three datasets at the individual (plant) level.

To determine which scan image contains a specific leaf, join on `individual_id` and filter on the leaf number range:

```python
merged = outline_meta.merge(image_meta, on="individual_id")
mask = (
    (merged["leaf_number"] >= merged["leaf_start"])
    & (merged["leaf_number"] <= merged["leaf_end"])
)
containing_images = merged[mask]
```

Note: the image dataset covers only a subset of individuals (8-10 species, 1 plant each). Not every leaf in the outline/landmark datasets will have a corresponding image.

### Metadata Columns for outline and landmark Datasets

At minimum, the metadata CSV for both coordinate datasets must include:

| Column | Description |
|--------|-------------|
| `leaf_id` (index) | Unique leaf identifier |
| `individual_id` | Plant identifier; join key to image dataset |
| `leaf_number` | Position on the vine (1 = shoot tip) |
| `abbreviation` | Species abbreviation |
| `genus` | Genus name (`Passiflora`) |
| `species` | Species epithet |

## Species Abbreviation Reference

From the original GigaDB readme (40 species, all genus Passiflora):

| Abbreviation | Species epithet |
|--------------|-----------------|
| Pact | actinia |
| Pala | alata |
| Pame | amethystina |
| Pbif | biflora |
| Pcae | caerulea |
| Pcap | capsularis |
| Pcin | cincinnata |
| Pcoc | coccinea |
| Pcor | coriacea |
| Pcri | cristalina |
| Pedm | edmundoi |
| Pedu | edulis |
| Pfoe | foetida (first abbreviation) |
| Pfii | foetida (second abbreviation) |
| Pgal | galbana |
| Pgib | gibertii |
| Pgra | gracilis |
| Phat | hatschbachii |
| Pker | kermesina |
| Plig | ligularis |
| Pmaf | maliformis |
| Pmal | malacophylla |
| Pmic | micropetala |
| Pmie | miersii |
| Pmin | miniata |
| Pmis | misera |
| Pmol | mollissima |
| Pmor | morifolia |
| Pmuc | mucronata |
| Pnit | nitida |
| Porg | organensis |
| Ppoh | pohlii |
| Prac | racemosa |
| Prub | rubra |
| Pset | setacea |
| Psid | sidifolia |
| Psub | suberosa |
| Pten | tenuifila |
| Ptlo | triloba |
| Ptri | tricuspis |
| Pvil | villosa |

Note: P. foetida appears under two abbreviations (Pfoe and Pfii) in the original dataset. Treat them as the same species when computing species-level statistics.

## ktch Integration Notes

### Existing Loader (image_passiflora_leaves)

The loader is implemented in `ktch/datasets/_base.py`. The current function signature:

```python
load_image_passiflora_leaves(
    *, return_paths=False, as_frame=False, version=None
) -> Bunch
```

Return value attributes:

- `images`: list of ndarray (H, W, C) when `return_paths=False`; list of str paths when `return_paths=True`
- `meta`: dict (default) or DataFrame (`as_frame=True`) with columns matching metadata.csv
- `DESCR`: dataset description string
- `data_dir`: str path to the extracted directory
- `version`: the resolved dataset version string

The existing metadata columns in the current v0.7.0 data include `genus` and `species` but not `abbreviation`, `individual_id`, `leaf_start`, or `leaf_end`. The new version must add these columns. The loader code reads metadata with `index_col=0` and the image paths are resolved as `data_dir / "images" / f"{image_id}.png"`, so the directory structure must match the specification above.

### Remote Delivery Infrastructure

- Storage: Cloudflare R2, base URL `https://pub-c1d6dba6c94843f88f0fd096d19c0831.r2.dev`
- Path pattern: `releases/v{version}/{filename}`
- Registry: `ktch/datasets/_registry.py`, `versioned_registry` dict maps version -> filename -> SHA256 hash
- Registry update: `scripts/update_registry.py` is used after uploading new data

### Pillow Dependency

`load_image_passiflora_leaves` uses `PIL.Image` when `return_paths=False`. Pillow is not currently declared in the package extras. It should be added to the `[data]` extras group alongside pooch.

### Coordinate Dataset Loaders (outline and landmark)

Follow the existing patterns in `_base.py`:

- Use `_resolve_data_path(dataset_name, filename)` for bundled CSV files
- Return a `Bunch` with `coords`, `meta`, `DESCR`, `filename`
- Support `as_frame` parameter with the same numpy/dict vs DataFrame/dict behavior as existing loaders

## Open Questions

- [ ] Which 8-10 species to select for `image_passiflora_leaves`? Confirm against diversity and image quality criteria.
- [ ] Should the new image dataset replace v0.7.0 or be registered as a new version entry?
- [ ] Sample selection: the available leaf counts differ across sources — 3,319 (landmark data, GitHub repo), 3,294 (outline mask images, `Leaf_outlines.zip` in GigaDB), 3,217 (outline coordinate values, GitHub repo). Each step (scan → mask extraction → coordinate extraction) likely filters out some leaves. The exact discrepancies and filtering criteria should be investigated when preparing the outline and landmark datasets. Validate against the original paper at that time.
- [ ] For `outline_passiflora_leaves`: what outline point count per leaf? (Number of sampled points on the contour.)
- [ ] For `landmark_passiflora_leaves`: confirm the 15-landmark configuration against the original paper's supplementary data.
- [ ] For `landmark_passiflora_leaves`: confirm that raw (pre-Procrustes) coordinates are available. If only Procrustes-adjusted coordinates are provided in the source, document this clearly.
- [ ] Confirm that Pillow is the correct image loading dependency (vs. imageio or skimage.io).

## References

- Chitwood, D.H., Otoni, W.C., 2017. Morphometric analysis of Passiflora leaves: the relationship between landmarks of the vasculature and elliptical Fourier descriptors of the blade. GigaScience, 6(1), giw008. https://doi.org/10.1093/gigascience/giw008
- Chitwood, D.H., Otoni, W.C., 2016. Supporting data for "Morphometric analysis of Passiflora leaves: the relationship between landmarks of the vasculature and elliptical Fourier descriptors of the blade". GigaScience Database. https://doi.org/10.5524/100251
- Chitwood, D.H. PassifloraLeaves. GitHub repository. https://github.com/DanChitwood/PassifloraLeaves
- Source readme: `ref/readme.txt` in this repository
- Existing loader implementation: `ktch/datasets/_base.py`
- Dataset registry: `ktch/datasets/_registry.py`
