# Changelog

All notable changes to the ZNA project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.3.0] - 2026-03-25

### Added
- **Per-sequence label columns** for storing SAM-tag metadata (alignment scores,
  edit distances, etc.) alongside compressed sequences
  - Labels defined via `--label NAME:TYPE` on the CLI (repeatable)
  - 11 numeric dtypes: `A`, `c`/`C`, `s`/`S`, `i`/`I`, `f`, `d`, `q`/`Q`
  - Per-label `missing` value for reads where a tag is absent
  - Columnar storage: each label column packed contiguously per block
- **Decoupled label name and tag** — label `name` (stored in ZNA, up to 16
  chars) can differ from the SAM `tag` parsed at encode time
  - CLI 3-part format: `--label edit_dist:C:NM` (name:type:tag)
  - YAML: optional `tag` field per label definition
  - Tag is encode-time only; not stored in the ZNA file
  - If only name or tag is given, the other defaults to the same value
- **YAML label specification files** (`--label-defs labels.yaml`)
  - Define labels, descriptions, dtypes, and missing values in a single file
  - CLI `--label` / `--label-desc` flags override YAML definitions
  - Sample template: `examples/labels.yaml`
- **C++ accelerated label encode/decode** — labeled files now use the C++
  backend for both encoding and decoding, eliminating the Python fallback
  - `encode_block_labeled`: encodes sequences + pre-packed label columns in C++
  - `decode_block_labeled`: decodes sequences and unpacks label columns by dtype
  - `extract_labels_fast`: zero-allocation C++ SAM-tag extraction from headers
- **Optimized Python header parsing** (fallback path)
  - Eliminated `bytes.decode()` calls — `int()`/`float()` accept bytes directly
  - Replaced lambda dispatch with integer conv-codes
  - Early-exit when all labels found

### Performance
- **Labeled encode**: 336K rec/s (was 80K rec/s) — **4.2× faster**
- **Labeled decode**: 2.23M rec/s (was 610K rec/s) — **3.7× faster**
- Label overhead reduced from ~39% to within typical I/O variance

### Changed
- Binary format: label definition on-disk size is now 89 bytes
  (16 name + 64 desc + 1 dtype_code + 8 missing)
- `LabelDef.param` replaced by `LabelDef.missing` (explicit missing sentinel)
- Header field splitting uses any whitespace (tabs and spaces) so fastp
  `merged_XX_YY` suffixes no longer corrupt trailing tag values
- Label tag parsing now supports variable-length keys in `KEY:TYPE:VALUE`
  header fields (not limited to 2-character SAM tags)
  - Python fallback parser and C++ `extract_labels_fast` use dynamic key parsing
  - Supports mixed key lengths in the same header (e.g. `NM` + `edit_distance`)
- Dropped fixstr (`Z`) label type — all labels are numeric

### Documentation
- Expanded README strand guidance to explicitly document
  `--strand-normalize` behavior and its interaction with `--strand-specific`
- Updated label docs and `examples/labels.yaml` to show `name` + `tag`
  decoupling and custom non-SAM, variable-length header keys

## [0.2.0] - 2026-03-24

### Added
- **Strand normalization** (`--strand-normalize`) for consistent strand orientation in ZNA files
  - Unstranded libraries: randomly reverse-complements one read per pair (or SE reads) with per-record `IS_RC` flag
  - Strand-specific libraries: deterministically RCs antisense reads to sense orientation
  - Orthogonal to library metadata (`--strand-specific`, `--read1-antisense`, etc.)
  - `--restore-strand` on decode reverses RC operations using per-record `IS_RC` flags
- New `IS_RC` record flag (bit 3) tracks which records were reverse-complemented
- New `STRAND_NORMALIZED` header flag (bit 3) indicates strand normalization was applied

### Fixed
- C++ accelerator bug: paired R2 reads could receive an unintended second random RC during unstranded normalization

### Changed
- Bioconda recipe: added `osx-arm64` and `linux-aarch64` to `additional-platforms`

## [0.1.8] - 2026-02-11

### Fixed
- macOS conda build: removed `llvm-tools` from host dependencies

## [0.1.7] - 2026-02-06

- **Documentation changes**

## [0.1.6] - 2026-02-06

### Added
- **Built-in shuffle command** (`zna shuffle`) for random shuffling of ZNA files with bounded memory
  - Bucket-shuffle algorithm with configurable memory budget (default 1GB)
  - Preserves paired-end read associations (R1+R2 stay together)
  - Deterministic shuffling with `--seed` option for reproducibility
  - Suitable for ML training data preparation
- **Encode with shuffle** (`zna encode --shuffle`) to shuffle during encoding
  - Eliminates need for separate shuffle step
  - Uses same memory-bounded algorithm as standalone shuffle command
- New `_shuffle.py` module with clean `shuffle_zna()` API for programmatic use

### Changed
- Refactored CLI: shuffle logic extracted from `cli.py` to `_shuffle.py` for maintainability
- Consolidated `--block-size` argument (now shared across encode and shuffle commands)

### Performance
- **Decode optimization**: Added `yield from` fast path when `restore_strand=False` (10% faster)
- **Encode optimization**: New `write_records()` batch method caches attribute lookups (20% faster)
- Overall throughput remains: 165 MB/s encode, 241 MB/s decode with 9.10x compression

### Documentation
- Updated README.md with shuffle command documentation and examples
- Updated PERFORMANCE.md with recent optimization notes
- All 86 tests passing

## [0.1.5] - 2026-02-04

### Changed
- Columnar block storage format for 2.5x better compression (9.10x vs 3.68x)
- Optimized flag operations (removed enum overhead for 60-135% speedup)
- Default block size: 128 KB (512 KB recommended for archival)

### Performance
- Compression: 9.10x (9.44x with 512 KB blocks)
- Speed: 165 MB/s encode, 241 MB/s decode (pure Python)
- File size competitive with BAM format

## [0.2.0] - 2026-02-03

### Added
- C++ acceleration with nanobind (9.5x speedup over pure Python)
- Block-based architecture for streaming I/O
- Strand-specific library support (dUTP, TruSeq protocols)
- N-policy handling (drop, random, replace)

### Performance
- Short reads: 189.5 MB/s encode, 668.8 MB/s decode
- Long reads: 2,824 MB/s encode, 3,393 MB/s decode

## [0.1.0] - 2026-01-15

### Added
- Initial ZNA format implementation
- 2-bit nucleotide encoding
- Zstd compression support
- CLI tools: encode, decode, inspect
- Basic FASTQ/FASTA support
- Paired-end and interleaved read handling

[0.1.6]: https://github.com/mkiyer/zna/compare/v0.1.5...v0.1.6
[0.1.5]: https://github.com/mkiyer/zna/releases/tag/v0.1.5
