🌌 FlowMeta Tutorial

🧰 1. Prerequisites

1. Platform: Linux or WSL (Windows Subsystem for Linux). SSD/NVMe storage keeps Bowtie2 and Kraken2 fast.
2. Python (≥3.8): Recreate the recommended environment with Conda or mamba:

   conda env create -f environment.yml
   conda activate meta
   # or
   mamba env create -f environment.yml
   mamba activate meta

   Prefer an isolated virtual environment?

   python -m venv .venv
   source .venv/bin/activate  # Windows PowerShell: .venv\Scripts\activate
   pip install -r docs/quickstart-requirements.txt  # tailor as needed

3. External tools: fastp, bowtie2, samtools, kraken2, bracken, pigz, seqkit. Ensure each command resolves from $PATH.
4. Reference assets:
   • Bowtie2 index prefix (e.g. /mnt/db/GRCh38_noalt_as/GRCh38_noalt_as).
   • Kraken2 database directory containing hash.k2d, opts.k2d, taxo.k2d, and optional Bracken taxonomy tables.

📦 2. Installation Options

2.1 🌐 PyPI (preferred)

pip install flowmeta

Ideal once the package is published publicly; every user can install directly.

2.2 💾 Local wheel

pip install dist/flowmeta-0.1.5-py3-none-any.whl

Use when sharing a pre-built artifact inside secure networks.

2.3 🔧 Build from source

pip install build

python -m build --wheel --sdist
pip install dist/flowmeta-0.1.5-py3-none-any.whl

Confirm README.md, README.zh.md, and docs/tutorial.html stay bundled in the sdist so downstream installs ship with documentation.

🧬 3. Database Preparation

3.1 🪐 Kraken2 reference libraries

1. Visit https://benlangmead.github.io/aws-indexes/k2.
2. Download and extract the desired database (e.g. k2_standard_20240112) to /mnt/db/k2ppf.
3. Double-check the directory contains hash.k2d, opts.k2d, taxo.k2d, library_report.tsv, and related files.
4. Optional performance boost: pass --shm_path /dev/shm/k2ppf so FlowMeta caches the DB in RAM.

3.2 🧠 Bowtie2 host index (GRCh38 example)

# fetch the genome
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.28_GRCh38.p13/GCA_000001405.28_GRCh38.p13_genomic.fna.gz

# optional: remove alternative contigs
seqkit grep -rvp "alt|PATCH" GCA_000001405.28_GRCh38.p13_genomic.fna.gz > GRCh38_noalt.fna

# build Bowtie2 index
mkdir -p /mnt/db/GRCh38_noalt_as
bowtie2-build GRCh38_noalt.fna /mnt/db/GRCh38_noalt_as/GRCh38_noalt_as

# FlowMeta configuration
flowmeta_base ... --db_bowtie2 /mnt/db/GRCh38_noalt_as/GRCh38_noalt_as

🔁 Swap the download URL and filtering logic to index other host species.

🚀 4. Quick Command Overview

4.1 ⚡ Fast start

flowmeta_base \
  --input_dir /mnt/data/flowmeta/01-raw \
  --output_dir /mnt/data/flowmeta/flowmeta-out \
  --db_bowtie2 /mnt/db/GRCh38_noalt_as/GRCh38_noalt_as \
  --db_kraken /mnt/db/k2ppf \
  --threads 32 \
  --project_prefix GLOBAL-

The pipeline auto-detects paired FASTQ files, writes outputs to the specified workspace, and drops .task.complete flags for safe restarts.

4.2 🛠️ Flag cheat sheet

4.3 🎯 Common scenarios

• 🔁 Resume classification: flowmeta_base ... --step 6
• 🧹 Force rebuild from Bowtie2: flowmeta_base ... --step 3 --force
• 🎯 Single-end reads: flowmeta_base ... --se --suffix1 _R.fastq.gz
• 🚫 Skip host FASTQ export: flowmeta_base ... --skip_host_extract

📂 5. Output Layout

02-qc/       fastp reports + trimmed reads
03-hr/       Host-depleted FASTQ files
04-bam/      Bowtie2 BAM and index files
05-host/     Optional host FASTQ exports
06-ku/       Kraken2 reports (first pass)
07-bracken/  Bracken abundance tables
08-ku2/      Host-filtered rerun outputs
09-mpa/      Final merged OTU / MPA matrices

🧩 Each directory stores per-sample .task.complete files so you can resume safely after interruptions.

🧭 6. Step-by-Step Logic

ℹ️ Integrity checks in Steps 2 and 4 run only when you supply --check_result (default is off to save time) and keep them enabled. If you pass --skip_integrity_checks, all FASTQ integrity checks are skipped to maximize throughput—use this only when storage/media is trusted. At startup the CLI also prints a concise path overview for all step directories.

1. fastp quality control.
2. fastp integrity verification (requires --check_result).
3. Bowtie2 host filtering (FASTQ + BAM outputs).
4. Host-removed FASTQ validation (requires --check_result).
5. Optional host read export (samtools fastq).
6. Kraken2 database staging (shared memory).
7. Kraken2 + Bracken classification.
8. Kraken report validation.
9. Host-taxid filtering and Bracken rerun.
10. Final OTU/MPA/Bracken matrix merges.

🔄 Steps map to functions in flowmeta/steps/ for advanced customization and can be resumed with --step.

🛡️ 7. Logging & Troubleshooting

• Each stage prints a STEP n banner; unrecoverable errors raise FlowMetaError.
• Remove a stale .task.complete file to re-run a specific sample.
• Adjust --fastp_retries, --host_retries, or --kraken_retries after transient failures.
• For slow Kraken2 runs, avoid --no_shm and copy databases to SSD/RAM using --shm_path.
• Detailed stdout/stderr for external tools lives beside step outputs.

❓ 8. FAQ

1. How do I rebuild after a database update?
   Use --step 3 --force (Bowtie2) or a later step depending on the scope.
2. Can I add new samples incrementally?
   Drop FASTQ files into 01-raw; FlowMeta processes only the samples lacking .task.complete markers.
3. Where are the fastp QC reports?
   Check 02-qc for both HTML reports and logs; ensure disk space is sufficient.
4. CLI help?
   flowmeta_base -h prints the entire argument list.

🤝 9. Support & Citation

Maintainer: Dongqiang Zeng · Southern Medical University · interlaken@smu.edu.cn

Please cite the GitHub repository if FlowMeta contributes to your research: https://github.com/SkinMicrobe/FlowMeta.

🎉 Happy sequencing!