# documint2md - Convert PDF, DOCX and CSV to Markdown
 
documint2md is a small Python CLI and library (package `doc2md`) that turns PDF, DOCX, CSV, and image files into consistent, deterministic Markdown. It is built for documentation flows where the same source should always produce the same Markdown output, even when run on different machines or in CI.

## Highlights

- Text-first conversions for PDF (`pdfminer.six`), DOCX (Mammoth → BeautifulSoup → `markdownify`), and CSV (Pandas + Markdown table) controls the format you care about.
- OCR support for images and scanned PDFs (opt-in for PDFs).
- Small CLI plus a library API that can drop right into scripts, CI, or exploratory sessions.
- Deterministic normalization (newline, whitespace, blank lines) and CLI contracts that keep automation predictable.
- Interactive terminal UI with a short `/` command list plus `/more` for advanced tools and OCR/session controls.

## Quick start

1. Create a virtualenv, install reproducible dependencies, and activate it (Python 3.11+):
   ```powershell
   Set-Location 'C:\path\to\documint2md'
   py -m venv .venv
   & .\.venv\Scripts\Activate.ps1
   python -m pip install --upgrade pip
   python -m pip install --require-hashes -r requirements.txt
   ```
2. Convert a few sample files so “it works”:
   ```powershell
   doc2md .\tests\fixtures\in\sample.docx
   python -m doc2md.cli .\tests\fixtures\in\sample.pdf
   python -m doc2md.cli .\tests\fixtures\in\sample.csv
   python -m doc2md.cli .\tests\fixtures\in\sample.png
   ```
3. Drop into interactive mode (no inputs) to explore `/files`, `/format`, and `/output`.

### Reproducible installs (Windows)

- Core runtime:
  ```powershell
  python -m pip install --require-hashes -r requirements.txt
  ```
- Full feature set (PDF engines + OCR):
  ```powershell
  python -m pip install --require-hashes -r requirements-all.txt
  ```
- Dev/test dependencies:
  ```powershell
  python -m pip install --require-hashes -r requirements-dev.txt
  ```
- Regenerate lock files when dependencies change:
  ```powershell
  .\scripts\lock_requirements.ps1
  ```

## Installation

### From TestPyPI (for testing)

```powershell
py -m pip install --upgrade pip
py -m pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ --pre documint2md
doc2md --help
```

### From PyPI (production)

```powershell
py -m pip install --upgrade pip
py -m pip install documint2md
doc2md --help
```

Optional extras (PDF engines + OCR) when installing from PyPI:

```powershell
py -m pip install "documint2md[all]"
```

## CLI usage

Run `doc2md <file>` (or `python -m doc2md.cli <file>`) to convert a single input. By default the Markdown lands in `docs_out/<input filename>.md`. Use `-o <file>` to force a path and `-o -` to stream to stdout. Omit inputs to open the interactive picker, or pass `--interactive` for the picker even inside scripts.

```
python -m doc2md.cli file.docx -o file.md
python -m doc2md.cli file.pdf
python -m doc2md.cli table.csv
python -m doc2md.cli scan.png
doc2md  # interactive mode
```

### CLI contract

- Default output is `docs_out/<input filename>.md`; `-o <file>` overrides the destination, `-o -` writes to stdout.
- Interactive mode (no input) opens a curses-like UI tied to `docs_in`; `/files` loads the list and `/more` exposes advanced commands (history, profiles, UI, session toggles).
- Errors and diagnostics stream to stderr.
- Exit codes: `2` usage/argument error, `3` unsupported format, `4` conversion failure, `5` output write failure.

### CLI options

- `--format pdf|docx|csv|image` forces the parser instead of inferring from the extension.
- `--engine pdfminer|pdftext|marker` selects the PDF engine (default `pdfminer`; `marker` stays text-only unless assets are enabled explicitly).
- `--ocr` or `--ocr-mode auto` enables OCR fallback for PDFs when text extraction is empty.
- `--ocr-mode never|auto|always` controls OCR behavior for PDFs (default `never`).
- `--ocr-lang es` sets OCR language (default `es`).
- `--ocr-device cpu|gpu:0` overrides OCR device selection.
- `--ocr-render-scale 2.0` controls PDF render scale for OCR.
- `--ocr-min-score 0.5` filters low-confidence OCR text.
- `--csv-na ""` controls how empty values render.
- `--csv-float-format "%.6g"` stabilizes floating-point output when needed.
- `--profile <name>` loads defaults from `doc2md.toml`
- `--stats`, `--profile-report`, `--quiet`, `--debug`, `--version`, `--theme`, `--interactive`, `--no-input` toggle output, logging, and interactivity.

### OCR setup (optional)

Recommended (CPU + GPU side-by-side):

```powershell
.\scripts\setup_ocr_envs.ps1
```

See `docs/OCR Dual Environment Setup.md` for GPU verification, fallback index, and usage.

Quick run (GPU):

```powershell
.\scripts\doc2md-gpu.cmd docs_in\ocr_samples\sample_text.png --ocr-lang en --ocr-device gpu:0 --yes -o docs_out\sample_text.gpu.md
```

Quick run (CPU):

```powershell
.\scripts\doc2md-cpu.cmd docs_in\ocr_samples\sample_text.png --ocr-lang en --ocr-device cpu --yes -o docs_out\sample_text.cpu.md
```

CPU:
```powershell
python -m pip install paddlepaddle==3.2.0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/
python -m pip install paddleocr==3.4.0
```

GPU (Windows; choose one CUDA index):
```powershell
python -m pip install paddlepaddle-gpu==3.2.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
python -m pip install paddleocr==3.4.0
```

If model download issues:
```powershell
$env:PADDLE_PDX_MODEL_SOURCE = "BOS"
$env:PADDLE_PDX_DISABLE_MODEL_SOURCE_CHECK = "True"
```

Performance tips:
- Batch multiple files in one command to reuse OCR initialization.
- For scanned PDFs, use `--ocr-render-scale 1.0` to trade accuracy for speed.
- Prefer `--ocr-mode auto` for PDFs so OCR runs only on textless pages.
- First OCR run is slow due to model downloads; subsequent runs are faster.

## Interactive mode

When you run `doc2md` without inputs, the CLI opens a full-screen picker. Interact with `/files` (space to select, enter to convert), type `/` to see the short command list, and use `/more` for advanced tools (history, profiles, UI theme, session toggles). OCR is configured via `/ocr` subcommands (e.g. `/ocr mode auto`, `/ocr lang es`). The footer keeps the current format/engine/output in view while the header shows version + cwd. Use `Ctrl+P/Ctrl+N` for command history.

## Library API

- `doc2md.pdf_to_markdown(path)` – extracts text-only Markdown from PDFs (OCR optional via `ocr_mode`).
- `doc2md.docx_to_markdown(path)` – converts DOCX → Mammoth HTML → Markdown via `markdownify` with deterministic heading/list settings.
- `doc2md.csv_to_markdown(path)` – parses CSV files with `pandas` and emits clean Markdown tables.
- `doc2md.image_to_markdown(path)` – runs OCR on image files and returns Markdown text.
- Input types: `str | PathLike`; return type: `str`.
- Exceptions: `ConversionError` for failures, `UnsupportedFormatError` for unsupported formats/engines.

## Normalization rules

- Normalize newlines to `\n`.
- Strip trailing whitespace per line.
- Cap consecutive blank lines at two.
- Remove trailing blank lines and end every non-empty output with a single newline.

## Testing & fixtures

```powershell
python -m pip install --require-hashes -r requirements-dev.txt
python -m pytest
python -m compileall .
python -m doc2md.cli .\tests\fixtures\in\sample.docx -o .\docs_out\sample.docx.md
python -m doc2md.cli .\tests\fixtures\in\sample.pdf -o .\docs_out\sample.pdf.md
python -m doc2md.cli .\tests\fixtures\in\sample.csv -o .\docs_out\sample.csv.md
```

Edge-case fixtures live in `tests/fixtures/in` with golden Markdown in `tests/fixtures/out`. Use `docs_in` as your local drop zone.

## Publishing

Releases are tag-driven via GitHub Actions + Trusted Publishing.

- TestPyPI: push a tag like `v1.0.1rc1` to trigger `release-testpypi.yml`.
- PyPI: push a tag like `v1.0.1` to trigger `release-pypi.yml`.

## Release checklist

- Update `pyproject.toml` version.
- Regenerate `requirements.txt`, `requirements-all.txt`, and `requirements-dev.txt`.
- Run tests and CLI smoke conversions.
- Build and check distributions before upload.

## Contributing

- Work on `dev`, open a PR to `main`, and keep `main` release-ready. Tags on `main` trigger publishing workflows.
- Drop samples into `docs_in` and run the CLI to confirm conversions. Read `.github/copilot-instructions.md` for repo-specific guidance, keep diffs small, and explain fixture changes when extraction output shifts.

## Notes

- The interactive UI pauses ~2 seconds after success so the confirmation stays on screen unless you pass `--quiet`.
- History helpers: `doc2md history`, `search`, `rerun`, `jump`, `recent`, `explain`, and `ui`.
- The CLI exposes both quick (`/files`, `/format`, `/output`) and advanced (`/more`) helpers to explore settings without re-running the command.
