<h1 align="center">
  <img src="https://raw.githubusercontent.com/kantord/headson/main/docs/assets/logo.svg" alt="headson" width="221" />
</h1>
<p align="center">
  <img src="https://raw.githubusercontent.com/kantord/headson/main/docs/assets/tapes/demo.gif" alt="Terminal demo" width="1560" height="900" />
  <br/>
</p>

`head`/`tail` for JSON, YAML — but structure‑aware. Get a compact preview that shows both the shape and representative values of your data, all within a strict byte budget. (Just like `head`/`tail`, `hson` can also work with unstructured text files.)

Available as:
- CLI (see [Usage](#usage))
- Python library (see [Python Bindings](#python-bindings))

![Codecov](https://img.shields.io/codecov/c/github/kantord/headson?style=flat-square) ![Crates.io Version](https://img.shields.io/crates/v/headson?style=flat-square) ![PyPI - Version](https://img.shields.io/pypi/v/headson?style=flat-square)


## Install


Using Cargo:

    cargo install headson


> Note: the CLI installs as `hson`. All examples below use `hson ...`.


From source:

    cargo build --release
    target/release/hson --help

 



## Features

- Budgeted output: specify exactly how much you want to see
- Output formats: `auto | json | yaml | text`
  - Styles: `strict | default | detailed`
    - JSON family: `strict` → strict JSON, `default` → human‑friendly Pseudo, `detailed` → JS with inline comments
    - YAML: always YAML; `strict` has no comments, `default` uses “# …”, `detailed` uses “# N more …”
    - Text: prints raw lines. In `default` style, omissions are shown as a single line `…`; in `detailed`, as `… N more lines …`. `strict` omits array‑level summaries.
- Multiple inputs: preview many files at once with a shared or per‑file budget
- Fast: processes gigabyte‑scale files in seconds (mostly disk‑bound)
- Available as a CLI app and as a Python library

## Fits into command line workflows

If you’re comfortable with tools like `head` and `tail`, use `hson` when you want a quick, structured peek into a JSON file without dumping the entire thing.

- `head`/`tail` operate on bytes/lines - their output is not optimized for tree structures
- `jq`: you need to craft filters to preview large JSON files
- `hson`: head/tail for trees—zero‑config by default; force text with `-i text` when you want raw lines

## Usage

    hson [FLAGS] [INPUT...]

- INPUT (optional, repeatable): file path(s). If omitted, reads from stdin. Multiple input files are supported.
- Prints the preview to stdout. On parse errors, exits non‑zero and prints an error to stderr.

Common flags:

- `-c, --bytes <BYTES>`: per‑file output budget (bytes). For multiple inputs, default total budget is `<BYTES> * number_of_inputs`.
- `-u, --chars <CHARS>`: per‑file output budget (Unicode code points). Behaves like `--bytes` but counts characters instead of bytes.
- `-C, --global-bytes <BYTES>`: total output budget across all inputs. With `--bytes`, the effective total is the smaller of the two.
- `-f, --format <auto|json|yaml|text>`: output format (default: `auto`).
  - Auto: stdin → JSON family; filesets → per‑file based on extension (`.json` → JSON family, `.yaml`/`.yml` → YAML, unknown → Text).
- `-t, --template <strict|default|detailed>`: output style (default: `default`).
  - JSON family: `strict` → strict JSON; `default` → Pseudo; `detailed` → JS with inline comments.
  - YAML: always YAML; style only affects comments (`strict` none, `default` “# …”, `detailed` “# N more …”).
- `-i, --input-format <json|yaml|text>`: ingestion format (default: `json`). For filesets in `auto` format, ingestion is chosen by extensions.
- `-m, --compact`: no indentation, no spaces, no newlines
- `--no-newline`: single line output
- `--no-header`: suppress fileset section headers (useful when embedding output in scripts)
- `--tree`: render filesets as a directory tree with inline previews (keeps code line numbers); uses per-file auto formatting.
- `--no-space`: no space after `:` in objects
- `--indent <STR>`: indentation unit (default: two spaces)
- `--string-cap <N>`: max graphemes to consider per string (default: 500)
- `--grep <REGEX>`: guarantee inclusion of values/keys/lines matching the regex (ripgrep‑style). Matches + ancestors are “free” against both global and per-file caps; budgets apply to everything else. If matches consume all headroom, only the must‑keep path is shown. Colors follow the normal on/auto/off rules; when grep is active, syntax colors are suppressed and only the match highlights are colored. JSON/YAML structural punctuation is not highlighted—only the matching key/value text.
- `--head`: prefer the beginning of arrays when truncating (keep first N). Strings are unaffected. Display styles place omission markers accordingly; strict JSON remains unannotated. Mutually exclusive with `--tail`.
- `--tail`: prefer the end of arrays when truncating (keep last N). Strings are unaffected. Display styles place omission markers accordingly; strict JSON remains unannotated. Mutually exclusive with `--head`.

Notes:

- Multiple inputs:
  - With newlines enabled, file sections are rendered with human‑readable headers (pass `--no-header` to suppress them). In compact/single‑line modes, headers are omitted.
  - Order: inputs are sorted by git frecency (via frecenfile) when available, then by mtime; pass `--no-sort` to keep the original input order without repo scanning.
  - Fairness: fileset nodes are interleaved round‑robin during selection so tight budgets don’t starve later files.
- In `--format auto`, each file uses its own best format: JSON family for `.json`, YAML for `.yaml`/`.yml`.
  - Unknown extensions are treated as Text (raw lines) — safe for logs and `.txt` files.
  - `--global-bytes` may truncate or omit entire files to respect the total budget.
  - The tool finds the largest preview that fits the budget; even if extremely tight, you still get a minimal, valid preview.
  - Directories and binary files are ignored; a notice is printed to stderr for each. Stdin reads the stream as‑is.
  - Head vs Tail sampling: these options bias which part of arrays are kept before rendering. Display styles may still insert internal gap markers to honor very small budgets; strict JSON stays unannotated.

### Working with multiple files

- Budgets: per-file caps (`--bytes`/`--chars`/`--lines`) apply to each input; global caps (`--global-*`) constrain the combined output when set. Default byte/char budgets scale by input count when no globals are set; line caps stay per-file unless you pass `--global-lines`.
- One metric per level: pick at most one per-file budget flag (`--bytes` | `--chars` | `--lines`) and at most one global flag (`--global-bytes` | `--global-lines`). Mixing per-file and global kinds is allowed (e.g., per-file lines + global bytes); conflicting flags error.
- Sorting: inputs are pre-sorted by git frecency (frecenfile) with last-modified-time fallback so recently touched files appear first. Pass `--no-sort` to preserve the order you provided and skip repo scanning.
- Headers: fileset sections get `==>` headers when newlines are enabled; hide them with `--no-header`. Compact and single-line modes omit headers automatically.
- Formats: in `--format auto`, each file picks JSON/YAML/Text based on extension; unknowns fall back to Text so mixed filesets “just work.”
- Per-file caps: omission markers count toward per-file line budgets; a per-file line cap of zero suppresses the file entirely, even when headers are counted.

## Grep mode

Use `--grep <REGEX>` to guarantee inclusion of values/keys/lines matching the regex (ripgrep-style). Matches plus their ancestors are “free” against budgets; everything else must fit the remaining headroom.

- Matching: values/lines are checked; object keys match too, except for filenames in filesets (no filename-only hits).
- Colors: only the matching text is highlighted; syntax colors are suppressed in grep mode. Disable color entirely with `--no-color`.
- Weak grep: `--weak-grep <REGEX>` biases priority toward matches but does not guarantee inclusion, expand budgets, or filter filesets. Budgets stay exact and matches can still be pruned if they do not fit.
- Filesets (strong `--grep` only):
  - Default (`--grep-show=matching`): files without matches are dropped from the render and summary. If no files match at all, the output is empty and the CLI prints a notice to stderr.
  - `--grep-show=all`: keep non-matching files in the render; only matching files are highlighted.
  - Headers respect `--no-header` as usual.
- Mutual exclusion: `--grep-show` requires `--grep` and cannot be used with `--weak-grep`; `--weak-grep` cannot be combined with `--grep`.
- Context: there are no explicit `-C/-B/-A` style flags; per-file budgets decide how much surrounding structure/lines can stay alongside the must-keep matches.
- Budgets: matches and ancestors always render; remaining budget determines what else can appear. Extremely tight budgets may show only the must-keep path.
- Text/code: works with `-i text` and code-like files; when using `--format auto`, file extensions still decide ingest/rendering.

## Tree mode

Use `--tree` to render filesets as a directory tree (like `tree`) with inline structured previews instead of per-file headers. Works with grep/weak-grep; matches are shown inside the tree.

- Layout: classic tree branches (`├─`, `│`, `└─`) with continuous guides; code gutters stay visible under the tree prefix.
- Headers: `--tree` is mutually exclusive with `--no-header`; tree mode never prints `==>` headers and relies on the tree structure instead. Files are still auto-formatted per extension (`--format` must be `auto` for filesets).
- Budgets: tree scaffolding is treated like headers (free unless you set `--count-headers`); per-file budgets always apply to file content and omission markers, and global caps apply only when provided. Tight budgets can truncate file previews within the tree, and entire files may be omitted under tiny global line budgets—omitted entries are reported as `… N more items` on the relevant folder/root. When scaffold is free, the final output can exceed the requested caps by the tree gutters/indentation; set `--count-headers` if those characters must be bounded.
- Empty sections: under very small per-file caps (or a tiny global cap, if set), files or code blocks may render only their header/tree entry with no body; omission markers appear only when at least one child fits. This is expected when nothing fits beneath the budget.
- Sorting: respects `--no-sort`; otherwise uses the usual frecency/mtime ordering before tree grouping.
- Fairness: file contents are interleaved round‑robin in the priority order so later files still surface under tight budgets.
## Budget Modes

- Bytes (`-c/--bytes`, `-C/--global-bytes`)
  - Measures UTF‑8 bytes in the output.
  - Default per‑file budget is 500 bytes when neither `--lines` nor `--chars` is provided.
  - Multiple inputs: total default budget is `<BYTES> * number_of_inputs`; `--global-bytes` caps the total.

- Characters (`-u/--chars`)
  - Measures Unicode code points (not grapheme clusters).

- Lines (`-n/--lines`, `-N/--global-lines`)
  - Caps the number of lines in the output.
  - Incompatible with `--no-newline`.
  - Multiple inputs: `<LINES>` is enforced per file; add `--global-lines` if you also need an aggregate cap.
  - Fileset headers, blank separators, and summary lines do not count toward the line cap by default; only actual content lines are considered. Pass `-H/--count-headers` to include headers/summaries in the line budget.
  - Tiny caps may yield omission markers instead of bodies (e.g., `…` for text/code, `{…}`/`[…]` for objects/arrays); a single-line file still renders when it fits.

- Interactions and precedence
  - All active budgets are enforced simultaneously. The render must satisfy all of: bytes (if set), chars (if set), and lines (if set). The strictest cap wins.
  - Outputs stay non-empty unless you explicitly set a per-file cap of zero; in that case that slot can be suppressed entirely (matching the CLI’s `-n 0` semantics). Extremely tight nonzero caps that cannot fit even an omission marker can also yield empty output; filesets/tree may show only omission counts in that scenario.
  - When only lines are specified, no implicit byte cap applies. When neither lines nor chars are specified, a 500‑byte default applies.

Quick one‑liners:

- Peek a big JSON stream (keeps structure):

      zstdcat huge.json.zst | hson -c 800 -f json -t default

- Many files with a fixed overall size:

      hson -C 1200 -f json -t strict logs/*.json

- Glance at a file, JavaScript‑style comments for omissions:

      hson -c 400 -f json -t detailed data.json

- YAML with detailed comments:

      hson -c 400 -f yaml -t detailed config.yaml

### Text mode

- Single file (auto):

      hson -c 200 notes.txt

- Force Text ingest/output (useful when mixing with other extensions, or when the extension suggests JSON/YAML):

      hson -c 200 -i text -f text notes.txt
      # Force text ingest even if the file looks like JSON
      hson -i text notes.json

- Styles on Text:
  - default: omission as a standalone `…` line.
  - detailed: omission as `… N more lines …`.
  - strict: no array‑level omission line (individual long lines may still truncate with `…`).

> **Note:** Filesets always render with per-file auto templates. When you need to preview a directory of mixed formats, skip `-f text` and let `-f auto` pick the right renderer for each entry.

Show help:

    hson --help

Note: flags align with head/tail conventions (`-c/--bytes`, `-C/--global-bytes`).

## Examples: head vs hson

Input:

```json
{"users":[{"id":1,"name":"Ana","roles":["admin","dev"]},{"id":2,"name":"Bo"}],"meta":{"count":2,"source":"db"}}
```

Naive cut (can break mid‑token):

```bash
jq -c . users.json | head -c 80
# {"users":[{"id":1,"name":"Ana","roles":["admin","dev"]},{"id":2,"name":"Bo"}],"me
```

Structured preview with hson (JSON family, default style → Pseudo):

```bash
hson -c 120 -f json -t default users.json
# {
#   users: [
#     { id: 1, name: "Ana", roles: [ "admin", … ] },
#     …
#   ]
#   meta: { count: 2, … }
# }
```

Machine‑readable preview (JSON family, strict style → strict JSON):

```bash
hson -c 120 -f json -t strict users.json
# {"users":[{"id":1,"name":"Ana","roles":["admin"]}],"meta":{"count":2}}
```

## Terminal Demos

Regenerate locally:

- Place tapes under docs/tapes (e.g., docs/tapes/demo.tape)
- Run: cargo make tapes
- Outputs are written to docs/assets/tapes


## Python Bindings

A thin Python extension module is available on PyPI as `headson`.

- Install: `pip install headson` (ABI3 wheels for Python 3.10+ on Linux/macOS/Windows).
- API:
  - `headson.summarize(text: str, *, format: str = "auto", style: str = "default", input_format: str = "json", byte_budget: int | None = None, skew: str = "balanced") -> str`
    - `format`: `"auto" | "json" | "yaml"` (auto maps to JSON family for single inputs)
    - `style`: `"strict" | "default" | "detailed"`
    - `input_format`: `"json" | "yaml"` (ingestion)
    - `byte_budget`: maximum output size in bytes (default: 500)
    - `skew`: `"balanced" | "head" | "tail"` (affects display styles; strict JSON remains unannotated)

Examples:

```python
import json
import headson

data = {"foo": [1, 2, 3], "bar": {"x": "y"}}
preview = headson.summarize(json.dumps(data), format="json", style="strict", byte_budget=200)
print(preview)

# Prefer the tail of arrays (annotations show with style="default"/"detailed")
print(
    headson.summarize(
        json.dumps(list(range(100))),
        format="json",
        style="detailed",
        byte_budget=80,
        skew="tail",
    )
)

# YAML support
doc = "root:\n  items: [1,2,3,4,5,6,7,8,9,10]\n"
print(headson.summarize(doc, format="yaml", style="default", input_format="yaml", byte_budget=60))
```

## Source Code Support

Source code support is a challenging area. While `headson`'s algorithm and code structure would allow for the use of
completely accurate parsing using language-specific parsers using `tree-sitter`, this would increase the complexity
of the application and its number of dependencies.

Instead of attempting a deep parse of source code files, we convert them into nested arrays based on a heuristic that
understands indentation patterns in the file.

When `headson` detects a code-like file, it uses a set of additional heuristics:
- **Atomic line ingest**: each line is treated as an atomic string so omission markers never split a code line.
- **Depth-aware sampling**:
  - We attempt to include more of the top level of the source code in order to give a good overview of classes, function and constants at the top level.
  - Nested blocks (function bodies, loops) prefer to omit lines in the middle to attempt to preserve natural "block" boundaries
- **Header priority**: lines that introduce a nested block (e.g., `def foo():`) get a small priority boost to ensure they survive tight budgets.

# Algorithm

![Algorithm overview](https://raw.githubusercontent.com/kantord/headson/main/docs/assets/algorithm.svg)

## Footnotes
 - <sup><b>[1]</b></sup> <b>Optimized tree representation</b>: An arena‑style tree stored in flat, contiguous buffers. Each node records its kind and value plus index ranges into shared child and key arrays. Arrays are ingested in a single pass and may be deterministically pre‑sampled: the first element is always kept; additional elements are selected via a fixed per‑index inclusion test; for kept elements, original indices are stored and full lengths are counted. This enables accurate omission info and internal gap markers later, while minimizing pointer chasing.
 - <sup><b>[2]</b></sup> <b>Priority order</b>: Nodes are scored so previews surface representative structure and values first. Arrays can favor head/mid/tail coverage (default) or strictly the head; tail preference flips head/tail when configured. Object properties are ordered by key, and strings expand by grapheme with early characters prioritized over very deep expansions.
 - <sup><b>[3]</b></sup> <b>Choose top N nodes (binary search)</b>: Iteratively picks N so that the rendered preview fits within the byte budget, looping between “choose N” and a render attempt to converge quickly.
 - <sup><b>[4]</b></sup> <b>Render attempt</b>: Serializes the currently included nodes using the selected template. Omission summaries and per-file section headers appear in display templates (pseudo/js); json remains strict. For arrays, display templates may insert internal gap markers between non‑contiguous kept items using original indices.
 - <sup><b>[5]</b></sup> <b>Diagram source</b>: The Algorithm diagram is generated from `docs/diagrams/algorithm.mmd`. Regenerate the SVG with `cargo make diagrams` before releasing.

## License

MIT
