# rusty-jq

A jq-like JSON query engine for Python, written in Rust using PyO3 and Maturin.

## Architecture

The crate is a **PyO3 `cdylib`** that exposes a `compile(query) → RustyProgram` function to Python.

### Modules

| Module | Path | Role |
|---|---|---|
| `lib.rs` | `src/lib.rs` | PyO3 bindings, simd-json → Python type conversion (`value_to_py`), module entry-point |
| `parser.rs` | `src/parser.rs` | Nom-based parser — tokenises jq expressions into `Vec<RustyFilter>` |
| `engine.rs` | `src/engine.rs` | Execution engine — walks a `BorrowedValue` tree using `Cow` semantics for zero-copy traversal |

### Data Flow

```
Python str ──► compile(query) ──► Vec<RustyFilter>  (parsed once, reused N times)
Python str ──► .input(json)   ──► simd_json::to_borrowed_value (in-place, mutates bytes)
                                   ──► process_rust_value(Cow::Borrowed, filters)
                                   ──► value_to_py  ──► RustyJqIter (Python iterator)

Python str ──► .first(json)   ──► same pipeline, returns only result[0] as PyObject (or None)
```

### Key Types

- **`RustyFilter`** (enum, `parser.rs`): `Identity | Select(String) | Index(i32) | Iterator | Object(Vec<(String, Vec<RustyFilter>)>)`
- **`RustyProgram`** (pyclass, `lib.rs`): holds compiled `Vec<RustyFilter>`; exposes `.input(json_text)` and `.first(json_text)` to Python
- **`RustyJqIter`** (pyclass, `lib.rs`): Python iterator — wraps `std::vec::IntoIter<PyObject>`

### Python API

```python
import rusty_jq

program = rusty_jq.compile(".users | .[] | .name")

# Returns a Python iterator over all results
for name in program.input(json_text):
    print(name)

# Returns the first result as a single Python object (None if no results)
first_name = program.first(json_text)
```

## Supported Filters

| Filter | Syntax | Engine Behaviour |
|---|---|---|
| Identity | `.` | Passes value through unchanged |
| Field access | `.field` | Descends into object key; silently drops if key missing |
| Index | `.[n]` | Positive or negative integer index into array; silently drops if out of bounds |
| Iterator | `.[]` | Expands every element of an array into separate results |
| Pipe | `\|` | Chains filters; each filter receives all outputs of the previous |
| Object construction | `{key: expr, ...}` | Builds a new object; uses Cartesian product if any field yields multiple values |

### Object Construction — Cartesian Product Semantics

When a field expression yields multiple values (e.g. via `.[]`), the engine produces one output object **per combination** of field values:

```
.[] | {id: .id}  over [{id:1},{id:2}]
→ {id:1}, {id:2}          ← two objects, not one
```

If any field expression yields **zero** results, the entire object is dropped.

### Field Name Grammar (`parse_word`)

Field names in `.field` and `{key: ...}` must match:

```
[a-zA-Z0-9_] [a-zA-Z0-9_-]*
```

Hyphens are allowed after the first character (e.g. `.some-field` works). Keys in object constructors follow the same rule.

## Parser — Important Implementation Notes

`parse_single_filter` uses `nom::branch::alt` with a fixed order that matters:

```rust
alt((
    parse_iterator,   // .[   ]  — must come before parse_index and parse_select
    parse_index,      // .[n]    — must come before parse_select (both start with `.[`)
    parse_select,     // .field
    parse_object,     // {...}
    parse_identity    // .       — must be last (matches any lone dot)
))
```

`parse_query` splits on `|` using `separated_list1`; each segment is a single filter.

## Tech Stack

- **Rust 2021 edition**
- **PyO3 0.20** — Rust ↔ Python FFI
- **Maturin ≥ 1.0** — build backend (`pyproject.toml`)
- **nom 7** — parser combinators
- **simd-json 0.17** — zero-copy JSON parsing (`BorrowedValue`)
- **pytest** — Python test suite

## Development

### Prerequisites

```bash
# A Python virtualenv with maturin and pytest installed
pip install maturin pytest jq
```

### Build & install (dev mode)

```bash
# Activate virtualenv first
maturin develop
```

### Run tests

```bash
pytest test/
```

### Run benchmarks

```bash
python test/benchmark.py
# Compares rusty_jq against jq (Python bindings) and jaq (CLI binary)
# Requires jaq to be on PATH
```

## Project Layout

```
rusty-jq/
├── Cargo.toml           # Rust crate config (cdylib)
├── pyproject.toml       # Python/Maturin build config
├── src/
│   ├── lib.rs           # PyO3 entry-point, value_to_py, RustyProgram, RustyJqIter
│   ├── parser.rs        # Nom query parser → Vec<RustyFilter>
│   └── engine.rs        # process_rust_value — Cow-based filter pipeline
└── test/
    ├── test_lib.py      # pytest suite (parametrized)
    └── benchmark.py     # Performance benchmarks vs jq and jaq
```

## Code Conventions

- Use `Cow<'a, BorrowedValue<'a>>` throughout the engine; only clone on the `Cow::Owned` path.
- Parser functions follow nom conventions: `fn parse_*(input: &str) -> IResult<&str, T>`.
- Adding a new filter requires **three** coordinated changes:
  1. Add a variant to `RustyFilter` in `parser.rs`
  2. Add a `parse_*` function and register it in `parse_single_filter`'s `alt(...)` (mind the order)
  3. Add the matching arm in `engine.rs::process_rust_value`
- Python-facing items use `#[pyclass]` / `#[pyfunction]` / `#[pymethods]`.
- Tests use `@pytest.mark.parametrize`; use `program.first()` for single-value assertions and `list(program.input())` for multi-value assertions.
- `simd_json::to_borrowed_value` mutates the input byte buffer in place — `json_text.as_bytes().to_vec()` is required to get an owned mutable copy.
