# Markov analysis

Biblicus provides a Markov analysis backend that learns a directed, weighted state transition graph from sequences of
text segments in a corpus. It is an exploratory analysis tool that produces structured, inspectable artifacts:

- A set of inferred states with per-state exemplars.
- A directed, weighted transition graph between states.
- A per-item decoded path that shows how each item traversed the state graph.
- Optional GraphViz exports for visualization.

Markov analysis is configured using YAML recipes, validated strictly, and stored as versioned run artifacts under the
corpus. It is designed for experimentation across segmentation strategies and observation encodings.

## Observation encoder configurability

The observation encoder configuration is fully configurable. For hybrid encoders, you can control:

- `observations.categorical_source`: which observation field supplies categorical labels
- `observations.numeric_source`: which observation field supplies the numeric scalar

This allows hybrid encodings to use fields other than the defaults (for example, `llm_summary` or `segment_index`)
without changing the pipeline code.

## Topic-driven observations

Markov analysis can run topic modeling over segments and use the resulting topic labels as categorical observations.
This is useful when you want topic buckets to act as the observation symbols.

The topic modeling recipe is embedded inside the Markov recipe:

```
schema_version: 1
segmentation:
  method: sentence
topic_modeling:
  enabled: true
  recipe:
    schema_version: 1
    llm_extraction:
      enabled: false
    lexical_processing:
      enabled: false
    bertopic_analysis:
      parameters: {}
observations:
  categorical_source: topic_label
model:
  family: categorical
  n_states: 6
```

When enabled, the Markov observations include `topic_id` and `topic_label`. Setting
`observations.categorical_source` to `topic_label` makes the topic labels the categorical symbols
used by the Markov model.

Topic-driven runs emit extra debugging artifacts:

- `topic_modeling.json`: the full topic modeling report used for the run.
- `topic_assignments.jsonl`: per-segment topic assignments with the original segment text.

## What Markov analysis does

Markov analysis treats each item as a sequence:

1) Start from extracted text artifacts for a corpus item.
2) Segment the text into an ordered sequence (sentences, fixed windows, or provider-backed segmentation).
3) Convert each segment into an observation vector (categorical labels, numeric features, embeddings, or combinations).
4) Fit a hidden-state Markov model to learn:
   - latent states,
   - transition probabilities between states,
   - and per-state emission distributions.
5) Decode a most-likely state sequence for each item and emit the state transition graph.

## Text extract segmentation

Text extract is a provider-backed segmentation strategy designed for long documents. Instead of asking the model to
re-emit the entire text with labels, Biblicus gives the model a virtual file and asks it to insert XML tags in-place.
The model uses **str_replace tool calls** (old_str/new_str pairs), which Biblicus applies in memory.

This pattern has two benefits:

1) The model only emits a small edit script, which is cheaper and faster than reprinting the full transcript.
2) The original text remains intact; validation can prove that only tags were inserted.

Biblicus uses XML-style tags (`<span>...</span>`) so the edits are well-formed and easy to validate deterministically.

After applying the tags, Biblicus parses **spans** (the tagged text ranges). Interstitial content remains available
in the marked-up string without forcing the model to cover every token.

Example snippet:

System prompt excerpt:

```
SYSTEM PROMPT (excerpt):
You are a virtual file editor. Use the available tools to edit the text.
Interpret the word "return" in the user's request as: wrap the returned text with
<span>...</span> in-place in the current text.
Current text:
---
Greeting. Verification. Resolution.
---
```

User prompt:

```
USER PROMPT:
Return the segments that represent contiguous phases in the text.
```

Input text:

```
INPUT TEXT:
Greeting. Verification. Resolution.
```

Marked-up text:

```
MARKED-UP TEXT:
<span>Greeting.</span> <span>Verification.</span> <span>Resolution.</span>
```

Structured data:

```
STRUCTURED DATA (result):
{
  "marked_up_text": "<span>Greeting.</span> <span>Verification.</span> <span>Resolution.</span>",
  "spans": [
    {"index": 1, "start_char": 0, "end_char": 10, "text": "Greeting."},
    {"index": 2, "start_char": 11, "end_char": 24, "text": "Verification."},
    {"index": 3, "start_char": 25, "end_char": 37, "text": "Resolution."}
  ],
  "warnings": []
}
```

## Run Markov analysis from the CLI

```
biblicus analyze markov --corpus corpora/ag_news_demo_2k --recipe recipes/markov/local-discovery.yml
```

Example span markup recipe (text extract provider-backed):

```
schema_version: 1
segmentation:
  method: span_markup
  span_markup:
    client:
      provider: openai
      model: gpt-4o-mini
      api_key: null
      response_format: json_object
    system_prompt: |
      You are a virtual file editor. Use the available tools to edit the text.
      Interpret the word “return” in the user’s request as: wrap the returned text
      with <span>...</span> in-place in the current text.

      Use the str_replace tool to insert <span>...</span> tags and the done tool when finished.
      When finished, call done. Do NOT return JSON in the assistant message.

      Rules:
      - Use str_replace only.
      - old_str must match exactly once in the current text.
      - old_str and new_str must be non-empty strings.
      - new_str must be identical to old_str with only <span> and </span> inserted.
      - Do not include <span> or </span> inside old_str or new_str.
      - Do not insert nested spans.
      - If a tool call fails due to non-unique old_str, retry with a longer unique old_str.
      - If a tool call fails, read the error and keep editing. Do not call done until spans are inserted.
      - Do not delete, reorder, paraphrase, or label text.
      Current text:
      ---
      {text}
      ---
    prompt_template: |
      Return the segments that represent contiguous phases in the text.

      Rules:
      - Preserve original order.
      - Do not add labels, summaries, or commentary.
      - Prefer natural boundaries like greeting/opening, identity verification, reason for call,
        clarification, resolution steps, handoff/escalation, closing.
      - Use speaker turn changes as possible boundaries, but keep multi-turn exchanges together if they
        form a single phase.
      - Avoid extremely short fragments; merge tiny leftovers into a neighboring span.
model:
  family: gaussian
  n_states: 4
observations:
  encoder: tfidf
```

To keep runs reproducible, pass an extraction run explicitly:

```
biblicus analyze markov \
  --corpus corpora/ag_news_demo_2k \
  --recipe recipes/markov/local-discovery.yml \
  --extraction-run pipeline:RUN_ID
```

### Cascading recipes and CLI overrides

Markov analysis recipes support cascading composition. You can pass multiple `--recipe` files; later recipes override
earlier recipes via a deep merge:

```
biblicus analyze markov \
  --corpus corpora/ag_news_demo_2k \
  --recipe recipes/markov/base.yml \
  --recipe recipes/markov/guided.yml
```

To override the composed configuration view from the command line, use `--config key=value` with dotted keys:

```
biblicus analyze markov \
  --corpus corpora/ag_news_demo_2k \
  --recipe recipes/markov/base.yml \
  --recipe recipes/markov/guided.yml \
  --config model.n_states=14
```

Omitted fields use the default values from the Markov analysis schema. Missing required fields remain hard errors.

## Output location and artifacts

Markov analysis output is stored under:

```
analysis/markov/<run_id>/
```

The run directory contains a manifest and structured artifacts. The canonical output is `output.json`. Additional files
provide intermediate visibility, such as segments and observations used to fit the model. When enabled, GraphViz output
is written as `transitions.dot`.

## Working demo

The integration demo script is a working reference you can use as a starting point:

```
python scripts/markov_analysis_demo.py --corpus corpora/ag_news_demo_2k
```

If you want the script to download and ingest AG News into a fresh corpus directory, pass `--download` (this requires the
optional datasets dependency).

Markov analysis requires an optional dependency:

```
python -m pip install "biblicus[markov-analysis]"
```

The demo builds or reuses an extraction run, executes Markov analysis with example recipes, and prints the resulting run
paths. Inspect the emitted `output.json` and graph artifacts to understand states and transitions.
