# Evaluation Notes

This document describes the local evaluation flow used during release work.
These materials are intended for repository-local validation and are not part
of the published package surface.

## Goals

The evaluation flow answers three questions:

1. Does the current workspace build and pass repository quality checks?
2. Does the default runtime path actually use the spaCy extractor and Leiden
   detector?
3. How does the default runtime path compare with the deterministic fallback
   path on a small TechQA sample?

## Environment

Use an isolated Python 3.11 environment for release validation. The scripts
assume the current repository is available locally and that the validation
environment can install the repository in editable mode.

Recommended validation environment:

- `~/Repo/temp/.venv`

## Runtime Check

Run the minimal runtime validation:

```bash
~/Repo/temp/.venv/bin/python scripts/check_runtime.py
```

This script executes:

- the default pipeline
- the deterministic fallback pipeline

It prints a compact JSON summary containing extractor and detector names,
concept counts, community counts, and label IDs.

## TechQA Comparison

Run the bounded TechQA comparison:

```bash
~/Repo/temp/.venv/bin/python scripts/run_techqa_eval.py
```

Optional output file:

```bash
~/Repo/temp/.venv/bin/python scripts/run_techqa_eval.py \
  --output /Users/huruilizhen/Repo/temp/techqa_eval_rerun.json
```

The script samples a small number of paragraphs directly from the cached
TechQA tarball and evaluates:

- the default pipeline
- the deterministic fallback pipeline

The summary includes:

- sampled paragraph previews
- extractor and detector names
- concept count
- community count
- first extracted concepts
- first paragraph labels

## Repository Checks

Run repository quality checks from the project root:

```bash
.venv/bin/ruff check .
.venv/bin/pyright
.venv/bin/pytest -q
```

## Recommended Release Validation Sequence

1. Confirm the current branch and commit SHA.
2. Install the current workspace into the isolated validation environment.
3. Run `scripts/check_runtime.py`.
4. Run `scripts/run_techqa_eval.py`.
5. Run repository quality checks.
6. Record any dependency anomalies before release.

## Notes

- If the default spaCy path fails, verify the configured model installation
  first.
- If the default spaCy path still fails, inspect the current spaCy and
  pydantic combination before assuming the model is missing.
- The fallback path should remain available as a deterministic baseline for
  debugging and comparison.
