# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- CONTRIBUTING.md with development setup, testing, code style, and PR guidelines

### Changed
- Eval script confirmation prompt now shows full script content (up to 10 lines) instead of only the first line
- Cache hashing uses SHA-256 instead of MD5 (invalidates existing eval caches)
- Prompt template loading uses `safe_substitute` — unrecognized `$`-patterns in eval content pass through instead of raising
- Isolated eval HOME copies only root-level credential files from `~/.claude` instead of symlinking the entire directory
- Bandit subprocess suppressions (B603, B607) scoped to inline comments instead of global skip list

### Fixed
- Bump cachetta minimum to 0.6.0 for pickle deserialization security fix

### Changed
- Patch releases now support switchable strategy via `RELEASE_STRATEGY` repo variable: `nightly` (default, current behavior) or `immediate` (release on every push to main). Includes `[no-release]` commit message flag to skip releases in immediate mode
- CLI: removed short flags that collided across commands — `-t` from `eval --tools` (conflicts with `tune --target`), `-p` from `create --prompt` (conflicts with `eval/tune --parallel`), `-m` from `generate-evals --max` (conflicts with `eval --max-evals`). Use the long form instead
- Coverage threshold raised from 55% to 85% to match actual coverage floor
- `evaluate()` returns `EvaluateResult` dataclass instead of `dict`
- `compare()` returns `CompareResult` dataclass instead of `dict`
- `show()` returns `ShowResult` dataclass instead of `dict`
- `create_skill()` returns `CreateSkillResult` dataclass instead of `dict`
- Performance: `hash_directory()` caches results per-process, eliminating redundant file reads across eval iterations
- Performance: `evaluate()` and `tune()` accept optional `evals_list` parameter, avoiding redundant YAML loading
- Performance: package imports now use lazy loading; `import skillet` no longer triggers heavy DSPy/numpy imports
- **Breaking:** `evals_to_trainset()` now accepts `list[dict]` (from `load_evals()`) instead of an eval name string

### Added
- `show` function exported from `skillet` package
- Result dataclass types: `EvaluateResult`, `IterationResult`, `PerEvalMetric`, `CompareResult`, `CompareEvalResult`, `ShowResult`, `ShowEvalResult`, `CreateSkillResult`
- `skillet eval --no-summary` flag to skip the failure summary LLM call

### Fixed
- Docs: `result.result.best_pass_rate` in README and Python API reference corrected to `result.result.final_pass_rate`
- Docs: added missing `show` command to CLI reference, all 7 commands to README
- Docs: added `per_eval_metrics` to `evaluate()` return schema, `show()` and `lint_skill()` to Python API reference
- Docs: documented `SKILLET_DIR` environment variable in getting-started guide and CLI reference

### Changed
- Eval cache now uses [cachetta](https://pypi.org/project/cachetta/) for persistence instead of bespoke YAML serialization. Existing cached results will be invalidated
- E2E tests: converted remaining subprocess-based tests (lint, error cases) to curtaincall PTY-based testing with exit code assertions

### Added
- Display: compact mode for eval/tune when many evals exceed terminal height — shows a single summary line with status counts instead of per-eval rows
- CLI: `skillet show <name>` command to inspect cached eval results. Use `--eval` to view detailed iteration transcripts (response, tool calls, judgment) and `--skill` to view results with a skill loaded instead of baseline
- Docs: conceptual guides — Skills vs Agents, Capability vs Regression Evals, Balanced Problem Sets
- README: inline Python API examples for `evaluate()` and `tune()`

### Fixed
- `skillet tune` no longer overwrites the original skill file during iteration — the file is only updated with the best version after tuning completes

### Changed
- `resolve_skill_path` now accepts any file when passed directly, not just `SKILL.md`. Directory resolution still looks for `SKILL.md` by convention
- Dev dependency: added `curtaincall>=0.2.0` for PTY-based terminal testing of CLI output
- E2E tests: converted eval, compare, create, generate-evals, and tune tests from subprocess to curtaincall PTY-based testing

### Added
- CLI: `skillet eval` now reports per-eval pass@k and pass^k metrics when running with multiple samples, measuring capability and consistency respectively
- Eval domains: generated evals are now tagged with a domain (`triggering`, `functional`, or `performance`) indicating what aspect of the skill is being tested
- CLI: `--domain` / `-d` flag on `generate-evals` to filter generated evals to specific domains (e.g., `--domain triggering`)

### Fixed
- `query_structured()` now defers exceptions to avoid abandoning the SDK async generator, preventing `RuntimeError` from anyio cancel scopes on error paths (validation failures, backtick canary)
- `skillet.__version__` now reports the correct package version instead of hardcoded `"0.1.0"`

### Added
- CLI: `skillet lint <path>` command with 14 built-in rules covering naming conventions, frontmatter structure, field presence, and file layout. Supports `--no-llm` to skip LLM-assisted checks and `--list-rules` to list available rules
- CLI: `skillet generate-evals <skill>` command to generate candidate eval files from a SKILL.md
- `query_structured()` SDK helper for type-safe structured output with Pydantic models
- `StructuredOutputError` canary to detect misconfigured structured output (backticks in response)
- Docs: comprehensive documentation for `tune` command, eval format, Python API, and `/skillet:add` guide
- E2E test: pirate skill tuning validation with calibrated evals
- Tune display: live progress with per-eval pass/fail symbols and pass rate percentages
- DSPy integration for skill tuning with MIPROv2-inspired instruction optimization
- `TuneConfig` and `TuneCallbacks` dataclasses for cleaner tune API
- `ClaudeAgentLM` - Custom DSPy language model using Claude Agent SDK
- `SkilletMIPRO` - MIPROv2 wrapper with callback hooks for progress reporting
- `propose_instruction` function for DSPy-powered instruction generation
- Security: `--trust` flag for eval command to skip script confirmation prompt
- Security: Interactive confirmation prompt when evals contain setup/teardown scripts
- Static analysis: mccabe complexity (C90) and pylint refactor (PLR) rules in ruff config
- Shared `get_rate_color()` helper for consistent pass rate color coding

### Changed
- **BREAKING:** CLI command `new` renamed to `create` for clarity
- Removed `notes/` folder from version control (now gitignored)
- **BREAKING:** CLI flag `--gaps/-g` renamed to `--max-evals/-m` for consistency
- Renamed `skillet/gaps/` module to `skillet/evals/` for terminology consistency
- Renamed `get_cached_results_for_gap` to `get_cached_results_for_eval`

### Fixed
- `query_structured()` now fully consumes the SDK async generator, preventing `RuntimeError` from anyio cancel scopes when running evals in parallel
- E2E test for `create` command (was still using old `new` command name)
- Add `SlashCommand` to `DEFAULT_SKILL_TOOLS` so eval prompts with `/command` syntax work
- Judge uses SDK structured output to guarantee clean JSON (removes need for backtick stripping)
- Skill draft and improve functions now use structured output (removes backtick stripping)
- Resource leaks: use `TemporaryDirectory` context manager instead of manual cleanup
- File operations now have proper error handling with `SkillError` exceptions
- Judge now uses structured JSON output instead of text parsing for reliable pass/fail detection
- Critical exceptions (KeyboardInterrupt, SystemExit) now propagate instead of being suppressed
- Setup/teardown scripts now have a 30-second timeout to prevent hanging
- `asyncio.run()` in DSPy metric no longer fails in async contexts
- Race condition in parallel cache access using file locking
- Docs site body text now uses Inter font instead of JetBrains Mono
- Docs navigation uses flat pages (removed group labels, matches agentskills.io)
- Docs footer removed (matches agentskills.io)
- Mintlify docs.json navigation schema updated to object format

### Changed
- Docs accent color changed to warm taupe (#8B7355) for brand identity
- Docs headings use Plus Jakarta Sans font for visual distinction
- Refactored `cli/commands/` to folder structure with helper functions in separate files
- Extracted prompts from Python code into colocated `.txt` files with `${var}` templating
- Refactored `eval/` module to split `run.py` into `evaluate.py`, `isolated_home.py`, `run_script.py`, `run_prompt.py`
- Refactored `compare/` module to split `run.py` into `compare.py`, `calculate_pass_rate.py`, `get_cached_results_for_eval.py`
- Migrated documentation from VitePress to Mintlify platform
- Bump actions/checkout from v4 to v6 in CI workflows

### Added
- `docs/components/reactive-docs/` - Reactive documentation system with [Show me] automation
- `docs/components/ReactiveDocsLayout` - Split-pane layout with docs panel and terminal
- `docs/components/DocsPanel` - Tutorial step display with progress indicator
- `Terminal` component now exposes `executeCommand` via ref for programmatic control
- `docs/components/UnifiedTerminal` - Unified LLM interface with seamless backend switching
- `docs/components/llm/` - Unified LLM backend interface abstracting over Claude API and WebLLM
- `docs/components/LocalLLMTerminal` - Browser-based LLM using WebLLM for zero-friction onboarding
- `docs/components/webllm.ts` - WebLLM client wrapper with same interface as Claude client
- `docs/components/ClaudeTerminal` - Terminal with Claude AI integration for interactive docs
- `docs/components/ApiKeyInput` - BYOK (Bring Your Own Key) component for Anthropic API key
- `docs/components/claude.ts` - Lightweight Claude client for browser environments
- `docs/components/Terminal` - React component with xterm.js and WebContainer for interactive docs
- Unit tests for cache, gaps/load, tune/improve, compare/run, eval/judge, cli/display, eval/run, cli/commands/tune, cli/commands/compare, sdk modules (coverage raised to 55%)
- Comprehensive unit tests for eval/run, eval/judge, tune/run, tune/improve, skill/create, skill/draft, cli/main, cli/commands/compare (coverage raised to 77%)
- Tool call capture in eval results - judge now sees which tools Claude used
- `--skip-cache` flag for eval command to ignore cached results
- `TuneResult` dataclass with full iteration history (inspired by DSPy)
- `--output` flag for tune command to save results JSON
- DSPy integration: `skillet.optimize` module with metric adapter for prompt optimization
- `SkillModule` class for wrapping skills as DSPy modules
- `load_evals` function replaces `load_gaps` (breaking change for direct imports)
- `evals_to_trainset()` function to convert skillet evals to DSPy Examples
- `optimize_skill()` function for DSPy-based skill optimization (BootstrapFewShot, MIPROv2)
- Unit tests for `skillet.optimize` module (100% coverage)
- `notes/` folder for project documentation and strategy discussions

### Changed
- Tune no longer modifies original skill file - uses tmpfile during tuning
- Tune returns `TuneResult` with all rounds, evals, and best skill content
- Tune output now compares against baseline (round 1) instead of 100% target
- Tune auto-saves results to `~/.skillet/tunes/{eval_name}/{timestamp}.json` by default

### Removed
- Duplicate output in tune command (finalize was printing after live display)

### Fixed
- Tune now accepts direct .md file paths (not just directories with SKILL.md)
- Tune display now shows all evals from the start of each round (was only showing first few)
- Tune evals now run correctly (was crashing due to env=None passed to SDK)
- Capture and display stderr output from Claude CLI during evals
- Symlink ~/.claude to isolated HOME so evals can access credentials

### Documentation
- CLAUDE.md: Always run worktree setup steps, never give user commands to run

### Changed
- Renamed "Gaps" to "Evals" in CLI output
- Renamed eval files with descriptive suffixes (e.g., `001.yaml` → `001-ask-expectation.yaml`)
- **Nightly releases are now patch-only.** Minor releases are triggered manually via the "Minor Release" workflow. This prevents version inflation from mislabeled commits.
- Renamed `version-bump.yml` to `patch-release.yml` for clarity
- Updated commit type guidelines in CLAUDE.md with clearer distinction between `feat:`, `test:`, and `chore:`

### Added
- `minor-release.yml` workflow for manual minor version bumps
- Single eval file support (`skillet eval ./evals/my-skill/001.yaml`)
- ROADMAP.md with planned features
- CLAUDE.md guidelines: check uv.lock for versions, verify commands in docs
- 7 folder-suggestion evals for `/skillet:add` command testing
- Recursive eval loading (supports subdirectories in eval folders)
- Container development docs in CLAUDE.md (use `/.venv` for isolated venv)
- Isolated HOME environment for every eval execution
- Optional `setup` and `teardown` scripts in eval YAML for pre/post test hooks
- Git worktree workflow documentation in CLAUDE.md
- RELEASING.md documenting the fully automated release process
- Template-based command system for `{{SKILLET_DIR}}` substitution
- `scripts/build_claude_config.py` for building `.claude/commands/` from templates
- `just build-claude` command for template building
- E2E test helpers with `Conversation` class for multi-turn testing
- Auto-build of commands in e2e test conftest
- 13 eval files for testing `/skillet:add` command behavior
- Multi-turn conversation support in eval system
- Session resumption for sequential prompts in evals
- Automated nightly release workflow (2am UTC) triggered by git tags
- Manual trigger for release workflow (workaround for GITHUB_TOKEN limitation)
- Skip release if tag already exists (prevents duplicate tag errors)

### Changed
- Consolidated release workflow into nightly workflow (fully automated, no manual trigger needed)
- Fixed README install command (`skillet` → `pyskillet`)
- Switched to dynamic versioning via `hatch-vcs` (version derived from git tags at build time)
- Simplified nightly release workflow to only create/push tags (no commits to main needed)
- Simplified release workflow to always publish (removed unnecessary changes detection)
- Renamed PyPI package from `skillet` to `pyskillet` (name was taken)
- Renamed `/skillet:gap` command to `/skillet:add`
- Increased `max_turns` from 3 to 10 for complex command workflows
- Reorganized codebase into domain modules (eval/, compare/, tune/, skill/, gaps/)
- Separated CLI layer from core API logic
- Renamed public API functions: `run_eval` -> `evaluate`, `run_compare` -> `compare`, `run_tune` -> `tune`
- Moved internal utilities to `_internal/` module

### Fixed
- `load_gaps` now accepts paths (e.g., `./evals/add-command/`) in addition to names
- Cache normalizes paths to directory name for consistent caching
- Eval CLI live display now updates in real-time during execution
- Release workflow permissions for checkout
- Nightly release tag check now uses remote refs instead of local (semantic-release creates local tags)
- Removed redundant commit/tag step since semantic-release already creates them
- Release workflow now uses `fetch-depth: 0` so hatch-vcs can read tags for versioning
- semantic-release config to update pyproject.toml version

### Removed
- `GapError` exception (use `EvalError` instead)
- Separate `release.yml` workflow (now part of nightly workflow)
- GitHub Pages deployment from release workflow (not needed)

### Added
- Initial CLI with `eval`, `new`, `tune`, and `compare` commands
- Gap capture via `/skillet:gap` Claude Code command
- Evaluation caching system
- LLM-as-judge for response validation
- Live display with spinner animation for parallel evals
- Python linting with Ruff
- pytest testing infrastructure with pytest-describe
- Pre-commit hooks for linting and tests
- CI workflows for lint, test, and build
- PR monitor for CI gating
- PEP 561 py.typed marker for type checking support
