# Poetry Benchmarking - Quick Start Guide

## Overview

The benchmarking script (`benchmarking.py`) runs poetry domain scenarios and reports planning performance. It uses a `PoetryBenchmark` class that extends the shared benchmarking infrastructure from `ipc-2020-total-order/` to support task-list-based goals (rather than goal dictionaries).

## Prerequisites

- **GTPyhop 1.9.0+** installed
- **psutil** package (for resource tracking)
- Python 3.8 or later

### Installing Dependencies

```bash
pip install gtpyhop psutil
```

Or for local development:
```bash
cd C:\Users\Eric JACOPIN\Documents\Code\Source\GTPyhop
pip install -e .
pip install psutil
```

## Available Examples

| # | Example | Description | Scenarios | Actions | Strategy Required |
|---|---------|-------------|-----------|---------|-------------------|
| 1 | `structured_poetry` | Baseline poetry generation with one method per task | 6 (couplet, limerick, haiku, sonnet) | 8-44 | Any |
| 2 | `backtracking_poetry` | Extension testing HTN backtracking for rhyme selection | 3 (couplet, limerick, haiku) | 8-17 | Backtracking* |
| 3 | `candidate_planning_poetry` | Multi-candidate rhyme selection pipeline | 3 (couplet, limerick, haiku) | 8-27 | Any |
| 4 | `bidirectional_planning_poetry` | Decomposed backward line construction | 3 (couplet, limerick, haiku) | 8-22 | Any |
| 5 | `replanning_poetry` | Post-generation evaluation and steering/revision | 3 (couplet, limerick, haiku) | 8-28 | Backtracking* |

\* Backtracking required for rhymed forms. Haiku works with any strategy.

All examples follow the **GTPyhop 1.9.0+ style guide** with unified scenario block format.

## Running the Benchmarking Script

### Basic Usage

```bash
cd src/gtpyhop/examples/poetry

# List available domains
python benchmarking.py --list-domains

# Run structured poetry scenarios (works with default strategy)
python benchmarking.py structured_poetry

# Run candidate planning poetry scenarios (works with default strategy)
python benchmarking.py candidate_planning_poetry

# Run bidirectional planning poetry scenarios (works with default strategy)
python benchmarking.py bidirectional_planning_poetry

# Run backtracking poetry scenarios (requires backtracking strategy)
python benchmarking.py backtracking_poetry --strategy recursive_dfs

# Run replanning poetry scenarios (requires backtracking strategy)
python benchmarking.py replanning_poetry --strategy iterative_dfs_backtracking
```

### Command-Line Options

```bash
# Run with minimal output (verbosity 0)
python benchmarking.py structured_poetry --verbose 0

# Run with detailed output (verbosity 2)
python benchmarking.py structured_poetry --verbose 2

# Run with maximum debugging output (verbosity 3)
python benchmarking.py backtracking_poetry --strategy recursive_dfs --verbose 3

# Run using legacy mode (global state, not recommended)
python benchmarking.py structured_poetry --legacy-mode

# Combine options
python benchmarking.py replanning_poetry --strategy recursive_dfs --verbose 2
```

### Planning Strategies

The `--strategy` option selects the planning strategy:

| Strategy | Backtracking | Default | When to Use |
|----------|-------------|---------|-------------|
| `iterative_greedy` | None | Yes | Examples 1, 3, 4 (no backtracking needed) |
| `recursive_dfs` | Via call stack | No | Examples 2, 5 (backtracking required) |
| `iterative_dfs_backtracking` | Via explicit stack | No | Examples 2, 5 (backtracking required) |

**Note:** Running examples 2 or 5 without a backtracking strategy will produce `False` for scenarios that require backtracking. This is expected behavior, not a bug.

### Verbosity Levels

- **0**: Silent (only final summary)
- **1**: Normal (default) - shows plan found/not found
- **2**: Detailed - shows task decomposition and method selection
- **3**: Debug - shows all internal planner operations

### Planning Modes

- **session** (default): Uses PlannerSession (thread-safe, recommended for GTPyhop 1.9.0+)
- **legacy**: Uses global state (enabled with `--legacy-mode` flag, not recommended)

## Running with Different Planning Strategies via Python API

For more control, use the Python API directly:

```python
import gtpyhop
from gtpyhop import PlannerSession
from gtpyhop.examples.poetry.replanning_poetry import the_domain, get_problems

problems = get_problems()
state, tasks, desc = problems['scenario_1_couplet_stars']

# Recursive DFS (backtracks via Python call stack)
with PlannerSession(domain=the_domain, strategy="recursive_dfs", verbose=1) as session:
    with session.isolated_execution():
        result = session.find_plan(state, tasks)
        print(f"Recursive DFS: {len(result.plan)} actions" if result.success else "Failed")

# Iterative DFS Backtracking (backtracks via explicit stack)
with PlannerSession(domain=the_domain, strategy="iterative_dfs_backtracking", verbose=1) as session:
    with session.isolated_execution():
        result = session.find_plan(state, tasks)
        print(f"Iterative DFS BT: {len(result.plan)} actions" if result.success else "Failed")
```

## Expected Results by Strategy

### Examples 1, 3, 4 (No Backtracking Needed)

All three strategies produce identical plans for these examples.

#### Structured Poetry

| Scenario | Actions |
|----------|---------|
| `scenario_1_couplet_autumn` | 8 |
| `scenario_2_limerick_cat` | 17 |
| `scenario_3_haiku_ocean` | 8 |
| `scenario_4_sonnet_time` | 44 |
| `scenario_5_couplet_stars` | 8 |
| `scenario_6_limerick_code` | 17 |

#### Candidate Planning Poetry

| Scenario | Actions |
|----------|---------|
| `scenario_1_couplet_stars` | 12 |
| `scenario_2_limerick_cat` | 27 |
| `scenario_3_haiku_ocean` | 8 |

#### Bidirectional Planning Poetry

| Scenario | Actions |
|----------|---------|
| `scenario_1_couplet_stars` | 10 |
| `scenario_2_limerick_cat` | 22 |
| `scenario_3_haiku_ocean` | 8 |

### Example 2: Backtracking Poetry

| Scenario | Recursive DFS | Iterative Greedy | Iterative DFS BT |
|----------|:---:|:---:|:---:|
| `scenario_1_couplet_stars` | 8 | 8 | 8 |
| `scenario_2_limerick_cat` | 17 | **False** | 17 |
| `scenario_3_haiku_ocean` | 8 | 8 | 8 |

The limerick fails with iterative greedy because label "A" is used 3 times in the AABBA scheme, and `a_select_rhyme_target_strict` fails on the 3rd use. Without backtracking, the planner cannot recover by trying `m_write_rhymed_line_relaxed`.

### Example 5: Replanning Poetry

| Scenario | Recursive DFS | Iterative Greedy | Iterative DFS BT |
|----------|:---:|:---:|:---:|
| `scenario_1_couplet_stars` | 12 | **False** | 12 |
| `scenario_2_limerick_cat` | 28 | **False** | 28 |
| `scenario_3_haiku_ocean` | 8 | 8 | 8 |

The couplet and limerick fail with iterative greedy because `a_evaluate_line` fails for lines requiring revision (subsequent uses of each rhyme label). Without backtracking, the planner cannot recover by trying `m_revise_line` after `m_accept_line` fails.

## Interpreting Results

### Successful Run Example

```
Loading domain: structured_poetry
Found 6 problems in structured_poetry

Running benchmarks for structured_poetry using Thread-Safe Sessions planning (strategy: iterative_greedy)...
Solving scenario_1_couplet_autumn...
Solving scenario_2_limerick_cat...
Solving scenario_3_haiku_ocean...
Solving scenario_4_sonnet_time...
Solving scenario_5_couplet_stars...
Solving scenario_6_limerick_code...

=== structured_poetry Benchmark Results ===

=== Benchmark Summary ===
Problem                      | Status | Plan Len | Time (s) |  CPU % | Mem (KB) | Peak Mem (KB)
---------------------------------------------------------------------------------------------
scenario_1_couplet_autumn    |   OK   |        8 |    0.001s |    0.0% |      ... |          ...
scenario_2_limerick_cat      |   OK   |       17 |    0.001s |    0.0% |      ... |          ...
scenario_3_haiku_ocean       |   OK   |        8 |    0.001s |    0.0% |      ... |          ...
scenario_4_sonnet_time       |   OK   |       44 |    0.002s |    0.0% |      ... |          ...
scenario_5_couplet_stars     |   OK   |        8 |    0.001s |    0.0% |      ... |          ...
scenario_6_limerick_code     |   OK   |       17 |    0.001s |    0.0% |      ... |          ...
```

### Expected Plan Lengths (Structured Poetry Baseline)

| Form | Formula | Actions |
|------|---------|---------|
| Couplet | 2 + (3 x 2) | 8 |
| Limerick | 2 + (3 x 5) | 17 |
| Haiku | 2 + (2 x 3) | 8 |
| Sonnet | 2 + (3 x 14) | 44 |

Where: `init + assemble + (select + generate + verify) x lines` for rhymed forms, and `init + assemble + (generate + verify) x lines` for unrhymed forms (haiku).

## Adding New Poetry Examples

To add a new poetry domain:

1. **Create a new subdirectory** under `poetry/` following the **GTPyhop 1.9.0+ style guide**:
   ```
   poetry/
   +-- your_new_example/
       +-- domain.py             # Required: domain creation + actions + methods
       +-- problems.py           # Required: defines problems with unified scenario blocks
       +-- __init__.py           # Required: exports domain and get_problems()
       +-- README.md             # Required
   ```

2. **Required files**:
   - `domain.py` - Must create domain, define all actions and methods following the [domain style guide](../../../../docs/gtpyhop_domain_style_guide.md)
   - `problems.py` - Must define problems using unified scenario blocks following the [problems style guide](../../../../docs/gtpyhop_problems_style_guide.md)
   - `__init__.py` - Must export `the_domain` and implement `get_problems()` function

3. **Run the benchmarking script** - It will automatically discover your new example:
   ```bash
   python benchmarking.py your_new_example
   # Or with backtracking if needed:
   python benchmarking.py your_new_example --strategy recursive_dfs
   ```

## Troubleshooting

### "Error: psutil module is required"

**Solution**: Install psutil:
```bash
pip install psutil
```

### "Could not import gtpyhop"

**Solution**: Install GTPyhop:
```bash
pip install gtpyhop
```

### "No module named 'your_domain_name'"

**Solution**: Ensure your domain directory has:
- `__init__.py` with proper exports
- `domain.py` with domain creation
- `problems.py` with unified scenario blocks

### "No plan found" for backtracking_poetry limerick

**This is expected behavior** with the default (iterative greedy) planner. The limerick requires backtracking. Use `--strategy recursive_dfs` or `--strategy iterative_dfs_backtracking`. See the "Planning Strategies" section above.

### "No plan found" for replanning_poetry couplet or limerick

**This is expected behavior** with the default (iterative greedy) planner. These scenarios require backtracking because `a_evaluate_line` fails for lines requiring revision. Use `--strategy recursive_dfs` or `--strategy iterative_dfs_backtracking`. See the "Planning Strategies" section above.

### "No plan found" (other scenarios)

**Possible causes**:
- Initial state doesn't satisfy preconditions
- Methods or actions have bugs
- Task decomposition is incorrect

**Debug steps**:
1. Run with `--verbose 3` to see detailed planner output
2. Check that initial states have all required properties
3. Verify method preconditions match action effects

## Next Steps

- Review individual domain README.md files for detailed documentation
- Examine generated plans to understand task decomposition
- Compare planning strategies on the backtracking_poetry and replanning_poetry domains
- Add new poetry forms or domains following the style guides

---
*Updated 2026-02-12*
