# Investigation Report: Issue #1370

## Issue Summary

**Title:** Detected changes across multiple hosts are wrong

**Reporter:** @dfaerch

**Labels:** bug, CLI

**URL:** https://github.com/pyinfra-dev/pyinfra/issues/1370

### Problem Description

When running pyinfra with multiple hosts where each host has a different list of items to iterate over in a loop, the "Detected changes" output displays incorrect information:

1. Operations are incorrectly attributed to hosts that shouldn't have them
2. Some operations are missing from the output entirely
3. Operation names from one host appear on another host's operations

**Critical Note:** The actual deployment works correctly - only the "Detected changes" display is wrong.

### Reproduction Case

**inventory.py:**
```python
hosts = [
    ('server01/incus01', {
        'users': ['user-1', 'user-2', 'user-3', 'user-4'],
        'ssh_hostname': 'server01.example.com',
        'ssh_port': 2222,
        'ssh_user': 'pyinfra+incus01'
    }),
    ('server01/incus02', {
        'users': ['user-5'],
        'ssh_hostname': 'server01.example.com',
        'ssh_port': 2222,
        'ssh_user': 'pyinfra+incus02'
    })
]
```

**deploy_test.py:**
```python
from pyinfra.operations import server
from pyinfra import host

users = host.data.get("users")

for user in users:
    server.shell(
        name=f"Create {user} user on {host.name}",
        commands=[f"echo {user}"],
    )
```

**Actual Output:**
```
--> Detected changes:
    Operation                                 Change
    Create user-5 user on server01/incus02   2 (server01/incus01, server01/incus02)
    Create user-2 user on server01/incus01   1 (server01/incus01)
    Create user-3 user on server01/incus01   1 (server01/incus01)
    Create user-4 user on server01/incus01   1 (server01/incus01)
```

**Expected Output:**
```
--> Detected changes:
    Operation                                 Change
    Create user-1 user on server01/incus01   1 (server01/incus01)
    Create user-2 user on server01/incus01   1 (server01/incus01)
    Create user-3 user on server01/incus01   1 (server01/incus01)
    Create user-4 user on server01/incus01   1 (server01/incus01)
    Create user-5 user on server01/incus02   1 (server01/incus02)
```

---

## Root Cause Analysis

### The Operation Hash Problem

The issue is in how pyinfra generates and handles operation hashes when the same code location is executed with different data on different hosts.

#### Key Files Involved:

1. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py`** - Operation hash generation
2. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/util.py`** - Hash creation utilities
3. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra_cli/prints.py`** - "Detected changes" output
4. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/state.py`** - State management and operation ordering

### How Operation Hashing Works

In `operation.py`, the `solve_operation_consistency()` function (lines 410-441) generates the operation hash:

```python
def solve_operation_consistency(names, state, host):
    if pyinfra.is_cli:
        op_order = get_operation_order_from_stack(state)
    else:
        op_order = [len(host.op_hash_order)]

    if host.loop_position:
        op_order.extend(host.loop_position)

    # Make a hash from the call stack lines
    op_hash = make_hash(op_order)

    # Avoid adding duplicates! This happens if an operation is called within
    # a loop - such that the filename/lineno/code _are_ the same, but the
    # arguments might be different.
    duplicate_op_count = 0
    while op_hash in host.op_hash_order:
        logger.debug("Duplicate hash ({0}) detected!".format(op_hash))
        op_hash = "{0}-{1}".format(op_hash, duplicate_op_count)
        duplicate_op_count += 1

    host.op_hash_order.append(op_hash)
```

### The Bug

**The operation hash is based solely on the call stack (line numbers), NOT on the operation arguments or name.**

When the deploy file is executed:
1. **Host 1 (incus01)** runs the loop with 4 iterations, creating hashes based on line numbers + loop position
2. **Host 2 (incus02)** runs the loop with 1 iteration, also creating a hash based on line numbers

**The first iteration for BOTH hosts gets the same base hash** because:
- Both are at the same line number in the deploy file
- `host.loop_position` starts at `[0]` for both

From the debug log:
```
[pyinfra.api.operation] Adding operation, {'Create user-5 user on server01/incus02'}, opOrder=(0, 8), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3
[pyinfra.api.operation] Adding operation, {'Create user-1 user on server01/incus01'}, opOrder=(0, 8), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3) detected!
```

**Both hosts get the same hash `e7b37f7edef2ef843cea2aa2075da5d7f922fee3` for their first loop iteration!**

### How This Causes the Display Bug

In `prints.py`, the `print_meta()` function (lines 238-280) iterates through operations:

```python
def print_meta(state: "State"):
    for op_hash in state.get_op_order():
        hosts_in_op = []
        for host in state.inventory.iter_activated_hosts():
            if op_hash in state.ops[host]:
                op_data = state.get_op_data_for_host(host, op_hash)
                if op_data.operation_meta._maybe_is_change:
                    # ...
                    hosts_in_op.append(host.name)

        rows.append((
            logger.info,
            [
                pretty_op_name(state.op_meta[op_hash]),  # Uses shared op_meta!
                # ... host count
            ],
        ))
```

The problem:
1. `state.op_meta[op_hash]` stores shared metadata including operation `names`
2. When both hosts have the same hash, the name from the **second host processed** overwrites the first
3. The `pretty_op_name()` function picks **the first name from the set**, which may be from a different host

In `ensure_shared_op_meta()` (lines 445-466):
```python
def ensure_shared_op_meta(state, op_hash, op_order, global_arguments, names):
    op_meta = state.op_meta.setdefault(op_hash, StateOperationMeta(op_order))
    # ...
    op_meta.names.update(names)  # Adds names from ALL hosts with this hash!
    return op_meta
```

---

## Proposed Fix

### Option 1: Include Host Name in Operation Hash (Recommended)

Modify `solve_operation_consistency()` to include host identification in the hash:

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py`

```python
def solve_operation_consistency(names, state, host):
    if pyinfra.is_cli:
        op_order = get_operation_order_from_stack(state)
    else:
        op_order = [len(host.op_hash_order)]

    if host.loop_position:
        op_order.extend(host.loop_position)

    # Make a hash from the call stack lines AND include a host-specific component
    # to ensure operations with different arguments on different hosts get unique hashes
    hash_input = list(op_order)
    hash_input.append(host.name)  # ADD THIS LINE
    op_hash = make_hash(hash_input)  # CHANGE THIS LINE

    # Rest of function remains the same...
```

**Pros:**
- Simple change
- Each host gets unique hashes for their operations
- Maintains correct ordering within each host

**Cons:**
- Breaks cross-host operation grouping (operations on different hosts will always be separate)
- May affect features that rely on operations having the same hash across hosts

### Option 2: Store Per-Host Operation Names (Alternative)

Instead of modifying the hash, store operation names per-host:

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/state.py`

Change `StateOperationMeta`:
```python
class StateOperationMeta:
    names: set[str]  # Keep for backward compat
    host_names: dict[str, set[str]]  # ADD: host-specific names
    args: list[str]
    op_order: tuple[int, ...]
    global_arguments: "AllArguments"

    def __init__(self, op_order: tuple[int, ...]):
        self.op_order = op_order
        self.names = set()
        self.host_names = {}  # ADD THIS
        self.args = []
        self.global_arguments = {}
```

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py`

In `ensure_shared_op_meta()`:
```python
def ensure_shared_op_meta(state, op_hash, op_order, global_arguments, names):
    host = context.host  # ADD: Get current host
    op_meta = state.op_meta.setdefault(op_hash, StateOperationMeta(op_order))
    # ...
    op_meta.names.update(names)
    op_meta.host_names.setdefault(host.name, set()).update(names)  # ADD THIS
    return op_meta
```

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra_cli/prints.py`

In `print_meta()`, use host-specific names:
```python
def print_meta(state: "State"):
    for op_hash in state.get_op_order():
        hosts_in_op = []
        host_specific_names = set()  # ADD
        for host in state.inventory.iter_activated_hosts():
            if op_hash in state.ops[host]:
                op_data = state.get_op_data_for_host(host, op_hash)
                if op_data.operation_meta._maybe_is_change:
                    # ...
                    hosts_in_op.append(host.name)
                    # ADD: Collect host-specific names
                    if host.name in state.op_meta[op_hash].host_names:
                        host_specific_names.update(state.op_meta[op_hash].host_names[host.name])

        # Use host-specific names if available
        display_names = host_specific_names or state.op_meta[op_hash].names
```

**Pros:**
- Preserves cross-host operation grouping
- Only changes display, not execution logic

**Cons:**
- More complex implementation
- Requires changes to multiple files

### Option 3: Include Operation Arguments in Hash (Most Robust)

The comment in GitHub from @SteadBytes suggests including operation arguments in the hash:

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py`

```python
def solve_operation_consistency(names, state, host):
    if pyinfra.is_cli:
        op_order = get_operation_order_from_stack(state)
    else:
        op_order = [len(host.op_hash_order)]

    if host.loop_position:
        op_order.extend(host.loop_position)

    # Include names in the hash to differentiate operations with different args
    hash_input = list(op_order)
    hash_input.extend(sorted(names))  # ADD: Include operation names
    op_hash = make_hash(hash_input)
```

**Pros:**
- Operations with different names get different hashes
- Most semantically correct solution
- Works across hosts naturally

**Cons:**
- Significant behavioral change
- May break existing workflows that rely on current hashing behavior
- Needs thorough testing

---

## Recommended Solution

**Option 3 (Include Operation Names in Hash)** is the most semantically correct solution because:

1. Operations with different arguments/names are fundamentally different operations
2. The current behavior of grouping them under the same hash is conceptually wrong
3. It fixes both the display issue AND potential execution ordering issues

However, **this is a significant change** that could affect existing behavior. A safer short-term fix is **Option 2** which only changes the display without affecting execution.

---

## Related Files

| File | Purpose |
|------|---------|
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py` | Operation decorator, hash generation, `solve_operation_consistency()` |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/util.py` | `make_hash()` function |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/state.py` | `StateOperationMeta`, `get_op_order()` |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/host.py` | `op_hash_order` list, `loop_position` |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra_cli/prints.py` | `print_meta()`, `pretty_op_name()` |

---

## Testing Recommendations

1. Create a test case with the exact reproduction scenario from the issue
2. Test with varying loop sizes across hosts
3. Test with identical operations across hosts (should still group correctly)
4. Test nested loops
5. Test `_run_once` operations
6. Test operations with `_serial` flag
7. Verify actual deployment still works correctly (not just display)
