# Investigation Report: GitHub Issue #1154

## Issue Summary

**Title:** Systemd service failure not detected correctly

**Reporter:** ReggieMarr

**Created:** 2024-07-25

**Labels:** bug

**pyinfra Version:** v3.0b4

### Problem Description

The user reports that when using `systemd.service()` to restart a service, pyinfra does not correctly detect that the service has failed. The user is attempting to check the operation result but is getting the error:

```
RuntimeError: Cannot evaluate operation result before execution
```

The user's code attempts to access `response.stdout_lines` before the operation has actually executed, which is the immediate cause of the error.

### User's Code

```python
response = systemd.service(
    name="Re-Start fsw service",
    service="fsw.service",
    running=True,
    restarted=True,
    user_mode=True,
    user_name=username
)
if (not response.is_complete):
    logger.info("Waiting for systemdservice to complete...")
    time.sleep(0.2)
logger.info(response.stdout_lines)
```

### Service State

The systemd service shows as failed:
```
Active: failed (Result: exit-code) since Thu 2024-07-25 19:06:08 EDT
Process: 33526 ExecStart=/home/rdx/.local/share/rdx/runtime/FlightComputer (code=exited, status=2)
fsw.service: Failed with result 'exit-code'.
```

---

## Root Cause Analysis

### Issue 1: Misunderstanding of pyinfra's Execution Model

The primary issue in the user's code is a misunderstanding of how pyinfra operations work. pyinfra uses a **two-phase execution model**:

1. **Prepare Phase**: Operations are called and queued. The `OperationMeta` object is returned immediately, but the operation has NOT executed yet. At this point, `is_complete()` returns `False`, and attempting to access `stdout_lines`, `stderr_lines`, or call `did_succeed()` will raise `RuntimeError`.

2. **Execute Phase**: Operations are executed either by the CLI runner or by explicitly calling `run_ops()` (API mode).

**Relevant code** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py`):
```python
def _raise_if_not_complete(self) -> None:
    if not self.is_complete():
        raise RuntimeError("Cannot evaluate operation result before execution")

@property
def stdout_lines(self) -> list[str]:
    self._raise_if_not_complete()
    assert self._combined_output is not None
    return self._combined_output.stdout_lines
```

The user's `time.sleep(0.2)` will not help because pyinfra operations don't execute asynchronously in the background - they execute synchronously when `run_ops()` is called or when using the CLI.

### Issue 2: Systemd Service Failure Detection Mechanism

The second part of the issue relates to whether pyinfra correctly detects that a systemd service has failed. Let's examine the detection mechanism:

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/systemd.py`

The `SystemdStatus` fact uses:
```python
state_key = "SubState"
state_values = ["running", "waiting", "exited", "listening", "mounted"]
```

The fact queries: `systemctl show --all --property Id --property SubState <service>`

A service is considered "running" (status = True) if its `SubState` is in `["running", "waiting", "exited", "listening", "mounted"]`.

**Important:** When a service fails, its `SubState` becomes `"failed"` or `"dead"`, which is NOT in the `state_values` list. This means:

- A failed service will have `SystemdStatus` return `False` (not running)
- The `systemd.service()` operation checks this status
- If `running=True` is requested and the service shows as not running, a `start` command is issued

**The detection works correctly at the FACT level** - a failed service is properly detected as "not running".

### Issue 3: The systemctl restart Command Exit Code

**This is where the actual bug may lie.** When a service fails to start/restart:

1. `systemctl restart <service>` is issued
2. If the service fails to start, `systemctl restart` exits with a non-zero exit code
3. pyinfra detects this non-zero exit code as an operation failure
4. The `OperationMeta.did_succeed()` returns `False`, and `did_error()` returns `True`

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operations.py`, lines 116-142:

```python
elif isinstance(command, StringCommand):
    output_lines = CommandOutput([])
    try:
        status, output_lines = command.execute(
            state,
            host,
            connector_arguments,
        )
    except (timeout_error, socket_error, SSHException) as e:
        log_host_command_error(host, e, timeout=timeout)
    all_output_lines.extend(output_lines)
    # If we failed and have not already printed the stderr, print it
    if status is False and not state.print_output:
        print_host_combined_output(host, output_lines)

# Break the loop to trigger a failure
if status is False:
    did_error = True
    if continue_on_error is True:
        continue
    break
```

**The operation failure IS being detected.** The issue is that:

1. The user is trying to check results BEFORE execution
2. The service failure would be detected by `did_error()` after execution

### Issue 4: restart vs start Behavior in handle_service_control

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/service.py`

```python
# Only restart if the service is already running
if restarted and is_running:
    yield formatter.format(shlex.quote(name), "restart")
```

**Important finding:** The restart command is ONLY issued if the service is currently running (`is_running=True`).

In the user's case:
- The service has already failed (`SubState=failed`)
- This means `is_running=False`
- With `running=True, restarted=True`:
  - Since `is_running=False`, only a `start` command is issued (line 31)
  - The `restart` is NOT issued because `restarted and is_running` is `False`

This is actually correct behavior - you can't "restart" a service that isn't running; you can only "start" it.

---

## Proposed Fix

### The core issue is user education, not a code bug.

The user needs to understand pyinfra's execution model. However, there are some improvements that could be made:

### Fix 1: Improve Documentation for Checking Operation Results (Recommended)

The documentation should clearly explain how to properly check operation results:

**For CLI mode:**
```python
# In deploy.py
response = systemd.service(
    name="Re-Start fsw service",
    service="fsw.service",
    running=True,
    restarted=True,
    user_mode=True,
    user_name=username
)

# The operation hasn't executed yet - don't access results here!
# Use callbacks or check results in a handler
```

**For API mode:**
```python
from pyinfra.api import State, Inventory
from pyinfra.api.operations import run_ops

# Setup state and inventory
state = State(inventory)

# Prepare operations
response = systemd.service(...)

# Execute operations
run_ops(state)

# NOW you can check results
if response.did_error():
    logger.error("Service failed to start")
    logger.error(response.stderr)
```

### Fix 2: Add a More Helpful Error Message (Code Change)

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py`

Change the error message to be more helpful:

```python
def _raise_if_not_complete(self) -> None:
    if not self.is_complete():
        raise RuntimeError(
            "Cannot evaluate operation result before execution. "
            "Operations are queued during the prepare phase and executed later. "
            "In API mode, call run_ops(state) first. "
            "In CLI mode, check results in callbacks or after pyinfra completes."
        )
```

### Fix 3: Consider Adding ActiveState Check (Enhancement)

Currently, `SystemdStatus` only checks `SubState`. For better failure detection, we could also check `ActiveState`:

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/systemd.py`

A service's `ActiveState` can be:
- `active` - Service is running
- `inactive` - Service is not running
- `failed` - Service has failed
- `activating` - Service is starting up
- `deactivating` - Service is shutting down

Adding `ActiveState` to the fact could provide more nuanced status information:

```python
class SystemdStatus(FactBase[Dict[str, bool]]):
    # ... existing code ...

    @override
    def command(self, ...):
        return StringCommand(
            fact_cmd,
            "show",
            "--all",
            "--property", "Id",
            "--property", "SubState",
            "--property", "ActiveState",  # Add this
            *service_strs,
        )
```

However, this would require changes to the `process()` method and potentially the return type.

---

## Related Files

| File | Purpose | Modification Needed |
|------|---------|---------------------|
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operation.py` | Operation execution model, OperationMeta class | Error message improvement |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/systemd.py` | SystemdStatus fact | Optional: Add ActiveState check |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/systemd.py` | systemd.service operation | No changes needed |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/service.py` | Service control handler | No changes needed |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/api/operations.py` | Operation execution | No changes needed |

---

## Conclusion

**The "bug" is primarily a documentation/UX issue, not a code defect.**

1. **Systemd service failure IS detected** - when a `systemctl start/restart` command fails, the exit code is non-zero, and pyinfra correctly marks the operation as failed.

2. **The user's immediate error** (RuntimeError) is caused by accessing operation results before execution completes. This is expected behavior but the error message could be more helpful.

3. **The `restarted=True` behavior** correctly does NOT issue a restart if the service is already stopped/failed - instead it just starts it.

### Recommendations:

1. **Primary:** Improve the error message in `_raise_if_not_complete()` to explain pyinfra's execution model
2. **Secondary:** Add documentation examples showing how to properly handle operation results in both CLI and API modes
3. **Optional:** Consider adding `ActiveState` to `SystemdStatus` for richer status information (would be a separate enhancement)

### Example of Correct Usage for the Reporter:

```python
# deploy.py
from pyinfra.operations import systemd

# This just queues the operation
response = systemd.service(
    name="Re-Start fsw service",
    service="fsw.service",
    running=True,
    restarted=True,
    user_mode=True,
    user_name=username,
    _ignore_errors=False,  # Default - will fail deploy on error
)

# Results are available AFTER pyinfra executes
# In CLI mode: check response.will_change before execution
# After execution (or in API mode after run_ops): use response.did_error()
```

For API mode:
```python
from pyinfra.api import State, Inventory
from pyinfra.api.operations import run_ops
from pyinfra.operations import systemd

state = State(inventory)

response = systemd.service(
    service="fsw.service",
    running=True,
    restarted=True,
    user_mode=True,
    user_name=username
)

run_ops(state)  # Execute all queued operations

# NOW check results
if response.did_error():
    print("Service failed!")
    print("stderr:", response.stderr)
```
