# Phase 2: Core VM Lifecycle — Research

**Researched:** 2026-03-22
**Phase:** 02-core-vm-lifecycle
**Requirements:** CLUS-01, CLUS-02, CLUS-03, CLUS-04, CLUS-05, CLUS-06, CLUS-07

---

## 1. libvirt-python API Surface

### Connection & Domain Lifecycle

```python
import libvirt

conn = libvirt.open("qemu:///system")        # Read-write connection
dom = conn.defineXML(xml_string)              # Define persistent domain (does not start)
dom = conn.lookupByName("domain-name")        # Lookup existing domain
dom.create()                                  # Start a defined domain
dom.shutdown()                                # Graceful ACPI shutdown
dom.destroy()                                 # Force-kill (like pulling power)
dom.undefine()                                # Remove persistent definition
dom.info()                                    # [state, maxMem, memory, nrVirtCpu, cpuTime]
dom.isActive()                                # Returns 1 if running, 0 if not
dom.state()                                   # Returns [state, reason] tuple
```

**Domain states** (from `libvirt.VIR_DOMAIN_*`):
- `VIR_DOMAIN_NOSTATE` = 0
- `VIR_DOMAIN_RUNNING` = 1
- `VIR_DOMAIN_BLOCKED` = 2
- `VIR_DOMAIN_PAUSED` = 3
- `VIR_DOMAIN_SHUTDOWN` = 4
- `VIR_DOMAIN_SHUTOFF` = 5
- `VIR_DOMAIN_CRASHED` = 6

Map to our `NodeStatus.domain_state`: running=1, shut off=5, paused=3, crashed=6, unknown=else.

### Network Lifecycle

```python
net = conn.networkDefineXML(xml_string)       # Define persistent network
net = conn.networkLookupByName("net-name")     # Lookup existing
net.create()                                   # Activate network
net.destroy()                                  # Deactivate network
net.undefine()                                 # Remove persistent definition
net.isActive()                                 # Returns 1 if active
net.bridgeName()                               # Returns bridge device name
```

### Thread Safety Constraint

Per STATE.md decision: all libvirt calls through a single `ThreadPoolExecutor(max_workers=1)`. The libvirt C library is not thread-safe with concurrent connection use. Pattern:

```python
import asyncio
from concurrent.futures import ThreadPoolExecutor

_executor = ThreadPoolExecutor(max_workers=1)

async def _run_libvirt(func, *args):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_executor, func, *args)
```

## 2. Domain XML Structure (KVM/QEMU)

Minimal working domain XML for a KVM guest with qcow2 overlay disk and bridge network:

```xml
<domain type='kvm'>
  <name>{cluster}_{node}</name>
  <memory unit='MiB'>{memory_mb}</memory>
  <vcpu>{vcpus}</vcpu>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <!-- Primary overlay disk -->
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='{overlay_path}'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <!-- Cloud-init NoCloud ISO -->
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='{cidata_iso_path}'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
    </disk>
    <!-- Extra disks (from topology) -->
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='{extra_disk_path}'/>
      <target dev='vdb' bus='{bus}'/>
    </disk>
    <!-- Network interface on cluster bridge -->
    <interface type='bridge'>
      <source bridge='{bridge_name}'/>
      <mac address='{mac_address}'/>
      <model type='virtio'/>
    </interface>
    <!-- Serial console for debugging -->
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>
```

**Key decisions from CONTEXT.md:**
- D-06: Build XML programmatically with `xml.etree.ElementTree` (no Jinja2)
- D-10: Overlay paths: `{overlay_dir}/{cluster_name}/{node_name}.qcow2`
- MAC addresses: generate deterministic MACs from cluster+node name (e.g., `52:54:00:xx:xx:xx` range, derive from hash)

## 3. Network XML Structure

For an isolated cluster network with static IPs (no DHCP per D-03 — cloud-init handles IPs):

```xml
<network>
  <name>{network_name}</name>
  <bridge name='{bridge_name}' stp='on' delay='0'/>
  <forward mode='nat'/>
  <ip address='{gateway_ip}' netmask='{netmask}'>
  </ip>
</network>
```

- `forward mode='nat'`: Allows outbound NAT (nodes can reach internet for package installs)
- No `<dhcp>` block: cloud-init configures static IPs (per D-03)
- Gateway IP: derived from subnet (e.g., `192.168.100.1` for `192.168.100.0/24`)

## 4. Cloud-init NoCloud ISO (D-03)

Cloud-init NoCloud datasource reads from a mounted ISO containing `meta-data` and `network-config` files.

### ISO Generation

```bash
genisoimage -output {cidata_path} -volid cidata -joliet -rock \
  meta-data network-config
```

Or using `mkisofs` (equivalent). Both are commonly available. Use `subprocess.run()`.

### meta-data (per node)

```yaml
instance-id: {cluster_name}-{node_name}
local-hostname: {node_name}
```

### network-config (per node, v2 format)

```yaml
version: 2
ethernets:
  enp1s0:
    addresses:
      - {ip}/24
    routes:
      - to: default
        via: {gateway_ip}
    nameservers:
      addresses:
        - 8.8.8.8
```

Note: The interface name depends on the guest OS. Common: `ens3`, `enp1s0`, `eth0`. Using `match` by MAC is more reliable:

```yaml
version: 2
ethernets:
  id0:
    match:
      macaddress: "{mac_address}"
    addresses:
      - {ip}/{prefix}
    routes:
      - to: default
        via: {gateway_ip}
```

### ISO location

Store at `{overlay_dir}/{cluster_name}/{node_name}-cidata.iso` alongside overlay disks.

## 5. Overlay Disk Creation (D-07)

```bash
qemu-img create -b {base_image} -F qcow2 -f qcow2 {overlay_path}
```

- `-b`: backing file (the base image)
- `-F qcow2`: format of backing file
- `-f qcow2`: format of new overlay
- Overlay path: `{overlay_dir}/{cluster_name}/{node_name}.qcow2` (per D-10)

**Safety assertion:** `assert base_image != overlay_path` (per D-10)

For extra disks from topology: create fresh qcow2 (no backing file):
```bash
qemu-img create -f qcow2 {extra_disk_path} {size_gb}G
```

## 6. Idempotency (CLUS-07)

`cluster_define` must be safe to call on already-defined clusters:

1. Check if domain already exists: `conn.lookupByName(domain_name)` — if found and defined, skip
2. Check if network already exists: `conn.networkLookupByName(net_name)` — if found, skip
3. Check if overlay disk already exists: `os.path.exists(overlay_path)` — if found, skip creation
4. Return success with current status

## 7. SSH Wait Loop Pattern

For `cluster_start` with `wait_for_ssh=True`:

```python
import asyncio
import asyncssh

async def wait_for_ssh(ip, port=22, user="root", key_path="...", timeout=120):
    deadline = asyncio.get_event_loop().time() + timeout
    while asyncio.get_event_loop().time() < deadline:
        try:
            async with asyncssh.connect(ip, port=port, username=user,
                                         client_keys=[key_path],
                                         known_hosts=None) as conn:
                result = await conn.run("echo ok", check=True)
                return True
        except (OSError, asyncssh.Error):
            await asyncio.sleep(2)
    return False
```

Per D-04: strict all-or-nothing — if any node fails SSH within timeout, entire call returns `success=False`.
Per D-11: asyncssh over paramiko.

## 8. Module Structure

```
vmcluster_mcp/cluster/
  __init__.py          # Tool registrations (cluster_define, start, stop, destroy, status, handle)
  libvirt_client.py    # ThreadPoolExecutor wrapper, connection management, domain/network ops
  domain_builder.py    # xml.etree.ElementTree domain XML generation
  network_builder.py   # xml.etree.ElementTree network XML generation  
  cloud_init.py        # NoCloud ISO generation (meta-data, network-config, genisoimage call)
```

## 9. Key Risks & Mitigations

| Risk | Mitigation |
|------|------------|
| libvirt segfault from concurrent access | Single-worker ThreadPoolExecutor (D-08) |
| Partial define leaves orphan resources | Error result reports what was/wasn't created; agent calls destroy to clean up (D-05) |
| Cloud-init interface naming varies by guest | Match by MAC address in network-config v2 |
| genisoimage not available on all systems | Check for genisoimage, fall back to mkisofs |
| Base image path collision with overlay | Assert base_image != overlay_path (D-10) |

## 10. Standard Stack / Don't Hand-Roll

| Need | Use | Don't |
|------|-----|-------|
| Domain XML generation | `xml.etree.ElementTree` (D-06) | Jinja2 templates, string concatenation |
| Overlay disk creation | `qemu-img` subprocess (D-07) | libvirt storage pool API |
| Cloud-init ISO | `genisoimage`/`mkisofs` subprocess | Manual ISO assembly |
| SSH wait | `asyncssh` (D-11) | paramiko, fabric |
| Thread safety | `ThreadPoolExecutor(max_workers=1)` (D-08) | Raw threading, async libvirt |

---

*Research completed: 2026-03-22*
