# SDN & Network Tooling Research

Research into software-defined networking tools for Gaol's container and VM networking, covering single-host and multi-host scenarios.

## Current Architecture

- **Host OS**: Debian trixie (testing), kernel 6.12
- **Containers**: systemd-nspawn via `machinectl`, declarative `.nspawn` configs
- **VMs**: libvirt/QEMU (for full isolation workloads)
- **Bridge**: Linux bridge `br0` for containers/VMs needing LAN access
- **Private links**: veth pairs with static addressing for chain topologies (e.g. research chain)
- **Firewall**: nftables (`ip research_chain` table for chains; systemd-networkd's `io.systemd.nat` for masquerade)
- **Policy routing**: iproute2 tables + rules for multi-hop chains (research → tor → VPN)
- **DNS**: per-container resolv.conf, Tor DNSPort for research chain

### What works well

- Private veth pairs provide strong L2 isolation without VLAN complexity
- nftables separate tables avoid conflicts with Docker/libvirt/Tailscale
- systemd-networkd `.network` files handle static addressing and policy rules declaratively
- Policy routing with `iif` restricts forwarding rules to actual container traffic

### Pain points

- br0 is a flat L2 domain — all bridged containers can see each other's traffic
- No per-flow visibility or statistics on br0
- Adding new chain topologies requires manual veth/routing setup
- No VLAN tagging for L2 isolation between bridged containers

## Evaluated Tools

### Open vSwitch (OVS)

**What it is**: Production-grade virtual switch with OpenFlow support, flow tables, and extensive monitoring.

**Strengths**:
- Per-flow statistics (`ovs-ofctl dump-flows`) — see exactly what traffic each container generates
- VLAN tagging at the port level — isolate bridged containers without separate bridges
- OpenFlow rules for fine-grained traffic control (drop, rate-limit, mirror)
- Mature, well-documented, used in production by OpenStack, oVirt, Proxmox
- `ovs-vsctl` CLI is straightforward for single-host use

**Weaknesses**:
- Extra daemon (ovs-vswitchd + ovsdb-server) to manage
- OpenFlow learning curve for advanced rules
- Kernel datapath module (openvswitch.ko) needs to match kernel version
- Overkill if you only need a few containers on one host

**Fit for Gaol**:
- **Single-host**: Good replacement for br0. VLAN tagging isolates work/personal/untrusted containers at L2. Flow stats aid debugging.
- **Multi-host**: OVS supports GRE/VXLAN tunnels between hosts, but OVN (see below) is better for multi-host orchestration.

**Recommendation**: Adopt for br0 replacement in Phase 3. Keep policy routing for chain topologies (OVS handles L2, iproute2 handles L3 chains).

### OVN (Open Virtual Network)

**What it is**: Distributed virtual networking layer built on OVS. Provides logical switches, routers, ACLs, and DHCP across multiple hosts.

**Strengths**:
- Logical network abstraction — define networks that span hosts
- Distributed routing (no single chokepoint)
- Built-in ACLs (stateful firewall at the logical switch level)
- Native GENEVE tunneling between hosts (auto-configured)
- `ovn-nbctl` for declarative network definitions

**Weaknesses**:
- Significant complexity: needs ovn-central (northbound + southbound DBs) and ovn-controller per host
- Designed for multi-host — adds overhead on single-host deployments
- Documentation assumes OpenStack/Kubernetes context
- Debugging requires understanding both OVN logical and OVS physical layers

**Fit for Gaol**:
- **Single-host**: Overkill. OVS alone covers the use case.
- **Multi-host**: Strong candidate if Gaol expands to manage containers across multiple machines. Provides the "distributed firewall + logical network" abstraction that iproute2 can't do across hosts.

**Recommendation**: Defer until multi-host is a priority. When it is, OVN is the natural next step from OVS.

### CNI (Container Network Interface)

**What it is**: Plugin-based networking standard from CNCF. Plugins handle interface creation, IP allocation, and routing for container runtimes.

**Strengths**:
- Standard interface — write once, use with any CNI-compatible runtime
- Plugin ecosystem: bridge, macvlan, ipvlan, host-local IPAM, DHCP, etc.
- Chainable plugins (e.g. bridge + bandwidth limiting + firewall)
- Reference plugins are simple C/Go binaries

**Weaknesses**:
- Designed for Kubernetes — assumes pod lifecycle semantics (ADD/DEL on container start/stop)
- No runtime daemon — plugins are invoked per-event, no centralized state
- systemd-nspawn doesn't natively call CNI plugins (would need wrapper)
- Limited visibility — no built-in flow stats or monitoring

**Fit for Gaol**:
- **Single-host**: Possible but awkward. Would need a CNI wrapper around nspawn's network setup. Gains are marginal over networkd + nftables.
- **Multi-host**: Only useful if adopting Kubernetes or a CNI-native runtime (Podman, CRI-O).

**Recommendation**: Skip. systemd-networkd + nftables already handle single-host networking well. CNI adds an abstraction layer without clear benefit for nspawn.

### Cilium / eBPF

**What it is**: eBPF-based networking and security for containers. Replaces iptables/nftables with kernel-level programs for packet processing.

**Strengths**:
- Extremely high performance (datapath in kernel, no user-space switching)
- Fine-grained network policies (L3/L4/L7, DNS-aware, identity-based)
- Hubble observability — flow logs, service maps, DNS monitoring
- Transparent encryption (WireGuard or IPsec between nodes)
- Replaces kube-proxy with eBPF — no iptables chains to manage

**Weaknesses**:
- Requires kernel 5.10+ (met) but heavily optimized for 5.15+/6.x
- Designed for Kubernetes — agent assumes pod CIDRs, node identity, etc.
- Standalone mode exists but is experimental and poorly documented
- Heavy Go binary (~200MB) with etcd or Kubernetes API dependency
- eBPF program loading requires CAP_BPF + CAP_NET_ADMIN on host

**Fit for Gaol**:
- **Single-host**: The observability (Hubble) is compelling, but the Kubernetes dependency makes it impractical for nspawn containers.
- **Multi-host with Kubernetes**: Excellent choice if Gaol ever orchestrates containers via Kubernetes.

**Recommendation**: Skip for now. The eBPF datapath concepts are relevant, but Cilium's Kubernetes coupling makes it unsuitable. Consider standalone eBPF programs (via `bpftool` or libbpf) for specific use cases like transparent encryption.

### VPP (Vector Packet Processing)

**What it is**: High-performance user-space network stack from fd.io. Processes packets in vectors (batches) for cache efficiency.

**Strengths**:
- Multi-million PPS on commodity hardware
- Full L2-L4 stack (switching, routing, NAT, tunnels)
- Plugin architecture for custom packet processing
- DPDK integration for hardware offload

**Weaknesses**:
- Requires dedicated CPU cores (polling mode, no interrupts)
- Complex configuration (VPP CLI or API)
- Designed for network appliances, not container management
- Small community compared to OVS or eBPF
- Overkill for anything under 10Gbps

**Fit for Gaol**:
- **Single-host**: Entirely overkill. The container traffic is nowhere near VPP's target workload.
- **Multi-host**: Only relevant for dedicated network function virtualization (NFV) appliances.

**Recommendation**: Skip. Not relevant to container management use case.

### Netavark

**What it is**: Rust-based container networking tool from the Podman project. Replacement for CNI plugins in rootless/rootful Podman.

**Strengths**:
- Purpose-built for container networking (bridge, macvlan, DHCP)
- Aardvark-dns for container DNS resolution
- Rootless networking support
- Firewall backend abstraction (iptables or nftables)
- Simple JSON config per network

**Weaknesses**:
- Tightly coupled to Podman's network model (netns-based)
- No OpenFlow or flow visibility
- No VLAN or tunnel support
- Small scope — bridge creation, IP allocation, firewall rules

**Fit for Gaol**:
- **Single-host**: Could replace manual bridge/IP/firewall setup if Gaol adopted Podman's network model. But nspawn's VirtualEthernet + networkd already does this declaratively.
- **Multi-host**: No support.

**Recommendation**: Skip. Useful reference for Rust network tooling, but doesn't add capabilities beyond what networkd + nftables provide.

### nftables (current)

**What it is**: Linux kernel packet classification framework, successor to iptables. Uses named tables with typed chains.

**Strengths**:
- Atomic ruleset loading (`nft -f`)
- Named tables for isolation (our `ip research_chain` table)
- Sets and maps for efficient matching
- Conntrack integration for stateful rules
- No extra daemons — kernel-native
- Co-exists safely with Docker/libvirt/Tailscale (separate tables)

**Weaknesses**:
- No flow statistics (counters only, no per-flow tracking)
- No L2 switching features (VLAN tagging requires ebtables or OVS)
- Rule management is imperative (no declarative reconciliation)

**Fit for Gaol**:
- **Single-host**: Already in use. Separate tables per chain topology keep rules clean. Good enough for firewall + NAT.
- **Multi-host**: Works per-host but can't coordinate across hosts.

**Recommendation**: Keep as the firewall/NAT layer. Pair with OVS for L2 features (VLANs, flow stats).

### systemd-networkd (current)

**What it is**: Declarative network configuration manager built into systemd. `.network`, `.netdev`, and `.link` files.

**Strengths**:
- Declarative — config files describe desired state
- Handles addresses, routes, policy rules, bridges, VLANs, tunnels
- Auto-applies when interfaces appear (udev integration)
- No extra dependencies (part of systemd)
- `.network` file wildcards for dynamic interface names

**Weaknesses**:
- `IncomingInterface=` in `[RoutingPolicyRule]` doesn't support wildcards (can't use for dynamic veth names with `iif`)
- `IPMasquerade=` injects rules into shared `io.systemd.nat` table (must clean up)
- No firewall rules — only routing/addressing
- Limited bridge configuration compared to `brctl`/OVS

**Fit for Gaol**:
- **Single-host**: Excellent for static addressing, policy rules, and routes. Already declaratively managing veth IPs and table 200 routes.
- **Multi-host**: Only handles local interface config (no remote coordination).

**Recommendation**: Keep as the addressing/routing layer. The `.network` files are the right abstraction for static veth configuration.

## Architecture Recommendations

### Single-Host (Current Priority)

**Stack**: systemd-networkd + nftables + OVS (Phase 3)

```
Layer       Tool              Purpose
─────────   ────────────────  ──────────────────────────────
L2 switch   OVS               br0 replacement, VLAN isolation, flow stats
L3 routing  systemd-networkd  Static addressing, policy rules, table routes
Firewall    nftables           Per-chain tables, NAT, forwarding rules
Chains      iproute2 + script  Dynamic table 201 route, dynamic iif rules
```

**Migration path**:
1. (Done) nftables separate tables for chain firewall rules
2. (Done) networkd `.network` files for static veth config
3. (Phase 3) Replace br0 Linux bridge with OVS bridge
4. (Phase 3) VLAN tags per domain: work=VLAN10, personal=VLAN20, untrusted=VLAN30
5. (Phase 3) OVS flow logging for traffic visibility

### Multi-Host (Future)

**Stack**: OVN + WireGuard mesh + nftables

```
Layer       Tool              Purpose
─────────   ────────────────  ──────────────────────────────
Overlay     OVN + GENEVE      Logical switches/routers across hosts
Encryption  WireGuard mesh    Encrypted tunnels between hosts
L3 routing  OVN logical       Distributed routing, no single chokepoint
Firewall    OVN ACLs          Centralized policy, distributed enforcement
Local       nftables           Host-specific rules (kill-switches, etc.)
```

**Key decisions**:
- OVN for logical network abstraction (avoids manual tunnel management)
- WireGuard for encryption (OVN's GENEVE tunnels are unencrypted by default)
- Keep nftables for host-local rules that don't fit OVN's model

## Secure Network Overlays

### WireGuard Mesh

- Point-to-point encrypted tunnels between hosts
- Each host gets a WireGuard interface with peer entries for every other host
- Tools: `wg-quick`, or systemd-networkd `[NetDev]` Kind=wireguard
- Low overhead (~5% at 1Gbps), kernel-native since 5.6
- Works with or without OVS/OVN — can be the underlay for GENEVE tunnels

**Use case**: Encrypt all inter-host container traffic. Each host's WireGuard interface carries OVN GENEVE tunnels, so overlay traffic is encrypted transparently.

### OVN GENEVE Tunnels

- Automatic tunnel mesh between OVN chassis (hosts)
- Encapsulates logical network traffic with metadata (VNI, direction)
- Unencrypted by default — pair with WireGuard underlay for encryption
- Managed by ovn-controller (no manual tunnel config)

**Use case**: Logical network spanning multiple hosts. Containers on different hosts appear to be on the same L2 segment.

### VXLAN (Manual)

- Standard overlay encapsulation (RFC 7348)
- Supported by OVS, Linux bridge, and most network hardware
- Can be configured via `ip link add type vxlan` or OVS `ovs-vsctl add-port`
- No built-in encryption — must pair with IPsec or WireGuard

**Use case**: Simpler alternative to OVN when logical routing isn't needed. Good for extending a single L2 domain across hosts without the OVN control plane.

### Comparison

| Feature | WireGuard Mesh | OVN GENEVE | Manual VXLAN |
|---------|---------------|------------|--------------|
| Encryption | Native | No (add WireGuard) | No (add WireGuard) |
| Auto-config | No (static peers) | Yes (ovn-controller) | No (static config) |
| Logical routing | No | Yes | No |
| ACLs | No | Yes (centralized) | No |
| Complexity | Low | High | Low |
| Best for | Encrypted underlay | Full SDN overlay | Simple L2 extension |

## Summary

For Gaol's single-host deployment, the current stack (systemd-networkd + nftables + iproute2) is the right choice. OVS is the natural next step for br0, adding VLAN isolation and flow visibility without disrupting the existing chain topology pattern.

Multi-host expansion should adopt OVN for logical networking over a WireGuard-encrypted underlay. This provides centralized policy with distributed enforcement — the network equivalent of what Gaol already does for container security domains.
