# FHS and 12-Factor Production Configuration Best Practices

*Cross-referenced and validated research from multiple sources with bias checking*
*Focus: Docker, Podman, and container runtimes*

## Executive Summary

Production container configuration requires **pragmatic hybridity** between two standards:
- **FHS 3.0** (Filesystem Hierarchy Standard) - Traditional Unix directory conventions
- **12-Factor App** - Cloud-native configuration methodology

Neither standard alone suffices. This guide documents validated patterns based on how major production systems actually operate.

---

## Part 1: FHS 3.0 for Containers

### Core Directory Purposes

| Directory | Purpose | Container Behavior |
|-----------|---------|-------------------|
| `/usr` | Static binaries, libraries | Read-only in image |
| `/etc` | Host-specific configuration | Mount configs read-only |
| `/var` | Variable data (logs, state, runtime) | Requires writable mounts |
| `/opt` | Third-party software packages | Read-only in image |
| `/tmp`, `/run` | Ephemeral data | tmpfs mounts |

### Directory Selection Matrix

| Package Location | Config | Variable Data | Use Case |
|---|---|---|---|
| `/opt/<pkg>` | `/etc/opt/<pkg>` | `/var/opt/<pkg>` | Third-party, add-on, commercial |
| `/usr/local/<pkg>` | `/etc` or `/usr/local/etc` | `/var/lib/<pkg>` | Locally compiled/installed |
| Distribution packages | `/etc/<pkg>` | `/var/lib/<pkg>` | OS package manager |

### Key FHS Relationships

1. **If config is in `/etc/<pkg>/`** → variable data goes in `/var/lib/<pkg>/`
2. **If app is in `/opt/<pkg>/`** → config in `/etc/opt/<pkg>/` AND variable data in `/var/opt/<pkg>/`

### Industry Reality Check

**How major official images actually deviate from FHS:**

| Image | Deviation | Rationale |
|-------|-----------|-----------|
| Redis | `/data` (not `/var/lib/redis`) | Simpler volume mounting |
| PostgreSQL | `/var/lib/postgresql/data` | FHS-compliant |
| Nginx | `/docker-entrypoint.d/` | Container init pattern |
| All | Logs → stdout/stderr | Container-native logging |

**Legitimate reasons for non-compliance:**
- Top-level data dirs (`/data`, `/app`) simplify volume mounts
- Log redirection aligns with container aggregation patterns
- Custom entrypoint dirs serve container initialization needs

---

## Part 2: 12-Factor Configuration

### Factor III: Config

**Core Principle:** Store config in environment variables, strict separation from code.

**What counts as config:**
- Resource handles (database URLs, cache endpoints)
- External service credentials
- Per-deploy values (hostnames, feature flags)

**What does NOT count as config:**
- Internal routing (application module connections)
- Structural code configuration

### When Environment Variables Are Insufficient

The 12-Factor "env vars only" approach breaks down for:

| Scenario | Problem | Solution |
|----------|---------|----------|
| Complex/hierarchical config | YAML/JSON can't flatten to env vars | File-based config |
| Large configs | Size limits on env vars | Volume-mounted files |
| Dynamic reloading | Env vars require restart | File-based with reload signals |
| Secrets | Env vars visible in `/proc`, crash dumps | File mounts with 0400 perms |
| TLS certificates | Binary data, multiple files | Volume-mounted secrets |

### Security Considerations 12-Factor Doesn't Address

1. **Environment variable visibility** - Exposed in:
   - `/proc/<pid>/environ`
   - Process listings (`ps e`)
   - Crash dumps and core files

2. **No file permissions** - Can't set 0400 on env vars

3. **No audit trails** - Harder to track config changes

4. **No atomic updates** - Volume mounts support atomic replacement

### Factor IV: Backing Services

**Core Principle:** Treat all external services identically via resource handles.

```
# Treat local and managed services identically
DATABASE_URL=postgres://localhost/app          # Development
DATABASE_URL=postgres://prod-db-host/app       # Production
CACHE_URL=redis://localhost:6379               # Development
CACHE_URL=redis://prod-cache-host/cluster      # Production
```

### Factor XI: Logs

**Core Principle:** Write to stdout/stderr, never to files.

| Context | Pattern |
|---------|---------|
| Development | stdout → terminal |
| Docker/Podman | stdout → container log driver |
| Systemd | stdout → journald |
| Traditional VM | stdout → syslog → /var/log |

---

## Part 3: Reconciling FHS with 12-Factor

### Conflict Resolution Matrix

| Aspect | FHS Says | 12-Factor Says | Production Reality |
|--------|----------|----------------|-------------------|
| Config | Files in `/etc/` | Env vars only | **Hybrid**: Defaults in `/etc/`, overrides via env |
| Logs | Files in `/var/log/` | stdout/stderr only | **12-Factor wins**: stdout in containers |
| State | `/var/lib/<app>/` | External services | **12-Factor wins**: All state in backing services |
| Secrets | Files in `/etc/` with 0600 | Env vars | **Neither**: Mounted secrets in `/run/secrets/` |

### Practical Reconciliation Patterns

**For Containers (Docker/Podman):**

```
# Image (read-only):
/usr/local/bin/     # Application binaries
/opt/<app>/         # Third-party packages (immutable)
/etc/<app>/         # Static defaults only (mounted read-only)
/app/               # Application code (immutable)

# Volumes (writable):
/var/lib/<app>/     # Persistent data → named volume
/var/opt/<app>/     # Optional package data → named volume
/tmp/               # Ephemeral → tmpfs
/run/               # Runtime state → tmpfs
/run/secrets/       # Secrets → mounted secrets

# NOT used:
/var/log/           # Logs go to stdout, not files
```

**Configuration Precedence (Recommended):**

```
Priority (highest to lowest):
1. Environment variables (runtime)
2. Bind-mounted config files (deploy-time)
3. /etc/<app>/defaults.yaml (image-time)
```

---

## Part 4: Security Best Practices

### Immutable Container Pattern

**Docker:**
```bash
docker run --read-only --tmpfs /tmp --tmpfs /run myimage
```

**Podman:**
```bash
podman run --read-only --tmpfs /tmp --tmpfs /run myimage
```

**Quadlet (systemd):**
```ini
[Container]
ReadOnly=true
Tmpfs=/tmp
Tmpfs=/run
```

### Volume Strategy for Immutable Containers

| Directory | Mount Type | Purpose |
|-----------|-----------|---------|
| `/tmp` | tmpfs | Temporary files |
| `/var/cache` | tmpfs or volume | Application cache |
| `/run` | tmpfs | Runtime state, PID files |
| `/var/lib/<app>` | Named volume | Persistent data |
| `/run/secrets` | Secret mount | Credentials |

### Secret Management Hierarchy

| Environment | Approach |
|-------------|----------|
| Development | `.env` files (gitignored) |
| Docker | `--secret` flag, `/run/secrets/` |
| Podman | `--secret` flag, `/run/secrets/` |
| Docker Compose | `secrets:` top-level element |
| Production | External managers (Vault, etc.) |

**Docker Secrets Pattern:**
```yaml
# docker-compose.yml
services:
  app:
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt
```

**"Secret Zero" Pattern:**
1. Bootstrap secret as short-lived env var
2. Authenticate to secret manager
3. Retrieve actual secrets
4. Discard bootstrap credential

---

## Part 5: Runtime-Specific Considerations

### Rootless Containers

| Consideration | Impact |
|---------------|--------|
| Privileged ports | Cannot bind <1024 without mapping |
| File ownership | User namespaces affect visibility |
| Socket access | Requires special handling |
| SELinux/AppArmor | Stricter policies for non-root |

**Recommendation:** Stick to FHS paths for easier security policy management.

### Platform Differences

| Platform | Config Approach | Secret Approach |
|----------|-----------------|-----------------|
| Docker | Env vars + bind mounts | `--secret` flag |
| Podman | Same as Docker | Same (rootless-first) |
| Docker Compose | Env files + secrets | `secrets:` top-level |
| Quadlets | Environment= + volume mounts | Secret= directive |

---

## Part 6: Anti-Patterns

### Critical Failures to Avoid

1. **Writing to image filesystem**
   - Causes copy-on-write overhead
   - Data lost on restart
   - Violates immutability
   - **Fix:** Use mounted volumes

2. **Mutating `/etc/` at runtime**
   - Makes containers non-reproducible
   - Blocks read-only filesystem
   - **Fix:** Use bind mounts, env var overrides

3. **Storing state locally**
   - Data lost on container removal
   - Breaks horizontal scaling
   - **Fix:** Use backing services (DB, cache)

4. **Env vars for secrets**
   - Visible in `/proc`, crash dumps
   - No file permissions
   - **Fix:** Volume-mounted secrets

5. **Logging to files**
   - Breaks container log collection
   - Requires volume management
   - **Fix:** stdout/stderr only

6. **Non-standard paths without reason**
   - Complicates SELinux/AppArmor policies
   - Confuses operators
   - **Fix:** Follow FHS unless specific benefit

---

## Part 7: Decision Framework

### When to Follow FHS

✅ **Do follow FHS when:**
- Building for enterprise/regulated environments
- Using SELinux/AppArmor with default policies
- Operators expect standard paths
- Running alongside traditional services

### When to Deviate

✅ **Acceptable to deviate when:**
- Top-level `/data` simplifies volume mounts
- Container-specific patterns (`/docker-entrypoint.d/`)
- Cloud-native logging (stdout over `/var/log/`)

### When to Follow 12-Factor

✅ **Do follow 12-Factor when:**
- Building microservices
- Using container orchestration
- Need horizontal scalability
- Want environment parity

### When to Deviate

✅ **Acceptable to deviate when:**
- Config too complex for env vars
- Secrets require file permissions
- Hot-reload needed without restart
- TLS certificates involved

---

## Part 8: ots-containers Specific Application

Based on the research, for ots-containers using `/etc/onetimesecret/` for configuration:

**Correct paths:**
- Config: `/etc/onetimesecret/` (read-only bind mount)
- Persistent data: `/var/lib/onetimesecret/` (named volume)
- Secrets: `/run/secrets/` (Docker/Podman secrets)
- Logs: stdout/stderr (not /var/log/)
- Quadlets: `/etc/containers/systemd/`

**Configuration precedence:**
1. Environment variables (highest priority)
2. `/etc/onetimesecret/config.yaml` (defaults)

---

## Sources

### FHS Documentation
- [Linux Foundation FHS 3.0](https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html)
- [FHS 3.0 Text Version](https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.txt)

### 12-Factor Methodology
- [12factor.net - Config](https://12factor.net/config)
- [12factor.net - Backing Services](https://12factor.net/backing-services)
- [12factor.net - Logs](https://12factor.net/logs)

### Container Documentation
- [Docker Secrets](https://docs.docker.com/compose/how-tos/use-secrets/)
- [Docker Rootless Mode](https://docs.docker.com/engine/security/rootless/)
- [Podman Documentation](https://docs.podman.io/)
- [Snyk Docker Best Practices](https://snyk.io/blog/10-docker-image-security-best-practices/)

### Critical Analysis Sources
- Redis, Nginx, PostgreSQL official Dockerfiles (GitHub docker-library)
- [12-Factor Limitations Discussion](https://medium.com/blablacar/the-twelve-factor-app-or-why-i-mostly-dont-use-environment-variables-for-configuration)
- OWASP Secrets Management recommendations

---

*Document created: 2025-12-15*
*Methodology: Multi-agent research with cross-referencing and bias throttling*
