# Celery Shelve Database Format

This document describes the data format and structure used by Celery's `--statedb` persistence mechanism.

## Overview

Celery's `Persistent` class (from `celery.worker.state`) stores worker state between restarts when `--statedb` is enabled. According to the official documentation in `celery/worker/state.py:192-199`:

> Stores worker state between restarts.
> This is the persistent data stored by the worker when :option:`celery worker --statedb` is enabled.
> Currently only stores revoked task id's.

## Storage Backend

**Default**: Python's `shelve` module (backed by dbm/gdbm)
- File-based key-value database
- Stores pickled Python objects


## Data Structure

The shelve database stores a dictionary with the following keys:

### 1. `__proto__` (Protocol Version)

**Type**: `int`
**Value**: `3` (current protocol version)
**Purpose**: Version identifier for backward compatibility

**History**:
- Protocol 1: Stored revoked tasks as a plain dict
- Protocol 2: Introduced `LimitedSet` data structure
- Protocol 3: Added compression with `zlib` (current)

### 2. `zrevoked` (Compressed Revoked Tasks)

**Type**: `bytes` (compressed pickle)
**Format**: `zlib.compress(pickle.dumps(LimitedSet))`
**Purpose**: Stores the set of revoked task IDs

**Data Structure**: `celery.utils.collections.LimitedSet`
- Bounded set with maximum capacity (default: 50,000 items)
- Automatic expiry of old items (default: 3 hours / 10,800 seconds)
- Contains task IDs as strings

**Task ID Format**:
- Production: UUIDs generated by `kombu.utils.uuid.uuid()`
- Example: `"3e8f9c8a-7b2d-4f1e-9a3c-5d6e7f8a9b0c"`

**Serialization Process**:

### 3. `clock` (Lamport Clock)

**Type**: `int`
**Purpose**: Logical clock for distributed coordination

**Behavior**:
- On save: Calls `clock.forward()` to increment and store current value
- On load: Calls `clock.adjust(stored_value)` to synchronize
- If no clock is configured, stores `0`

**Use Case**: Ensures causally ordered operations across multiple workers

### Worker State (In-Memory)

The worker maintains these data structures in memory (`celery.worker.state`):

  **Persisted**:
  - `revoked`: `LimitedSet` - Revoked task IDs (saved to statedb)
  
  **Not Persisted** (lost on restart):
  - `requests`: `dict` - Mapping of task_id → Request objects
  - `reserved_requests`: `WeakSet` - Currently reserved tasks
  - `active_requests`: `WeakSet` - Currently executing tasks
  - `successful_requests`: `LimitedSet` - Recently successful task IDs
  - `total_count`: `Counter` - Task counts by task name
  - `all_total_count`: `list` - Total task count
  - `revoked_stamps`: `dict` - Revoked task stamps

## Persistence Lifecycle

### 1. Worker Startup (Merge)

```python
def _merge_with(self, db):
    """Load persisted state and merge with current worker state."""
    self._merge_revoked(db)  # Load revoked tasks
    self._merge_clock(db)     # Synchronize clock
```

**Steps**:
1. Open shelve database file
2. Read `zrevoked` key (or fall back to `revoked`)
3. Decompress and unpickle the LimitedSet
4. Merge loaded revoked tasks into worker's `state.revoked`
5. Purge expired revoked tasks
6. Adjust Lamport clock if present

### 2. Worker Shutdown (Sync)

```python
def _sync_with(self, db):
    """Save current worker state to persistent storage."""
    self._revoked_tasks.purge()  # Remove expired items
    db.update({
        '__proto__': 3,
        'zrevoked': self.compress(self._dumps(self._revoked_tasks)),
        'clock': self.clock.forward() if self.clock else 0,
    })
```

**Steps**:
1. Purge expired revoked tasks from memory
2. Pickle and compress the revoked set
3. Store compressed blob with protocol version
4. Update Lamport clock
5. Sync and close shelve database

### 3. Atexit Handler

```python
atexit.register(worker._persistence.save)
```
Ensures state is saved on graceful shutdown (SIGTERM, normal exit).

## References

- Celery source: `celery/worker/state.py` (class `Persistent`)
- Celery tests: `t/unit/worker/test_state.py`
- Data structure: `celery/utils/collections.py` (class `LimitedSet`)
- Serialization: `kombu/serialization.py`

## Summary

Celery's shelve database is a simple key-value store with three keys:
1. `__proto__`: Version marker (always `3`)
2. `zrevoked`: Compressed pickle of revoked task IDs (LimitedSet)
3. `clock`: Lamport clock value for coordination

Only revoked task IDs are persisted between worker restarts. All other worker state (active tasks, counters, etc.) is ephemeral and lost on restart.
