# Server Freeze Analysis - Dec 3, 2025 06:03

## Summary
Server froze at **06:03:45** while attempting to render plot for `index=-64` (dataset position 45 out of 110 samples). The server stopped responding and timed out after 120 seconds.

## Timeline

### Successful Operations (Before Freeze)
- **06:03:02**: Processed 110-sample dataset (handle: `b417a050`)
- **06:03:02-06:03:35**: Background calculation workers successfully processed indices 0-108
- **06:03:45.605**: Server received request for `/draw_a_plot` with `index=-64`
- **06:03:45.606**: Server logged "plot data found in result cache"
- **06:03:45.606**: **SERVER FROZE HERE** - No further logs

### Client Timeout Events
- **06:05:45.614**: Client timeout (120s elapsed) - request_id=22, index=-64
- **06:05:45.725**: Client retried with different index=-33
- **06:07:45.762**: Client timeout again (120s elapsed) - request_id=23, index=-33

## Root Cause Analysis

### Where It Froze
Looking at `core.py:340-344`:
```python
# 4. Render
logger.debug(f"Rendering plot for {handle[:8]} idx {index}")
loop = asyncio.get_running_loop()
png_bytes = await loop.run_in_executor(None, logic.render_plot, plot_data)
logger.info(f"Action: generate_plot_from_handle completed - png_size={len(png_bytes)} bytes")
```

The server logged "plot data found in result cache" but never logged "Rendering plot for...". This means **the freeze occurred between lines 299-341 in `core.py`**.

More specifically, looking at the flow:
```python
if is_calculated:
    logger.info(f"Action: generate_plot_from_handle - plot data found in result cache")  # ✓ This logged
    plot_data = row_data
else:
    # ... wait logic ...

# 4. Render
logger.debug(f"Rendering plot for {handle[:8]} idx {index}")  # ✗ This never logged
```

**The freeze happened AFTER line 300 but BEFORE line 341.**

### Suspected Issue: Plotly/Kaleido Deadlock

The `logic.render_plot()` function at line 588 calls:
```python
image_bytes = fig.to_image(format="png")
```

This is known to cause issues because:

1. **Plotly's `to_image()` internally spawns Kaleido subprocess** for rendering
2. **In threaded/async environments**, Kaleido can deadlock if:
   - Multiple plots are being rendered simultaneously
   - The event loop is blocked
   - There's a race condition with background workers

3. **The CalcWorkers were still running in background** at the time:
   - From logs: Workers were still calculating indices (last logged: index 108 at 06:03:35)
   - Request came at 06:03:45, only 10 seconds later
   - Background workers may have been rendering plots themselves

## Evidence Supporting Kaleido Deadlock

### 1. Timing Pattern
- Previous plots rendered successfully in ~0.1-0.2 seconds
- Plot at index=-64 never completed (>120s timeout)
- This suggests a **hang/deadlock** rather than slow computation

### 2. Background Workers Running
From server log lines 344-482:
```
2025-12-03 06:03:19.748 - CalcWorker 1: Completed calculation for index 1
...
2025-12-03 06:03:34.937 - CalcWorker 3: Completed calculation for index 108
```

Workers were actively processing, and **CalcWorkers call `render_plot()` internally** to pre-compute plot data.

### 3. No Error Message
- Server didn't log any exception
- Process didn't crash
- Just stopped responding → **deadlock signature**

## Technical Details

### The Problem: Concurrent Plotly Usage
Looking at the calculation worker code pattern (inferred from logs):
- CalcWorkers process indices in background
- Each worker likely calls plotting functions
- Main request handler also calls `render_plot()`
- **All competing for same Kaleido subprocess pool**

### Kaleido Architecture Issue
Kaleido (used by Plotly for image export):
- Maintains a pool of browser processes
- Not fully thread-safe in all configurations
- Can deadlock when multiple threads/async tasks call `to_image()` simultaneously
- Known issue in production async environments

## Verification from Logs

### Server Log Pattern (Last 15 lines)
```
Line 613: Request: /draw_a_plot - handle=b417a050..., index=-17
Line 625: Response: /draw_a_plot - ... png_size=70637 bytes
Line 626: Request: /draw_a_plot - handle=b417a050..., index=-64
Line 627: Action: generate_plot_from_handle - handle=b417a050..., index=-64
Line 628: Action: generate_plot_from_handle - plot data found in result cache
Line 629: [END OF LOG - FREEZE]
```

### Client Log Pattern
```
Line 744: Slider event: request_id=22, index=45, server_index=-64
Line 751: httpcore.http11 - DEBUG - receive_response_headers.started
[120 second gap]
Line 760: httpcore.http11 - DEBUG - receive_response_headers.failed exception=ReadTimeout
```

## Potential Race Condition

The freeze likely occurred due to:

1. **Main thread** (request handler) tries to render plot for index=-64
2. **Background CalcWorker** is simultaneously rendering plots
3. Both call `fig.to_image()` → Kaleido subprocess
4. **Deadlock** occurs in Kaleido's internal synchronization
5. Neither thread makes progress
6. Server appears frozen

## Recommendations

### Immediate Fix Options

#### Option 1: Serialize Plotly Rendering (Recommended)
Add a global lock around `fig.to_image()`:

```python
import threading
_plotly_lock = threading.Lock()

def render_plot(plot_data):
    # ... existing code to build figure ...
    
    logger.debug("Converting plot to PNG image")
    with _plotly_lock:  # ← ADD THIS
        image_bytes = fig.to_image(format="png")
    logger.info(f"Plot rendered successfully: {len(image_bytes)} bytes")
    return image_bytes
```

#### Option 2: Use Different Renderer
Replace Kaleido with:
- `plotly.io.write_image()` with `engine="orca"` (if available)
- Or use matplotlib as fallback renderer
- Or render to HTML/SVG and convert separately

#### Option 3: Separate Rendering Process
- Create dedicated rendering worker that handles ALL plot rendering
- Queue render requests instead of direct calls
- Ensures only one Kaleido instance active at a time

### Monitoring Improvements

Add timeout wrapper around render:
```python
from concurrent.futures import TimeoutError as FutureTimeoutError
import concurrent.futures

def render_plot_with_timeout(plot_data, timeout=30):
    with concurrent.futures.ThreadPoolExecutor() as executor:
        future = executor.submit(render_plot_unsafe, plot_data)
        try:
            return future.result(timeout=timeout)
        except FutureTimeoutError:
            logger.error(f"Plot rendering timed out after {timeout}s")
            raise RuntimeError("Plot rendering timeout")
```

### Long-term Solutions

1. **Pre-render all plots during calculation phase**
   - Store PNG bytes in cache instead of plot data
   - Eliminates on-demand rendering
   - Increases cache size but improves reliability

2. **Use stateless rendering service**
   - Separate microservice for plot rendering
   - HTTP-based, can timeout/restart independently
   - Better isolation from main server

3. **Upgrade to Plotly 6.x**
   - Newer versions have better Kaleido integration
   - May have fixes for threading issues

## Action Items

1. ✓ Document the issue (this file)
2. ⚠️ Implement serialization lock (Option 1)
3. ⚠️ Add timeout wrapper for safety
4. ⚠️ Add health check that detects rendering hangs
5. ⚠️ Consider pre-rendering strategy for production

## Related Files

- `/data/sources/glucosedao/gluRPC/src/glurpc/core.py` (lines 266-345)
- `/data/sources/glucosedao/gluRPC/src/glurpc/logic.py` (lines 507-590)
- `/data/sources/glucosedao/gluRPC/src/glurpc/engine.py` (CalcWorker implementation)

## References

- Plotly Kaleido threading issues: https://github.com/plotly/Kaleido/issues
- Async/threading best practices with Plotly
- Similar issues reported in production environments

