# Task Notification Debugging Notes

## Goal
Show per-task completion events in the Explorer UI for ROSE workflows:
- Task ID (counter) + one-line excerpt of stdout (ok) or stderr/exception (err)
- Append as colored lines in the workflow log entry, below the state/stats line

## What is implemented (and left in place)

### `rose/rose/service/api/rest.py` — `_run_workflow`
After `create_learner(...)`, wraps `learner._register_task` to attach an
`asyncio.Future.add_done_callback` to each task future:

```python
task_count     = 0
_orig_register = learner._register_task

def _tracked_register(task_obj, deps=None):
    nonlocal task_count
    task_count += 1
    tid    = task_count
    future = _orig_register(task_obj, deps)

    def _on_done(fut):
        if fut.cancelled():
            return
        exc    = fut.exception()
        is_ok  = exc is None
        raw    = str(fut.result() or "") if is_ok else str(exc)
        excerpt = next((l.strip() for l in raw.splitlines() if l.strip()), "")[:120]
        if self._notify:
            self._notify("task_event", {"wf_id": wf_id, "task_id": tid,
                                        "ok": is_ok, "excerpt": excerpt})

    future.add_done_callback(_on_done)
    return future

learner._register_task = _tracked_register
```

Sends SSE topic `task_event` with `{wf_id, task_id, ok, excerpt}`.

### `rose/rose/service/api/rose.js` — `onNotification`
Handles `task_event` topic: finds the workflow DOM entry and appends a colored
`[task.N] ✓/✗ <excerpt>` line via `insertAdjacentHTML('beforeend', ...)`.

### `src/radical/edge/data/edge_explorer.html` — `handleNotification`
Fixed: `task_event` notifications are no longer stored in `pendingNotifications`
(they were overwriting the `workflow_state` pending replay entry). Only
notifications matching `notificationConfig.topic` are stored for replay.

## What does NOT work yet
`task_event` notifications never reach the browser. Confirmed via:
- `console.log('[rose] onNotification topic=...')` added and removed — no
  `task_event` ever logged; `workflow_state` notifications DO work.
- Conclusion: Python is not sending `task_event`, OR they are silently dropped
  before reaching the JS.

## Hypotheses (in priority order)

1. **ParallelActiveLearner bypass** (most likely if workflow uses parallel mode):
   `_tracked_register` is set on the top-level `ParallelActiveLearner` instance.
   But `ParallelActiveLearner.start()` creates internal `SequentialActiveLearner`
   instances and those call their OWN `_register_task` — never the wrapped one.
   Fix: after `create_learner`, if learner is a `ParallelActiveLearner`, also
   wrap `_create_sequential_learner` to install the tracking on each sub-learner.

2. **`add_done_callback` not firing** (if using SequentialActiveLearner):
   Unclear why. The future is a standard `asyncio.Future`. `set_result` is called
   from the event loop via `handle_task_success`. Callbacks should fire.
   To test: temporarily add `log.info(...)` back to `_on_done` and check edge log.

3. **`_notify` called from wrong context**:
   `_on_done` fires in the event loop (via `call_soon`). `_notify` is thread-safe
   for the bridge. Should not be an issue, but worth checking.

## Next steps to resume

1. Check whether the workflow YAML uses `type: SequentialActiveLearner` or
   `type: ParallelActiveLearner` (or parallel_learners > 1 in config).

2. If Parallel: fix by wrapping `_create_sequential_learner` on the parallel
   learner so each new sequential sub-learner also gets the tracking:
   ```python
   _orig_create = learner._create_sequential_learner
   def _tracked_create(learner_id, config):
       seq = _orig_create(learner_id, config)
       seq._register_task = _tracked_register  # same wrapper
       return seq
   learner._create_sequential_learner = _tracked_create
   ```

3. If Sequential: add `log.info` back temporarily and check edge logs for
   `task N ok/err` — if absent, the callback is not firing at all and we need
   to investigate the asyncflow Future lifecycle more carefully.

## Pending feature: replay all pending notifications as a list
Discussed but deferred: change `pendingNotifications[key] = data` to a list
so that fast-completing workflows can replay all task events (not just the
last one). Currently only the latest notification per key is stored.
Implementation: `pendingNotifications[key] = (pendingNotifications[key] || []).concat([data])`
and iterate in `processPendingNotification`.
