[HELD — independent review pending] Release v0.51.340 — bg_task agent wakeup (trio #2968+#2971+#2979) (#3867)
Some checks failed
Release & Docker / release (push) Has been cancelled

* stage bg_task trio combined (#2979 superset) on master for deep review

* fix(bg_task): unsubscribe SessionChannel on header-write failure (Codex deep-review catch) + regression test

* test: realign on-subscribe-recovery anchor to subscribe_to_session_channel after leak fix

* CHANGELOG: bg_task trio as v0.51.340 LD (HELD pending independent review)

* bg_task trio: apply 3 independent-review (greptile) fixes

1. start_session_turn now threads the session PROFILE model defaults
   (_read_profile_model_config) into the wakeup model-resolve, so a brand-new
   session with an empty model falls back to the profile default not global
   DEFAULT_MODEL. Updated the white-box spy test signature accordingly.
2. /api/session/stream omits the Connection header (HTTP/1.1 keep-alive
   default) to match the #3103 long-lived-SSE pattern.
3. Reaper now prunes _LAST_EMIT_TS for collected sessions so the coalesce
   timestamp map can't grow one permanent entry per session forever.

nesquena APPROVED the PR; these are the 3 non-blocking greptile suggestions.

* test: realign _start_session_turn adapter stub lambda to new profile-defaults signature
This commit is contained in:
nesquena-hermes
2026-06-08 22:36:18 -07:00
committed by GitHub
parent c031cf5c6d
commit 26e133e3e8
27 changed files with 7317 additions and 95 deletions

View File

@@ -3,13 +3,30 @@
## [Unreleased]
## [v0.51.339] — 2026-06-09 — Release LC (targeted workspace create actions)
## [v0.51.340] — 2026-06-09 — Release LD (background-task agent wakeup in WebUI) — ⛔ HELD pending independent review
### Added
- **Create files and folders where you right-click in the workspace.** The workspace context menus (root heading, directory rows, file rows) now offer **New File** / **New Folder** that target the thing you clicked — a directory row creates inside that directory, a file row creates alongside it — instead of always targeting the current directory. The create prompt names the target (e.g. "New file name in `src`:"). (#3855, #3843, @b3nw)
- Background tasks started with `terminal(notify_on_complete=true)` now wake the WebUI agent turn server-side, so the wakeup fires even when no browser tab is open. Closed-tab parity with CLI / Telegram / gateway hosts. (#2968)
- New `bg_task_complete` SSE event with trimmed `{session_id, task_id, completed_at, summary?, event_id}` payload and per-emit `event_id`. The legacy `process_complete` event name is dual-emitted with the same `event_id` for one PR cycle so in-flight WebUI builds keep working, then removed by #2971. (#2968, #2971)
- WebUI surfaces a small toast on background task completion, suppressed when the user is already focused on the target session. (#2971, #2979)
- New `GET /api/session/stream` per-session SSE channel for live-view of server-initiated wakeup turns; the browser's chat-stream renderer is reused (no second renderer). (#2968, #2971)
- New `POST /api/bg-task-complete-ack` diagnostic endpoint accepts `task_id` (canonical) and `process_id` (transitional alias; alias responses include a `Deprecation` header). (#2971, #2979)
### Changed
- WebUI subscribes to `bg_task_complete`; the legacy `process_complete` listener is removed and the browser no longer re-POSTs `wakeup_prompt` (wakeup is server-driven). Dedupe uses a 60s TTL ring buffer keyed `(session_id, event_id)`. (#2971)
- `POST /api/process-complete-ack` is replaced by `/api/bg-task-complete-ack`; the legacy path returns HTTP 410 with `X-Replaced-By: /api/bg-task-complete-ack`. Wired ahead of the CSRF gate so stale tabs see a discoverable hard error instead of a silent 403/404. (#2968)
### Fixed
- **The "Add as space?" prompt after creating a folder now has a clear "No" option** instead of an unlabeled cancel. (#3858, #3856, @b3nw)
- Cross-session `notify_on_complete` wakeup no longer misroutes between concurrent WebUI sessions. Per-turn session identity is bound to a `contextvars.ContextVar` (`gateway.session_context` + `tools.approval`) inside the turn worker thread so concurrent background spawns inherit task/thread-local identity instead of racing on a process-global env slot, with a completion-time owner cross-check as defense-in-depth. (#2968)
- Fast (sub-teardown) background tasks completing while the WebUI session is technically still in its turn-teardown window now have their wakeup persisted in a deferred-wakeup map and drained at the turn's active→idle transition, so autonomous agent loops no longer lose the wakeup. (#2968)
- A background-task wakeup is no longer lost when the server-side wakeup turn races a human turn during session teardown. The teardown idle-hook atomically claims the deferred wakeup and discards the pending marker before starting the turn; if that turn then `409`s on a concurrent `/api/chat/start`, the wakeup is re-queued (idempotent per `process_id`) so a later teardown or the next-turn drain redelivers it instead of dropping it. (#2971)
- Server-initiated wakeup turn now renders live in an open tab by fanning a `server_turn_started` frame onto the per-session live-view channel; no more "needs manual refresh" after a server-driven wakeup. (#2968)
- SSE handlers now arm a 20s socket write deadline once per connection so a slow/backgrounded tab whose recv window fills no longer pins its HTTP worker thread indefinitely, applied uniformly to the 6 long-lived SSE endpoints (chat-stream, terminal, gateway, approval, clarify, session). (#2968)
- Focused background-task completion viewers still emit `/api/bg-task-complete-ack` for server cleanup/diagnostics while keeping the focused-session toast suppressed. (#2979)
- A failed session load (network error / server 4xx/5xx) no longer permanently silences the per-session SSE stream. `loadSession` stops the stream on entry but previously only restarted it on the success path; a mid-load failure left the on-screen session's stream closed, silently dropping `bg_task_complete` events until the user navigated again. The metadata-fetch error path now restarts the stream for the session still on screen, skipping the restart when a newer load is in flight or when the current session 404'd and self-healed away. (#2979)
## [v0.51.338] — 2026-06-09 — Release LB (saved prompts library)

1248
api/background_process.py Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -2791,6 +2791,142 @@ _available_models_cache_lock = threading.RLock() # must be RLock: cold path ref
_cache_build_cv = threading.Condition(_available_models_cache_lock) # shares underlying RLock so notify_all() is safe inside with _available_models_cache_lock
_cache_build_in_progress = False # True while a cold path is actively building
# Hard wall-clock budget for a COLD live provider-catalog rebuild when it is
# run from a foreground request path. The live rebuild does one network probe
# per detected provider (Copilot token-exchange HTTPS, OpenRouter /v1/models,
# Nous /models, ...). On a flaky / corp / WSL network any single probe can
# stall for its full per-call timeout (Copilot urllib timeout=10s) and, summed
# across N providers, block the request thread for tens of seconds. This bounds
# the time a foreground caller will wait: past the budget it returns a usable
# fallback (last-known disk cache or a network-free minimal catalog) and lets
# the rebuild finish out-of-band and populate the cache for the next call.
# Set HERMES_WEBUI_MODELS_REBUILD_BUDGET=0 to restore the legacy synchronous
# (unbounded) behaviour.
try:
_LIVE_REBUILD_BUDGET_SECONDS: float = float(
os.getenv("HERMES_WEBUI_MODELS_REBUILD_BUDGET", "4") or "4"
)
except (TypeError, ValueError):
_LIVE_REBUILD_BUDGET_SECONDS = 4.0
# ── Budget-exceeded warning rate-limit ───────────────────────────────────────
# Q-2979-A3 / Copilot discussion_r3305864400: the live-rebuild-budget-exceeded
# warning at _invoke_models_rebuild's slow-path is potentially high-volume —
# every provider catalog refresh that runs past _LIVE_REBUILD_BUDGET_SECONDS
# emits one, so a hung upstream probe (or a sustained burst of cold callers)
# could flood the log at warning level. Rate-limit per reason: the FIRST
# occurrence in a cooldown window logs at warning; subsequent occurrences in
# the same window log at info (so log signal stays useful but volume bounded).
# Override the default cooldown via HERMES_WEBUI_BUDGET_WARN_COOLDOWN (seconds).
try:
_BUDGET_WARN_COOLDOWN_SECONDS: float = float(
os.getenv("HERMES_WEBUI_BUDGET_WARN_COOLDOWN", "300") or "300"
)
except (TypeError, ValueError):
_BUDGET_WARN_COOLDOWN_SECONDS = 300.0
_BUDGET_WARN_STATE: dict[str, float] = {}
_BUDGET_WARN_LOCK = threading.Lock()
def _should_warn_budget(reason: str, cooldown_s: float | None = None) -> bool:
"""Return True iff the budget warning for ``reason`` should log at
warning level (first hit, or last warn-level emit was more than
``cooldown_s`` seconds ago). Otherwise False — the caller should demote
to info for the same payload so the signal is retained but the noise is
capped. Thread-safe; the cooldown is shared across all live-rebuild
callers in this process.
"""
cooldown = (
_BUDGET_WARN_COOLDOWN_SECONDS if cooldown_s is None else float(cooldown_s)
)
now = time.monotonic()
with _BUDGET_WARN_LOCK:
last = _BUDGET_WARN_STATE.get(reason)
if last is None or (now - last) >= cooldown:
_BUDGET_WARN_STATE[reason] = now
return True
return False
def _invoke_models_rebuild(builder):
"""Indirection seam around the cold catalog rebuild.
Production simply calls ``builder()``. Exists so tests can simulate a
slow / hanging provider probe without having to reach the closure that
actually does the per-provider network calls.
"""
return builder()
def _minimal_static_models_catalog() -> dict:
"""Return a network-free /api/models catalog derived from config + auth.
Used as the fast fallback when a foreground caller must NOT pay the live
provider probe: server-initiated wakeup turns (Option Z) and the
bounded-rebuild timeout path. It is enough for
``_resolve_compatible_session_model_state`` (which only needs
``default_model`` / ``active_provider`` plus the persisted session model)
and keeps the picker non-empty. Intentionally NOT written to the 24h
cache so a subsequent human ``/api/models`` still triggers a real rebuild.
"""
try:
active_provider = None
cfg_base_url = ""
model_cfg = cfg.get("model", {}) if isinstance(cfg, dict) else {}
if isinstance(model_cfg, dict):
active_provider = model_cfg.get("provider")
cfg_base_url = model_cfg.get("base_url", "") or ""
if active_provider:
try:
active_provider = _resolve_configured_provider_id(
active_provider, cfg, base_url=cfg_base_url
)
except Exception:
active_provider = str(active_provider or "").strip() or None
if not active_provider:
try:
_ap = _get_auth_store_path()
if _ap.exists():
_store = json.loads(_ap.read_text(encoding="utf-8"))
active_provider = (
_resolve_configured_provider_id(
_store.get("active_provider"), cfg, base_url=cfg_base_url
)
or None
)
except Exception:
pass
default_model = get_effective_default_model(cfg)
groups: list[dict] = []
if default_model:
try:
label = _get_label_for_model(default_model, [])
except Exception:
label = default_model
groups.append(
{
"provider": "Default",
"provider_id": active_provider or "default",
"models": [{"id": default_model, "label": label}],
}
)
return {
"active_provider": active_provider,
"default_model": default_model,
"configured_model_badges": {},
"groups": groups,
}
except Exception:
logger.debug("minimal static models catalog build failed", exc_info=True)
return {
"active_provider": None,
"default_model": "",
"configured_model_badges": {},
"groups": [],
}
# Cache for credential pool results -- calling load_pool() per-provider per-server
# session is expensive (~10s for zai due to endpoint probing). The credential pool
# only changes when the user adds/removes credentials, which is rare; a 24h TTL
@@ -3435,7 +3571,7 @@ def _read_visible_codex_cache_model_ids() -> list[str]:
return ordered
def get_available_models() -> dict:
def get_available_models(*, prefer_cache: bool = False) -> dict:
"""
Return available models grouped by provider.
@@ -3450,6 +3586,15 @@ def get_available_models() -> dict:
'default_model': str,
'groups': [{'provider': str, 'models': [{'id': str, 'label': str}]}]
}
``prefer_cache=True`` resolves WITHOUT ever triggering a live provider
probe: it serves the warm in-memory cache, then the last-known on-disk
cache, and only as a last resort a network-free minimal catalog
(config/auth derived). It NEVER does the per-provider live rebuild (the
Copilot token-exchange HTTPS call et al.). This is the path a
server-initiated wakeup turn (Option Z) takes so a cold catalog can never
block the wakeup chat/start on a flaky network. A normal human request
leaves this False and keeps the full live-discovery behaviour.
"""
global _cache_build_in_progress, _available_models_cache, _available_models_cache_ts, _available_models_cache_source_fingerprint, _cache_build_cv
# Config mtime check — must come before any config reads.
@@ -4867,25 +5012,175 @@ def get_available_models() -> dict:
_save_models_cache_to_disk(disk_groups)
return copy.deepcopy(disk_groups)
# ── prefer_cache: NEVER run the live provider rebuild ────────────────
# Server-initiated wakeup turns (Option Z) reach here with a cold
# cache (the drain thread fires while idle; the catalog warmed by a
# human's /api/models has expired or was never built). The live
# rebuild does a Copilot token-exchange HTTPS call per the proven
# thread-stack; on this WSL/corp network it stalls the wakeup
# chat/start indefinitely. A wakeup turn does NOT need the full live
# catalog — _resolve_compatible_session_model_state only needs
# default_model/active_provider and trusts the persisted session
# model. Serve a network-free minimal catalog instead and let a later
# human request do the real live rebuild.
if prefer_cache:
# NOTE (Greptile P1): do NOT touch _cache_build_in_progress here.
# This branch never set the flag (only the cold path below does),
# and `should_wait` is sampled outside the lock (line ~4964). A
# concurrent cold-path caller can flip the flag to True after our
# sample but before we acquire the lock; clearing it here would
# prematurely release that rebuild's serialization, waking waiters
# to an empty cache and triggering a second live rebuild. Just
# serve the network-free minimal catalog and leave the flag alone.
return copy.deepcopy(_minimal_static_models_catalog())
# Cold path: full rebuild — only one thread reaches here at a time
with _cache_build_cv:
_cache_build_in_progress = True
try:
result = _build_available_models_uncached()
except Exception:
# Always reset the flag so waiting threads don't block for 60s
# Legacy synchronous (unbounded) rebuild — opt-in via budget<=0.
if _LIVE_REBUILD_BUDGET_SECONDS <= 0:
try:
result = _invoke_models_rebuild(_build_available_models_uncached)
except BaseException:
# Always reset the flag so waiting threads don't block for 60s
with _cache_build_cv:
_cache_build_in_progress = False
_cache_build_cv.notify_all()
raise
with _cache_build_cv:
_available_models_cache = result
_available_models_cache_ts = time.monotonic()
_available_models_cache_source_fingerprint = _models_cache_source_fingerprint()
_cache_build_in_progress = False
_cache_build_cv.notify_all()
_save_models_cache_to_disk(result)
return copy.deepcopy(result)
# ── Bounded rebuild (defense-in-depth) ───────────────────────────────
# The live rebuild does a network probe per provider (Copilot token
# exchange over HTTPS, OpenRouter/Nous /models, ...). On a flaky / corp
# / WSL network any single probe can stall for its full per-call
# timeout and, summed across providers, pin a foreground request
# thread for tens of seconds (the wakeup-turn / chat-start hang).
#
# Run the rebuild on a daemon worker; the foreground waits at most
# _LIVE_REBUILD_BUDGET_SECONDS.
#
# WITHIN budget (the normal fast case): the FOREGROUND publishes the
# result synchronously and only then returns — preserving the exact
# pre-existing contract (cache + on-disk file populated by the time
# get_available_models() returns). The worker stays hands-off.
#
# OVER budget (a provider probe is slow/hung): the foreground returns
# the best fallback immediately and the still-running worker publishes
# its result out-of-band when it finally finishes, so the next caller
# gets a warm cache instead of paying the cold rebuild again.
#
# ``_publish_models_result`` / ``box["published"]`` ensure exactly one
# publisher even at the budget boundary (no double write, no lost
# refresh). The worker only touches _cache_build_cv after the
# foreground releases the RLock by returning, so no lock inversion.
build_done = threading.Event()
budget_exceeded = threading.Event()
publish_lock = threading.Lock()
box: dict = {}
def _publish_models_result(result):
global _cache_build_in_progress, _available_models_cache
global _available_models_cache_ts, _available_models_cache_source_fingerprint
with _cache_build_cv:
_available_models_cache = result
_available_models_cache_ts = time.monotonic()
_available_models_cache_source_fingerprint = (
_models_cache_source_fingerprint()
)
_cache_build_in_progress = False
_cache_build_cv.notify_all()
try:
_save_models_cache_to_disk(result)
except Exception:
logger.debug("models cache disk save failed", exc_info=True)
def _clear_build_in_progress():
global _cache_build_in_progress
with _cache_build_cv:
_cache_build_in_progress = False
_cache_build_cv.notify_all()
raise
with _cache_build_cv:
_available_models_cache = result
_available_models_cache_ts = time.monotonic()
_available_models_cache_source_fingerprint = _models_cache_source_fingerprint()
_cache_build_in_progress = False
_cache_build_cv.notify_all()
_save_models_cache_to_disk(result)
return copy.deepcopy(result)
def _claim_publish() -> bool:
"""Return True iff the caller won the right to publish."""
with publish_lock:
if box.get("published"):
return False
box["published"] = True
return True
def _rebuild_worker():
try:
box["result"] = _invoke_models_rebuild(_build_available_models_uncached)
except Exception as exc: # noqa: BLE001 — propagated to caller
box["error"] = exc
finally:
build_done.set()
# Only publish out-of-band if the foreground already gave up
# (over budget). Within budget the foreground publishes
# synchronously, so the worker must NOT touch the cache.
if budget_exceeded.is_set() and _claim_publish():
if "result" in box:
_publish_models_result(box["result"])
else:
_clear_build_in_progress()
_worker = threading.Thread(
target=_rebuild_worker,
name="models-catalog-rebuild",
daemon=True,
)
_worker.start()
if build_done.wait(timeout=_LIVE_REBUILD_BUDGET_SECONDS):
# Build finished within budget — foreground publishes
# synchronously, exactly like the legacy path.
if "error" in box:
_clear_build_in_progress()
raise box["error"]
if _claim_publish():
_publish_models_result(box["result"])
return copy.deepcopy(box["result"])
# Budget elapsed. Mark it so the worker knows it owns out-of-band
# publication. Handle the tiny race where the build completed between
# wait() returning False and here: if so, still publish synchronously
# so this caller honours the cache contract.
budget_exceeded.set()
if build_done.is_set() and "error" not in box and "result" in box:
if _claim_publish():
_publish_models_result(box["result"])
return copy.deepcopy(box["result"])
# Genuinely slow/hung probe: serve the best fallback now; the worker
# keeps going and refreshes the cache for the next caller.
# Rate-limit the warning per Q-2979-A3 — see _should_warn_budget; a
# sustained budget breach demotes to info after the first emit in
# each cooldown window so log volume stays bounded.
_budget_log_msg = (
"live provider-catalog rebuild exceeded %.1fs budget — serving "
"fallback, refreshing catalog out-of-band"
)
if _should_warn_budget("live_rebuild_budget_exceeded"):
logger.warning(_budget_log_msg, _LIVE_REBUILD_BUDGET_SECONDS)
else:
logger.info(_budget_log_msg, _LIVE_REBUILD_BUDGET_SECONDS)
# Note: ``disk_groups``, if non-None, was already consumed by the
# cold-path early-return at the "Cold path: disk cache hit" branch
# above (line ~4608). Any execution that reaches HERE necessarily
# took the live-rebuild branch, which means ``disk_groups is None``
# at this point — so we don't re-check it. Per Copilot review on
# PR #2971: the previous ``if disk_groups is not None`` branch
# here was dead code. Fall back directly to the static minimal
# catalog (no second disk read).
return copy.deepcopy(_minimal_static_models_catalog())
# ── Static file path ─────────────────────────────────────────────────────────
@@ -4987,6 +5282,51 @@ STREAM_GOAL_RELATED: dict = {} # stream_id -> bool: only evaluate goal for goal
STREAM_LAST_EVENT_ID: dict = {} # stream_id -> latest journal event_id for `id:` field on live SSE frames (stage-364)
PENDING_GOAL_CONTINUATION: set = set() # session_ids awaiting a goal continuation turn (#1932)
# ── notify_on_complete agent-wakeup wiring ─────────────────────────────────
# When terminal(notify_on_complete=true, background=true) fires, the process
# registry pushes a completion event onto tools.process_registry.completion_queue.
# A drain task spawned at WebUI startup (api/background_process.py) reads that
# queue and emits an SSE `process_complete` event to the matching session.
# PROCESS_SESSION_INDEX maps the per-process "session_key" (set in the spawned
# subprocess via HERMES_SESSION_KEY) back to the WebUI session_id that owns it,
# so the drain task can route the event to the right SSE channel.
# PENDING_BG_TASK_COMPLETIONS mirrors PENDING_GOAL_CONTINUATION: server-side
# marker discarded atomically by routes.py when the frontend re-POSTs the
# wakeup_prompt as the next user turn. (process_complete event, agent wakeup fix)
PROCESS_SESSION_INDEX: dict = {} # process_registry session_key -> WebUI session_id
PROCESS_SESSION_INDEX_LOCK = threading.Lock()
PENDING_BG_TASK_COMPLETIONS: set = set() # session_ids awaiting a process_complete wakeup turn
BG_TASK_COMPLETE_EVENTS_SEEN: dict = {} # session_id -> set[process_id] for idempotency
BG_TASK_COMPLETE_EVENTS_SEEN_LOCK = threading.Lock()
# Defer-path fix (fast-bg-task wakeup race): when a completion arrives while a
# turn is active, Option Z's drain branch CANNOT start a turn (would 409). The
# pre-existing PENDING_BG_TASK_COMPLETIONS marker was a bare session_id flag —
# the wakeup_prompt was DISCARDED, and the only consumer (PR #2279 next-turn
# drain) reads completion_queue, which the Option Z drain thread already
# emptied. So for an autonomous agent (no next user turn) the deferred wakeup
# was lost forever. DEFERRED_PROCESS_WAKEUPS persists the actual prompt(s) so a
# turn-teardown idle-hook (api/streaming) can redeliver them once the session
# goes idle — symmetric with the idle branch (idle now → fire now; busy now →
# fire at turn-end). Atomic claim (pop under lock) guarantees single delivery:
# whoever claims first (teardown hook OR next-turn drain) fires; the other
# finds nothing → no double-fire, no wakeup loop.
DEFERRED_PROCESS_WAKEUPS: dict = {} # session_id -> list[{"process_id", "wakeup_prompt"}]
DEFERRED_PROCESS_WAKEUPS_LOCK = threading.Lock()
# ── Persistent per-session SSE channel (Option X) ──────────────────────────
# A long-lived SSE channel scoped to a WebUI session_id rather than a single
# agent turn (stream_id). Subscribed to by the frontend on session mount,
# torn down on session unmount, and refcounted across tabs. Used to deliver
# events (currently process_complete) that fire while no agent turn is
# active — bridging the gap that PR #2242 + #2279 left when STREAMS has
# already been torn down. The registry lives in api.background_process; this
# constant is the idle-cap before the reaper collects an unsubscribed
# channel. 4h is a defensive ceiling against zombie connections; the
# subscribers-empty grace path (60s) handles ordinary tab-close traffic.
SESSION_CHANNEL_IDLE_TTL_SECS: int = 14400 # 4 hours
SESSION_CHANNEL_SUBSCRIBER_GRACE_SECS: int = 60 # subscribers-empty grace
# Active agent-run registry. This intentionally tracks worker lifecycle rather
# than SSE lifecycle: cancel/reconnect may remove STREAMS while the worker is
# still unwinding, blocked in a provider call, or waiting for delegated work.

View File

@@ -1105,6 +1105,7 @@ from api.config import (
_save_yaml_config_file,
reload_config,
_cfg_lock,
PENDING_BG_TASK_COMPLETIONS,
)
from api.helpers import (
require,
@@ -2255,6 +2256,7 @@ def _resolve_compatible_session_model_state(
profile_provider: str | None = None,
profile_default_model: str | None = None,
explicit_model_pick: bool = False,
prefer_cached_catalog: bool = False,
) -> tuple[str, str | None, bool]:
"""Return (effective_model, effective_provider, model_was_normalized).
@@ -2276,6 +2278,16 @@ def _resolve_compatible_session_model_state(
OpenRouter ``/models``, LM Studio ``/models``, credential pool refresh) —
those used to wedge the handler for >100s and trigger 502s on default-60s
reverse proxies, even though the WebUI itself eventually responded.
``prefer_cached_catalog=True`` (ours-original) makes the catalog lookup
non-blocking: it resolves from the warm/disk cache or a network-free
minimal catalog and NEVER triggers a live per-provider rebuild (the
Copilot token-exchange HTTPS call that hangs a server-initiated wakeup
turn — see rebase report §1/§3/model-resolve-hang). Human-initiated
chat/start leaves this False to keep full live discovery; a session that
already has a persisted model still resolves correctly because the
persisted model wins over the catalog and the catalog is only consulted
for the default-model backstop.
"""
model = str(model_id or "").strip()
requested_provider = _clean_session_model_provider(model_provider)
@@ -2292,7 +2304,33 @@ def _resolve_compatible_session_model_state(
if not explicit_provider and not stale_codex_openai_slash_id:
return model, requested_provider, False
catalog = get_available_models()
# Default (human chat/start) path calls get_available_models() with NO
# kwargs so it stays signature-compatible with the many tests that stub
# get_available_models as a zero-arg callable. Only the server-side wakeup
# path (prefer_cached_catalog=True) opts into the cache-only mode. Some
# tests monkeypatch get_available_models as a zero-arg callable, so probe
# the (possibly monkeypatched) signature for ``prefer_cache`` rather than
# catching TypeError — a blanket ``except TypeError`` would also swallow a
# genuine TypeError raised *inside* get_available_models(prefer_cache=True)
# and silently fall back to the slow live provider rebuild that
# prefer_cached_catalog=True is meant to avoid.
if prefer_cached_catalog:
import inspect as _inspect
try:
_gam_accepts_prefer_cache = (
"prefer_cache" in _inspect.signature(get_available_models).parameters
)
except (TypeError, ValueError):
# Builtins / C-callables can refuse introspection; assume the
# zero-arg stub shape in that case.
_gam_accepts_prefer_cache = False
if _gam_accepts_prefer_cache:
catalog = get_available_models(prefer_cache=True)
else:
catalog = get_available_models()
else:
catalog = get_available_models()
default_model = str(catalog.get("default_model") or DEFAULT_MODEL or "").strip()
# Profile-aware resolution: when the caller supplies profile context
@@ -2547,6 +2585,17 @@ def _resolve_effective_session_model_for_display(session) -> str:
requested_provider,
profile_provider=_pp_provider,
profile_default_model=_pp_default,
# GET /api/session is a hot, side-effect-free per-tab/per-poll path.
# It must never pay the cold live provider-catalog rebuild (a
# botocore IMDS probe that cannot resolve on a non-AWS / WSL / corp
# network, plus anthropic/openrouter /models). That rebuild is
# un-cacheable here (auth.json fingerprint churn) so every cold call
# cost ~10s and, run concurrently across browser tabs, serialized on
# the models-cache lock and starved SSE/streaming -> BrokenPipe storm
# (#multi-tab-streaming-interlock). The persisted session model is
# authoritative; the catalog is only a default-model backstop, which
# the network-free minimal catalog already provides.
prefer_cached_catalog=True,
)
return effective_model or original_model
@@ -2559,6 +2608,11 @@ def _resolve_effective_session_model_provider_for_display(session) -> str | None
requested_provider,
profile_provider=_pp_provider,
profile_default_model=_pp_default,
# See _resolve_effective_session_model_for_display: same hot
# side-effect-free GET /api/session path; must not trigger the cold
# live rebuild. prefer_cached_catalog resolves from warm/disk cache
# or the network-free minimal catalog.
prefer_cached_catalog=True,
)
return provider
@@ -3408,6 +3462,7 @@ from api.upload import (
)
from api.streaming import (
_sse,
_sse_set_write_deadline,
_run_agent_streaming,
cancel_stream,
_materialize_pending_user_turn_before_error,
@@ -6208,6 +6263,9 @@ def handle_get(handler, parsed) -> bool:
if parsed.path == "/api/clarify/stream":
return _handle_clarify_sse_stream(handler, parsed)
if parsed.path == "/api/session/stream":
return _handle_session_sse_stream(handler, parsed)
if parsed.path == "/api/clarify/inject_test":
# Loopback-only: used by automated tests; blocked from any remote client
if handler.client_address[0] != "127.0.0.1":
@@ -6662,6 +6720,37 @@ def handle_post(handler, parsed) -> bool:
finally:
if diag:
diag.finish()
# T1 deprecation alias for the legacy ack endpoint that the pre-rename
# WebUI used to POST to after handling ``process_complete``. The new
# canonical SSE event is ``bg_task_complete`` and the new ack endpoint
# will be ``/api/bg-task-complete-ack`` (introduced by PR (b), the WebUI
# half of the split). Until PR (b) lands we keep the old path responding
# with HTTP 410 Gone + ``X-Replaced-By`` so any stale tab posting under
# the old name fails loudly with a discoverable hint. The handler runs
# BEFORE the CSRF gate on purpose: an old tab will not carry a CSRF token
# for the deprecated path, and surfacing 410 (not 403) is the correct
# contract here.
if parsed.path == "/api/process-complete-ack":
if diag:
diag.stage("process_complete_ack_deprecated")
try:
j(
handler,
{
"error": (
"gone: /api/process-complete-ack was replaced by "
"/api/bg-task-complete-ack as part of the "
"process_complete -> bg_task_complete event rename"
),
"replaced_by": "/api/bg-task-complete-ack",
},
status=410,
extra_headers={"X-Replaced-By": "/api/bg-task-complete-ack"},
)
return True
finally:
if diag:
diag.finish()
# CSRF: reject cross-origin or tokenless authenticated browser requests.
# /api/auth/login has no authenticated session token yet, and /api/csp-report
# is intentionally unauthenticated for browser-generated violation reports.
@@ -7625,6 +7714,9 @@ def handle_post(handler, parsed) -> bool:
if parsed.path == "/api/goal":
return _handle_goal_command(handler, body)
if parsed.path == "/api/bg-task-complete-ack":
return _handle_bg_task_complete_ack(handler, body)
if parsed.path == "/api/chat/start":
return _handle_chat_start(handler, body, diag=diag)
@@ -9139,6 +9231,7 @@ def _handle_sse_stream(handler, parsed):
handler.send_header("X-Accel-Buffering", "no")
handler.send_header("Connection", "close")
handler.end_headers()
_sse_set_write_deadline(handler) # Defect A: slow tab can't pin this thread
replay_cutoff_seq = None
if qs.get("replay", [""])[0] or qs.get("after_seq", [None])[0] not in (None, "") or qs.get("after_event_id", [None])[0]:
snapshot_cutoff_seq = _run_journal_same_run_seq(
@@ -9318,6 +9411,7 @@ def _handle_terminal_output(handler, parsed):
handler.send_header("X-Accel-Buffering", "no")
handler.send_header("Connection", "close")
handler.end_headers()
_sse_set_write_deadline(handler) # Defect A: slow tab can't pin this thread
try:
while True:
try:
@@ -9403,6 +9497,7 @@ def _handle_gateway_sse_stream(handler, parsed):
# session list every ~1s. Letting the server close the socket
# naturally after the stream ends is sufficient.
handler.end_headers()
_sse_set_write_deadline(handler) # Defect A: slow tab can't pin this thread
q = watcher.subscribe()
try:
@@ -10478,6 +10573,7 @@ def _handle_approval_sse_stream(handler, parsed):
handler.send_header('X-Accel-Buffering', 'no')
handler.send_header('Connection', 'close')
handler.end_headers()
_sse_set_write_deadline(handler) # Defect A: slow tab can't pin this thread
from api.streaming import _sse
@@ -10579,6 +10675,7 @@ def _handle_clarify_sse_stream(handler, parsed):
handler.send_header('X-Accel-Buffering', 'no')
handler.send_header('Connection', 'close')
handler.end_headers()
_sse_set_write_deadline(handler) # Defect A: slow tab can't pin this thread
from api.streaming import _sse
@@ -10602,6 +10699,126 @@ def _handle_clarify_sse_stream(handler, parsed):
clarify_sse_unsubscribe(sid, q)
def _handle_session_sse_stream(handler, parsed):
"""SSE endpoint for the persistent per-session channel (Option X).
Subscribes to ``api.background_process.SESSION_CHANNELS[sid]`` — a channel
that lives across agent turns (unlike STREAMS, which is torn down at
end-of-turn). Used to deliver ``bg_task_complete`` events that fire while
no agent turn is active.
Lifecycle: opened by the frontend at session mount, closed at unmount or
on tab close. Multiple tabs share one SessionChannel (refcounted via
subscribe/unsubscribe). 30s SSE keepalive comments keep the proxy alive.
Reaper-driven idle TTL (default 4h) prevents zombie channels.
"""
sid = parse_qs(parsed.query).get("session_id", [""])[0]
if not sid:
return bad(handler, "session_id is required")
from api.background_process import (
subscribe_to_session_channel,
active_stream_id_for_session,
)
# Atomic get-or-create + subscribe under SESSION_CHANNELS_LOCK. Doing these
# two steps separately (get_or_create_session_channel then ch.subscribe)
# left a TOCTOU gap where the reaper — which also holds
# SESSION_CHANNELS_LOCK and collects idle 0-subscriber channels in one
# critical section — could collect the channel between the two calls,
# orphaning this subscriber on a channel no longer in SESSION_CHANNELS.
# bg_task_complete emits would then never reach this queue. See
# subscribe_to_session_channel for the full rationale (PR #2971 Greptile P1).
ch, q = subscribe_to_session_channel(sid, maxsize=64)
# NOTE: ``subscribe_to_session_channel`` above acquires a subscriber slot
# that MUST be released on every exit path. Header setup
# (``send_response`` / ``send_header`` / ``end_headers`` /
# ``_sse_set_write_deadline``) and the initial-frame + on-subscribe
# recovery writes below all touch the socket and can raise a member of
# ``_CLIENT_DISCONNECT_ERRORS`` (BrokenPipeError / ConnectionResetError) if
# the client drops immediately after subscribing. If that happened outside
# this try/finally the ``ch.unsubscribe(q)`` cleanup would be skipped,
# permanently leaking a subscriber. Because
# ``SessionChannel.reaper_should_collect()`` refuses to collect any channel
# with ``sub_count > 0``, a single ghost subscriber blocks the reaper
# forever and the channel zombies in SESSION_CHANNELS. So EVERYTHING from
# the subscribe onward — header setup included — runs inside one
# try/finally that unconditionally unsubscribes.
try:
handler.send_response(200)
handler.send_header('Content-Type', 'text/event-stream; charset=utf-8')
handler.send_header('Cache-Control', 'no-cache')
handler.send_header('X-Accel-Buffering', 'no')
# #3103: omit the Connection header — rely on the HTTP/1.1 keep-alive
# default, matching the other long-lived SSE handlers (gateway/session
# events) that fixed the reconnect-storm. An explicit value here is a
# third, inconsistent approach (greptile flag).
handler.end_headers()
_sse_set_write_deadline(handler) # Defect A: slow tab can't pin this thread
from api.streaming import _sse
# Push an initial frame so the client has confirmation the channel is
# live (mirrors approval/clarify which send an 'initial' frame). No
# snapshot data is needed — this channel only carries forward-looking
# events, not pending state.
_sse(handler, 'initial', {"session_id": sid})
# ── Open-tab live-view self-heal (root cause: lost server_turn_started) ──
# The `server_turn_started` fan-out (routes.start_session_turn) is a
# fire-and-forget SessionChannel.emit with NO replay buffer: it reaches
# only the subscribers connected at the exact emit instant. A tab whose
# per-session EventSource was momentarily absent at that instant — a
# transient SSE drop, a reverse-proxy idle-timeout, or browser
# connection-pool starvation (all common behind a corporate proxy) —
# misses the frame permanently, so a SERVER-initiated wakeup turn never
# renders live and the user must hard-refresh (the reported defect). The
# server-side wakeup itself ran and persisted fine; only the live-view
# was lost. On (re)subscribe, if the session has a live run RIGHT NOW,
# replay a synthetic `server_turn_started` to THIS new subscriber so the
# open tab attaches its existing chat-stream renderer (attachLiveStream)
# and self-heals with no refresh. `recovered: True` lets the frontend
# use the replay (reconnecting) attach so the renderer picks up the
# in-progress stream from the run journal rather than expecting token 0.
# Idempotent: the frontend dedupes by (session_id, stream_id) — if the
# original frame WAS delivered this is a harmless no-op there.
try:
recover_stream_id = active_stream_id_for_session(sid)
if recover_stream_id:
_sse(handler, 'server_turn_started', {
"session_id": sid,
"stream_id": recover_stream_id,
"source": "subscribe_recovery",
"recovered": True,
})
except _CLIENT_DISCONNECT_ERRORS:
# Client vanished mid-recovery — re-raise so the outer handler
# treats it as a normal disconnect and the finally still cleans up.
raise
except Exception:
logger.debug(
"session-stream on-subscribe recovery failed for %s", sid,
exc_info=True,
)
while True:
try:
payload = q.get(timeout=_SSE_HEARTBEAT_INTERVAL_SECONDS)
except queue.Empty:
handler.wfile.write(b': keepalive\n\n')
handler.wfile.flush()
continue
if payload is None:
break
event_name, data = payload
_sse(handler, event_name, data)
except _CLIENT_DISCONNECT_ERRORS:
pass # client went away — normal for long-lived connections
finally:
ch.unsubscribe(q)
def _handle_clarify_inject(handler, parsed):
"""Inject a fake pending clarify prompt -- loopback-only, used by automated tests."""
qs = parse_qs(parsed.query)
@@ -10800,15 +11017,15 @@ def _handle_live_models(handler, parsed):
ids = [m.get("id", "") for m in _data if m.get("id")]
elif isinstance(_body, list):
ids = [m.get("id", m) if isinstance(m, dict) else m for m in _body]
if ids:
logger.debug("Live-fetched %d models from custom provider %s", len(ids), _base_url)
else:
logger.debug("Custom provider returned no models from %s", _base_url)
except Exception as _fetch_err:
logger.debug("Live fetch from custom provider failed: %s", _fetch_err)
# If live fetch succeeded, merge with config entries (live takes
# priority). If live fetch failed, fall back to config-only list.
if ids:
@@ -11597,6 +11814,14 @@ def _start_chat_stream_for_session(
goal_related = True
PENDING_GOAL_CONTINUATION.discard(s.session_id)
# process_complete wakeup (ours-original, Option B): if this session has a
# pending process_complete marker (set by api/background_process.py drain),
# discard it atomically here. Mirrors the goal_continue pattern (#1932).
# The marker is server-internal telemetry; the actual wakeup is delivered
# either server-side (Option Z) or via the PR #2279 next-turn drain.
if s.session_id in PENDING_BG_TASK_COMPLETIONS:
PENDING_BG_TASK_COMPLETIONS.discard(s.session_id)
session_lock = _get_session_agent_lock(s.session_id)
diag.stage("session_lock_wait") if diag else None
while True:
@@ -11754,6 +11979,263 @@ def _runtime_adapter_goal_action(goal_args: str) -> str:
return "set"
def _start_run(
s,
*,
msg: str,
attachments,
workspace: str,
model,
model_provider,
normalized_model,
source: str,
route: str,
diag=None,
):
"""Shared start-run helper for /api/chat/start and start_session_turn.
Centralizes the runtime-adapter selection block (Q-2979-A2 / Copilot
discussion_r3305864087/r3305864173) so both entrypoints honor
``runtime_adapter_enabled()`` / ``runtime_adapter_runner_enabled()`` the
same way. Prior to this helper ``start_session_turn`` bypassed the
adapter path entirely, so a process-wakeup turn skipped the adapter that
a human-typed turn would have hit — a behavioral divergence.
``source`` is the StartRunRequest.source (``"webui"`` for browser POSTs,
``"process_wakeup"`` for the drain-thread wakeup). ``route`` is the
metadata.route label that lands on the run record for observability.
Returns a dict with ``_status`` plus the legacy chat-start response
fields (``stream_id``, ``session_id``, etc.). Adapter selection that
returns no adapter is surfaced as ``{"error": str(exc), "_status": 501}``
so both call sites can map it onto their own HTTP shape.
"""
from api.runtime_adapter import (
LegacyJournalRuntimeAdapter,
StartRunRequest,
build_runtime_adapter,
runtime_adapter_enabled,
runtime_adapter_runner_enabled,
)
if runtime_adapter_enabled() or runtime_adapter_runner_enabled():
def _legacy_start_run(request: StartRunRequest) -> dict:
return _start_chat_stream_for_session(
s,
msg=request.message,
attachments=request.attachments,
workspace=request.workspace or workspace,
model=request.model or model,
model_provider=request.provider or model_provider,
normalized_model=normalized_model,
diag=diag,
)
def _legacy_adapter_factory():
return LegacyJournalRuntimeAdapter(start_run_delegate=_legacy_start_run)
try:
adapter = build_runtime_adapter(
legacy_adapter_factory=_legacy_adapter_factory,
runner_client_factory=_runtime_runner_client_factory,
)
if adapter is None:
raise NotImplementedError("runtime adapter selection returned no adapter")
result = adapter.start_run(
StartRunRequest(
session_id=s.session_id,
message=msg,
attachments=attachments,
workspace=workspace,
profile=getattr(s, "profile", None),
provider=model_provider,
model=model,
source=source,
metadata={"route": route},
)
)
except NotImplementedError as exc:
return {"error": str(exc), "_status": 501}
return _chat_start_response_from_run_start(result)
return _start_chat_stream_for_session(
s,
msg=msg,
attachments=attachments,
workspace=workspace,
model=model,
model_provider=model_provider,
normalized_model=normalized_model,
diag=diag,
)
def start_session_turn(
session_id: str,
message: str,
*,
source: str = "process_wakeup",
):
"""Start a server-side agent turn for ``session_id`` with ``message``.
Option Z primary wakeup entrypoint. This is the minimal, HTTP-handler-free
core that ``/api/chat/start`` already reaches via ``_handle_chat_start`` →
``_start_chat_stream_for_session``. The drain thread
(``api/background_process._process_one``) calls this directly with a
synthetic ``[IMPORTANT: …]`` wakeup_prompt so a background process can wake
the agent server-side with NO browser round-trip — exactly how CLI /
gateway self-wake from a ``notify_on_complete`` completion.
Contract:
- Resolves the session record (profile/workspace/model/model_provider are
already persisted on it; no user auth needed — same trust level as
gateway/cron starting a turn).
- Resolves workspace + model/provider through the SAME helpers
``_handle_chat_start`` uses, so a process-wakeup turn is constructed
identically to a human-typed turn. If the session record has no model
persisted, ``_resolve_compatible_session_model_state`` falls back to the
configured default model/provider (documented in the impl report §1).
- Delegates to ``_start_chat_stream_for_session`` which spawns the agent
on a daemon worker thread (the drain thread NEVER blocks) and serializes
on the per-session agent lock + active-stream guard, so a concurrent
human ``/api/chat/start`` cannot double-start (one wins, the other gets
the existing 409 "session already has an active stream").
Returns the same dict ``_start_chat_stream_for_session`` returns, including
``_status`` (200 on start, 409 when a turn is already active). On 409 the
caller must leave the ``PENDING_BG_TASK_COMPLETIONS`` marker in place so the
PR #2279 next-turn drain delivers the wakeup when the active turn ends.
"""
msg = str(message or "").strip()
if not msg:
return {"error": "message is required", "_status": 400}
try:
s = get_session(session_id)
except KeyError:
return {"error": "Session not found", "_status": 404}
try:
workspace = _resolve_chat_workspace_with_recovery(s, None)
except ValueError as e:
return {"error": str(e), "_status": 400}
requested_model = s.model
requested_provider = getattr(s, "model_provider", None)
# Server-initiated wakeup (Option Z): resolve persisted model via the
# standard helper in cache-only mode so wakeups never trigger a cold
# catalog rebuild. Thread the session's PROFILE model defaults through too
# (mirrors _handle_chat_start) — a brand-new session that spawned a
# background task before its first human turn has an empty s.model, and
# without the profile defaults the resolver would fall back to the global
# DEFAULT_MODEL instead of the profile's configured default (greptile flag).
_pp_provider, _pp_default = _read_profile_model_config(s, requested_provider)
model, model_provider, normalized_model = _resolve_compatible_session_model_state(
requested_model,
requested_provider,
profile_provider=_pp_provider,
profile_default_model=_pp_default,
prefer_cached_catalog=True,
)
resp = _start_run(
s,
msg=msg,
attachments=[],
workspace=workspace,
model=model,
model_provider=model_provider,
normalized_model=normalized_model,
source="process_wakeup",
route="start_session_turn",
)
# ── Defect B: live-view of server-initiated turns ──────────────────────
# Option Z starts this turn server-side, so NO browser EventSource is
# attached to the new STREAMS[stream_id] (the browser only opens
# /api/chat/stream when IT POSTs /api/chat/start). An already-open tab
# would therefore see nothing until a manual refresh re-reads persisted
# state. Fix: fan a lightweight `server_turn_started` {stream_id} frame
# onto the persistent per-session live-view channel. messages.js handles
# it by attaching its EXISTING chat-stream renderer (attachLiveStream) to
# that stream_id — no second renderer, no chat/start POST.
#
# Idempotent with the closed-tab path: get_session_channel() is the
# NON-creating accessor, so when no tab is open this is a pure no-op and
# the server-side wakeup (the Option Z headline) is completely unaffected.
# If the user also has the per-turn chat-stream open, the frontend dedupes
# by stream_id so there is no double-render.
try:
status = int((resp or {}).get("_status", 200) or 200)
stream_id = (resp or {}).get("stream_id")
if status < 400 and stream_id:
from api.background_process import get_session_channel
ch = get_session_channel(session_id)
if ch is not None:
ch.emit(
"server_turn_started",
{
"session_id": str(session_id),
"stream_id": str(stream_id),
"source": source,
},
)
except Exception:
logger.debug(
"server_turn_started fan-out failed for session %s", session_id, exc_info=True
)
return resp
def _handle_bg_task_complete_ack(handler, body):
"""Acknowledge a bg_task_complete SSE event (diagnostic only).
Option Z PIVOT: the agent wakeup is now started SERVER-SIDE by the drain
thread (``api/background_process._process_one`` → ``start_session_turn``)
with NO browser round-trip — the closed-tab case works (parity with
CLI/Telegram). The frontend no longer re-POSTs ``wakeup_prompt`` to
/api/chat/start; the per-session SSE channel is demoted to pure live-view.
This endpoint is therefore a pure no-op for state — it exists so an open
tab can confirm receipt of the live-view event and so a future follow-up
(analytics, telemetry) has a stable hook. ``PENDING_BG_TASK_COMPLETIONS``
is consumed by ``_start_chat_stream_for_session`` when the server-side
wakeup turn (or the next human turn / PR #2279 next-turn drain) runs.
"""
from api.helpers import j
try:
require(body, "session_id")
except ValueError as e:
return bad(handler, str(e))
sid = str(body.get("session_id") or "").strip()
try:
s = get_session(sid)
except KeyError:
return bad(handler, "Session not found", 404)
# process_id accepted as transitional alias; see Deprecation response header
# + maintainer decision on removal milestone / future Sunset header. Only
# flag Deprecation when the alias was ACTUALLY used (i.e. process_id present
# and not empty), even if task_id is also present.
_task_id_present = bool(str(body.get("task_id") or "").strip())
_process_id_present = bool(str(body.get("process_id") or "").strip())
legacy_process_id_used = _process_id_present
pid = str(body.get("task_id") or body.get("process_id") or "").strip()
# Post Option-Z pivot this endpoint owns no state: the server-side drain
# thread starts the wakeup turn, the browser never re-POSTs /api/chat/start.
# `noop` is returned so the diagnostic shape stays explicit about that and
# matches the docstring ("pure no-op for state").
return j(
handler,
{
"ok": True,
"session_id": s.session_id,
"task_id": pid,
"noop": True,
},
extra_headers={"Deprecation": "true"} if legacy_process_id_used else {},
)
def _handle_goal_command(handler, body):
"""Handle WebUI /goal command controls and optional kickoff stream."""
try:
@@ -11959,64 +12441,27 @@ def _handle_chat_start(handler, body, diag=None):
profile_default_model=_pp_default,
explicit_model_pick=explicit_model_pick,
)
from api.runtime_adapter import (
LegacyJournalRuntimeAdapter,
StartRunRequest,
build_runtime_adapter,
runtime_adapter_enabled,
runtime_adapter_runner_enabled,
# NOTE: runtime-adapter selection is delegated to _start_run (shared
# with start_session_turn so both entry points behave identically
# under runtime_adapter_enabled() / runtime_adapter_runner_enabled()
# — Q-2979-A2 / Copilot discussion_r3305864087/r3305864173).
response = _start_run(
s,
msg=msg,
attachments=attachments,
workspace=workspace,
model=model,
model_provider=model_provider,
normalized_model=normalized_model,
source="webui",
route="/api/chat/start",
diag=diag,
)
if runtime_adapter_enabled() or runtime_adapter_runner_enabled():
def _legacy_start_run(request: StartRunRequest) -> dict:
return _start_chat_stream_for_session(
s,
msg=request.message,
attachments=request.attachments,
workspace=request.workspace or workspace,
model=request.model or model,
model_provider=request.provider or model_provider,
normalized_model=normalized_model,
diag=diag,
)
def _legacy_adapter_factory():
return LegacyJournalRuntimeAdapter(start_run_delegate=_legacy_start_run)
try:
adapter = build_runtime_adapter(
legacy_adapter_factory=_legacy_adapter_factory,
runner_client_factory=_runtime_runner_client_factory,
)
if adapter is None:
raise NotImplementedError("runtime adapter selection returned no adapter")
result = adapter.start_run(
StartRunRequest(
session_id=s.session_id,
message=msg,
attachments=attachments,
workspace=workspace,
profile=getattr(s, "profile", None),
provider=model_provider,
model=model,
source="webui",
metadata={"route": "/api/chat/start"},
)
)
except NotImplementedError as exc:
return j(handler, {"error": str(exc)}, status=501)
response = _chat_start_response_from_run_start(result)
else:
response = _start_chat_stream_for_session(
s,
msg=msg,
attachments=attachments,
workspace=workspace,
model=model,
model_provider=model_provider,
normalized_model=normalized_model,
diag=diag,
)
# Map adapter-selection NotImplementedError (501) onto the legacy
# bad-request response shape that this route exposed historically
# before the helper extraction.
if response.get("_status") == 501 and "error" in response:
return j(handler, {"error": response["error"]}, status=501)
status = int(response.pop("_status", 200) or 200)
diag.stage("response_write") if diag else None
return j(handler, response, status=status)

View File

@@ -1255,11 +1255,113 @@ def _build_agent_thread_env(profile_runtime_env: dict | None, workspace: str, se
'HERMES_SESSION_KEY': session_id,
'HERMES_SESSION_ID': session_id,
'HERMES_SESSION_PLATFORM': 'webui',
# process_complete agent-wakeup wiring (ours-original, Option B): the
# terminal_tool watcher routing gate (terminal_tool.py:~1940) reads
# HERMES_SESSION_CHAT_ID to populate pending_watchers for WebUI
# sessions so notify_on_complete completions enqueue and the agent
# can be woken. HERMES_SESSION_ID/PLATFORM come from upstream #2279.
'HERMES_SESSION_CHAT_ID': str(session_id),
'HERMES_HOME': profile_home,
})
return env
# ── Per-turn session identity (xsession wakeup misroute root fix — Option 1) ─
# WebUI bound per-turn session identity ONLY to the process-global
# os.environ['HERMES_SESSION_KEY'] (turn-start, line ~3263) and released the
# env lock BEFORE the agent ran. WebUI never called any contextvar setter, so
# gateway.session_context._SESSION_KEY stayed _UNSET and
# tools.approval.get_current_session_key (the EXACT call a
# notify_on_complete background spawn makes in terminal_tool.py:~1928) fell
# back to that racy process-global slot. Two concurrent WebUI turns therefore
# raced on one slot: session A's spawn could capture session B's id, and at
# completion the server-side wakeup turn started for the WRONG session
# (RCA t_f62ff1e8, agent.log:6632). The agent worker runs synchronously inside
# the _run_agent_streaming thread (concurrent tool batches use
# contextvars.copy_context() so children inherit this binding); binding the
# context-local here makes the capture task/thread-local and race-immune.
def _set_turn_session_identity(session_id: str):
"""Bind THIS turn's session identity to the current (task/thread-local)
context and return an opaque token for _reset_turn_session_identity.
Binds two context-locals so every session-key consumer is covered without
a race:
* ``tools.approval._approval_session_key`` — checked FIRST by
``get_current_session_key`` (the exact call terminal_tool.py makes for
a notify_on_complete background spawn: the bug path).
* ``gateway.session_context._SESSION_KEY`` — read by direct
``get_session_env("HERMES_SESSION_KEY")`` consumers (e.g. the sudo
password cache scope, terminal_tool.py:272).
It deliberately does NOT call ``gateway.session_context.set_session_vars``:
that blanket setter also zeroes the platform/chat_id/user contextvars,
flipping ``HERMES_SESSION_PLATFORM`` from its env fallback (``'webui'``,
still written to os.environ at turn-start) to an explicit ``""`` — which
would break the ``notify_on_complete`` watcher registration gate in
terminal_tool.py:~1966. Only the session-key identity is bound; every
other session var keeps its existing os.environ fallback (CLI/cron compat
preserved — when these contextvars are _UNSET, get_session_env still falls
back to os.environ).
"""
sid = str(session_id or "")
tokens: dict = {}
try:
from tools.approval import set_current_session_key
tokens["approval"] = set_current_session_key(sid)
except Exception:
logger.debug("per-turn approval session-key bind failed", exc_info=True)
try:
from gateway.session_context import _SESSION_KEY as _SK
tokens["session_key"] = _SK.set(sid)
except Exception:
logger.debug("per-turn _SESSION_KEY bind failed", exc_info=True)
return tokens
def _reset_turn_session_identity(tokens) -> None:
"""Restore the context-locals bound by ``_set_turn_session_identity`` via
contextvars reset-token semantics.
Reset-token (not a blanket clear) is the canonical idiom: it composes
correctly under nesting and restores ``_UNSET`` for the top-level turn so
a reused thread-pool worker leaks no identity and CLI/cron env fallback
resumes. Order mirrors the bind in reverse.
"""
if not tokens:
return
tok = tokens.get("session_key")
if tok is not None:
try:
from gateway.session_context import _SESSION_KEY as _SK
_SK.reset(tok)
except Exception:
logger.debug("per-turn _SESSION_KEY reset failed", exc_info=True)
tok = tokens.get("approval")
if tok is not None:
try:
from tools.approval import reset_current_session_key
reset_current_session_key(tok)
except Exception:
logger.debug("per-turn approval session-key reset failed", exc_info=True)
@contextlib.contextmanager
def _bind_turn_session_identity(session_id: str):
"""Context-manager form of the per-turn session-identity binding.
The ``_run_agent_streaming`` worker uses the explicit ``_set``/``_reset``
pair directly because its single ``try/finally`` already spans the whole
turn (~2k lines) and the binding must cover every mid-turn background
spawn; this wrapper is the canonical single-call API for other callers and
for tests, and shares the exact same code path.
"""
tokens = _set_turn_session_identity(session_id)
try:
yield
finally:
_reset_turn_session_identity(tokens)
def _format_process_notification(evt: dict) -> str:
"""Format a completed background process notification for agent input."""
if not isinstance(evt, dict):
@@ -4226,6 +4328,56 @@ def _sse(handler, event, data):
handler.wfile.flush()
# ── SSE write deadline (Defect A: per-connection thread exhaustion) ─────────
# server.py runs QuietHTTPServer(ThreadingHTTPServer): one OS thread per
# connection, no pool cap (request_queue_size=64). Every SSE endpoint holds
# its thread for the connection's whole lifetime. If a tab is slow or
# backgrounded its TCP receive window fills; the next handler.wfile.write()/
# flush() then blocks *indefinitely* (sockets have no write timeout by
# default). That thread is pinned forever — it never reaches its
# `finally: unsubscribe`, so the SessionChannel reaper can never reclaim the
# channel either. N such tabs * M sessions pile threads up until new
# requests queue past request_queue_size and the UI shows "streaming
# pending".
#
# Fix: arm a socket-level timeout on the connection. A genuinely healthy
# keepalive/event write completes in well under a millisecond, so a
# multi-second deadline never trips for a live tab; only a backpressured
# (stuck) socket blocks past it. When it trips, the write raises
# socket.timeout — which on Python 3.10+ *is* TimeoutError, already a member
# of api.routes._CLIENT_DISCONNECT_ERRORS — so each SSE handler's existing
# `except _CLIENT_DISCONNECT_ERRORS:` breaks the loop, `finally` drops the
# subscriber, the browser's EventSource auto-reconnects, and the OS thread
# is released. SessionChannel already supports reconnect + offline buffer,
# so no events are lost for a tab that comes back. Operators behind unusual
# proxies can tune the deadline without code changes.
try:
_raw_deadline = os.getenv("HERMES_WEBUI_SSE_WRITE_DEADLINE") or os.getenv("HERMES_SSE_WRITE_DEADLINE")
SSE_WRITE_DEADLINE_SECONDS = float(_raw_deadline or "20.0")
except (TypeError, ValueError):
SSE_WRITE_DEADLINE_SECONDS = 20.0
if SSE_WRITE_DEADLINE_SECONDS <= 0:
SSE_WRITE_DEADLINE_SECONDS = 20.0
def _sse_set_write_deadline(handler, seconds=None):
"""Best-effort: arm a socket write deadline on an SSE handler.
Call once, right after end_headers(), in every long-lived SSE endpoint.
Never raises — an unusual/missing transport just keeps the pre-fix
(no-deadline) behaviour for that single connection rather than breaking
the stream setup.
"""
if seconds is None:
seconds = SSE_WRITE_DEADLINE_SECONDS
try:
conn = getattr(handler, "connection", None)
if conn is not None and hasattr(conn, "settimeout"):
conn.settimeout(seconds)
except Exception:
logger.debug("Failed to arm SSE write deadline", exc_info=True)
def _materialize_pending_user_turn_before_error(session) -> bool:
"""Persist the pending user prompt before clearing runtime stream state.
@@ -4970,6 +5122,11 @@ def _run_agent_streaming(
if _is_fallback_notice:
put('warning', {'type': 'fallback', 'message': _message})
# xsession wakeup misroute root fix (Option 1): pre-init so the outer
# finally can always reset even if an exception fires before the bind.
# Placed ABOVE the _checkpoint_stop cluster so that cluster stays adjacent
# to the `try:` (preserves the Issue #765 static-locator invariant).
_turn_session_identity_tokens = None
# Initialised here (before any code that may raise) so the outer `finally`
# block can safely check `if _checkpoint_stop is not None` even when an
# exception fires before the checkpoint thread is created (Issue #765).
@@ -4977,6 +5134,12 @@ def _run_agent_streaming(
_ckpt_thread = None
_agent_lock = None
try:
# Bind THIS turn's session identity to the worker thread/context BEFORE
# any agent work (so every mid-turn notify_on_complete background spawn
# captures THIS session, not a concurrent turn's process-global env).
# Co-located with the existing env-restore lifecycle: set here, reset
# in the outer finally next to _clear_thread_env().
_turn_session_identity_tokens = _set_turn_session_identity(session_id)
s = get_session(session_id)
update_active_run(stream_id, phase="running", session_id=session_id)
s.workspace = str(Path(workspace).expanduser().resolve())
@@ -5052,7 +5215,15 @@ def _run_agent_streaming(
_profile_home,
)
_set_thread_env(**_thread_env)
# Prewarm skill-tool imports *before* acquiring the lock so that
# process_complete agent-wakeup wiring (ours-original, Option B): bind
# this session's HERMES_SESSION_KEY to its WebUI session_id so the
# drain thread can route notify_on_complete events back to the right
# SSE channel / server-side wakeup.
try:
from api.background_process import register_process_session
register_process_session(session_id, session_id)
except Exception:
logger.debug("register_process_session failed", exc_info=True)
# first-time module initialisation (which can be slow) does not
# block other concurrent sessions waiting on _ENV_LOCK (#2024).
_prewarm_skill_tool_modules()
@@ -5067,6 +5238,7 @@ def _run_agent_streaming(
old_session_key = os.environ.get('HERMES_SESSION_KEY')
old_session_id = os.environ.get('HERMES_SESSION_ID')
old_session_platform = os.environ.get('HERMES_SESSION_PLATFORM')
old_session_chat_id = os.environ.get('HERMES_SESSION_CHAT_ID')
old_hermes_home = os.environ.get('HERMES_HOME')
os.environ.update(_profile_runtime_env)
os.environ['TERMINAL_CWD'] = str(s.workspace)
@@ -5074,6 +5246,9 @@ def _run_agent_streaming(
os.environ['HERMES_SESSION_KEY'] = session_id
os.environ['HERMES_SESSION_ID'] = session_id
os.environ['HERMES_SESSION_PLATFORM'] = 'webui'
# process_complete wiring (ours-original, Option B): see
# _build_agent_thread_env above.
os.environ['HERMES_SESSION_CHAT_ID'] = str(session_id)
if _profile_home:
os.environ['HERMES_HOME'] = _profile_home
# Patch module-level caches to match the active profile.
@@ -7361,6 +7536,8 @@ def _run_agent_streaming(
else: os.environ['HERMES_SESSION_ID'] = old_session_id
if old_session_platform is None: os.environ.pop('HERMES_SESSION_PLATFORM', None)
else: os.environ['HERMES_SESSION_PLATFORM'] = old_session_platform
if old_session_chat_id is None: os.environ.pop('HERMES_SESSION_CHAT_ID', None)
else: os.environ['HERMES_SESSION_CHAT_ID'] = old_session_chat_id
if old_hermes_home is None: os.environ.pop('HERMES_HOME', None)
else: os.environ['HERMES_HOME'] = old_hermes_home
@@ -7588,6 +7765,12 @@ def _run_agent_streaming(
update_active_run(stream_id, phase="finalizing")
_last_resort_sync_from_core(s, stream_id, _agent_lock)
_clear_thread_env() # TD1: always clear thread-local context
# xsession wakeup misroute root fix (Option 1): restore the per-turn
# session-identity context-locals (reset-token semantics). MUST run on
# every exit path so a reused thread-pool worker leaks no identity and
# CLI/cron env fallback resumes — same lifecycle slot as the env
# restore above.
_reset_turn_session_identity(_turn_session_identity_tokens)
with STREAMS_LOCK:
STREAMS.pop(stream_id, None)
CANCEL_FLAGS.pop(stream_id, None)
@@ -7608,6 +7791,36 @@ def _run_agent_streaming(
# the next stream can read it, breaking the goal-continuation
# chain. Stage-326 critical fix per Opus advisor review.
# ── Defer-path fix: turn-teardown idle-hook ────────────────────────
# The session has just transitioned active→idle: unregister_active_run
# above cleared this stream's ACTIVE_RUNS row (under ACTIVE_RUNS_LOCK,
# independent of STREAMS_LOCK), so _session_has_active_turn() is now
# False for this session unless a *different* stream is still active
# (cancel/reconnect — drain_deferred_wakeups_for_session guards on
# that and leaves the marker for the later teardown). A FAST
# background task that completed while this turn was tearing down was
# deferred by api/background_process._process_one (it could not start
# a turn → would 409) and its wakeup_prompt persisted in
# DEFERRED_PROCESS_WAKEUPS. For an autonomous agent there is no next
# user turn, so the PR #2279 next-turn drain never runs; without this
# hook the deferred wakeup is lost forever (the Test B failure). This
# makes the busy-at-completion case symmetric with the idle case:
# idle now → fire now (Option Z idle branch); busy now → fire here at
# turn-end. claim_deferred_wakeups pops atomically, so this is
# idempotent with the next-turn drain (no double-fire) and the wakeup
# turn's own teardown finds nothing claimed (no wakeup loop). The
# drain spawns its own daemon thread, so teardown never blocks.
try:
from api.background_process import drain_deferred_wakeups_for_session
drain_deferred_wakeups_for_session(session_id)
except Exception:
logger.debug(
"turn-teardown deferred-wakeup drain failed for session %s",
session_id,
exc_info=True,
)
# ============================================================
# SECTION: HTTP Request Handler
# do_GET: read-only API endpoints + SSE stream + static HTML

View File

@@ -586,6 +586,27 @@ def main() -> None:
except Exception as e:
print(f'[!!] WARNING: Gateway watcher failed to start: {e}', flush=True)
# Start the bg_task_complete drain thread for terminal(notify_on_complete=true)
# agent wakeup. Reads tools.process_registry.completion_queue and emits SSE
# bg_task_complete events (canonical name; legacy process_complete alias is
# still emitted for back-compat) to the matching session's stream.
try:
from api.background_process import start_drain_thread
if start_drain_thread():
print('[ok] bg_task_complete drain thread started', flush=True)
except Exception as e:
print(f'[!!] WARNING: bg_task_complete drain failed to start: {e}', flush=True)
# Start the SessionChannel reaper for the persistent per-session SSE
# endpoint (/api/session/stream). Runs every 60s, collects channels with
# no subscribers past the grace period or past the idle TTL cap.
try:
from api.background_process import start_session_channel_reaper
if start_session_channel_reaper():
print('[ok] SessionChannel reaper thread started', flush=True)
except Exception as e:
print(f'[!!] WARNING: SessionChannel reaper failed to start: {e}', flush=True)
# Load WebUI dashboard plugins
try:
from api.plugins import load_plugins
@@ -633,6 +654,20 @@ def main() -> None:
drain_all_on_shutdown()
except Exception:
logger.debug("Failed to drain lifecycle on shutdown", exc_info=True)
# Stop bg_task_complete drain + SessionChannel reaper (ours-original).
# The drain thread emits the canonical ``bg_task_complete`` event
# (with ``process_complete`` kept as a temporary backward-compat
# alias for older clients — see start_drain_thread comment above).
try:
from api.background_process import stop_drain_thread
stop_drain_thread()
except Exception:
logger.debug("Failed to stop bg_task_complete drain thread during shutdown", exc_info=True)
try:
from api.background_process import stop_session_channel_reaper
stop_session_channel_reaper()
except Exception:
logger.debug("Failed to stop SessionChannel reaper during shutdown", exc_info=True)
if __name__ == '__main__':
main()

View File

@@ -4,6 +4,52 @@ function _markSessionViewed(sid, messageCount) {
_setSessionViewedCount(sid, next);
}
function _apiUrl(path) {
return new URL(path, document.baseURI || location.href).href;
}
// Module-scope dedupe ring buffer for bg_task_complete events. Shared between
// the in-turn STREAMS path (per-turn EventSource inside the chat-stream wirer)
// and the persistent session-scoped path (/api/session/stream), so the
// frontend never double-fires a toast or ack for the same (session_id,
// event_id) regardless of which channel delivered it first. (Option X)
//
// Keyed by `${session_id}|${event_id}` → expiry timestamp (ms since epoch).
// Bounded by a 60-second TTL plus a 256-entry soft cap with insertion-order
// eviction on overflow. Events without `event_id` are ignored by the caller
// (the server contract guarantees `event_id` on every completion emit).
const _BG_TASK_COMPLETE_TTL_MS = 60000;
const _BG_TASK_COMPLETE_CAP = 256;
const _bgTaskCompleteSeenIds = new Map();
function _bgTaskCompleteRingBufferAdd(sid, evt_id) {
// Missing key → treat as "seen/skip" (return true). The sole caller already
// guards with `if (!evt_id) return;` before invoking this, so this branch is
// defensive: returning true (skip) rather than false (proceed) means a
// future call site that forgets that guard drops the un-keyable event
// instead of processing a completion with no dedupe key.
if (!sid || !evt_id) return true;
const key = sid + '|' + evt_id;
const now = Date.now();
// Lazy purge: walk insertion-order; drop any entry whose expiry has passed.
// Map iteration is insertion-order so this also surfaces the oldest entries
// first when we need to evict for the soft cap below.
for (const [k, exp] of _bgTaskCompleteSeenIds) {
if (exp <= now) {
_bgTaskCompleteSeenIds.delete(k);
}
}
if (_bgTaskCompleteSeenIds.has(key)) return true; // duplicate
_bgTaskCompleteSeenIds.set(key, now + _BG_TASK_COMPLETE_TTL_MS);
// Soft cap: insertion-order eviction.
while (_bgTaskCompleteSeenIds.size > _BG_TASK_COMPLETE_CAP) {
const firstKey = _bgTaskCompleteSeenIds.keys().next().value;
if (firstKey === undefined) break;
_bgTaskCompleteSeenIds.delete(firstKey);
}
return false;
}
function _isDocumentVisibleAndFocused() {
if(typeof document!=='undefined' && document.visibilityState && document.visibilityState!=='visible') return false;
if(typeof document!=='undefined' && typeof document.hasFocus==='function' && !document.hasFocus()) return false;
@@ -2971,6 +3017,29 @@ function attachLiveStream(activeSid, streamId, uploaded=[], options={}){
}catch(_){}
});
// bg_task_complete: terminal(notify_on_complete=true) background process
// exited. Option Z PIVOT: the agent wakeup is started SERVER-SIDE by the
// drain thread (api/background_process._process_one →
// routes.start_session_turn) with NO browser round-trip — so the
// closed-tab case works (parity with CLI/Telegram). The browser does NOT
// re-POST /api/chat/start anymore. This SSE event is pure LIVE-VIEW: if
// a tab is open the server-initiated turn streams live via the normal
// /api/chat/stream EventSource; if the tab is closed the turn still runs
// server-side and persists to the session store.
//
// Idempotency: dedupe by (session_id, event_id) via a Map+TTL ring
// buffer (`_bgTaskCompleteRingBufferAdd`).
//
// Option X: this handler is the in-turn (STREAMS-bound) path. The server
// dual-emits to the persistent session-scoped channel too — the
// `_handleBgTaskCompleteEvent` function below is shared between both
// paths (dedupe only; the wakeup itself is server-side).
source.addEventListener('bg_task_complete',e=>{
if(typeof _handleBgTaskCompleteEvent==='function'){
_handleBgTaskCompleteEvent(e, activeSid, {source:'stream'});
}
});
source.addEventListener('done',e=>{
if(_streamFinalized) return;
if(_bailOutOfTerminalEventsFromStaleStream(source)) return;
@@ -4056,6 +4125,194 @@ function stopApprovalPolling() {
_approvalPollingSessionId = null;
}
// ── Session-scoped SSE stream (Option X) ──────────────────────────────────
// Long-lived EventSource bound to /api/session/stream?session_id=<sid>.
// Lives across agent turns (unlike the per-turn /api/chat/stream which is
// torn down at end-of-turn). Carries bg_task_complete events fired while no
// turn is active — the architectural fix for the notify_on_complete wakeup
// gap that #2242 + #2279 papered over.
//
// Lifecycle: opened on session mount (loadSession / newSession), closed on
// session switch / unmount. The browser closes it implicitly on tab close
// (server detects disconnect via the SSE read-loop and unsubscribes).
let _sessionEventSource = null;
let _sessionStreamSessionId = null;
let _sessionStreamReconnectTimer = null;
function startSessionStream(sid) {
if (!sid) return;
// Already on this session? No-op (loadSession is a no-op when re-selecting
// the same session; this defends against external re-callers).
if (_sessionStreamSessionId === sid && _sessionEventSource) return;
stopSessionStream();
_sessionStreamSessionId = sid;
try {
const es = new EventSource(_apiUrl('api/session/stream?session_id=' + encodeURIComponent(sid)));
_sessionEventSource = es;
es.addEventListener('initial', () => { /* connection confirmed */ });
es.addEventListener('bg_task_complete', e => {
// Shared handler — same dedupe set as the in-turn STREAMS path.
if (typeof _handleBgTaskCompleteEvent === 'function') {
_handleBgTaskCompleteEvent(e, sid, {source: 'session'});
}
});
// ── Defect B: live-view of server-initiated (Option Z) turns ──────────
// The drain thread starts the wakeup turn server-side and the server
// fans a `server_turn_started` {stream_id} frame onto this per-session
// channel. No browser POSTed /api/chat/start, so nothing is attached to
// that STREAMS[stream_id] yet. Attach the EXISTING chat-stream renderer
// (attachLiveStream — the exact path /api/chat/start uses) to the
// server-created stream so the open tab renders the turn live. Reuses
// the one renderer; does NOT hand-roll a second one.
es.addEventListener('server_turn_started', e => {
try {
const d = JSON.parse(e.data || '{}');
const evSid = d.session_id || sid;
const streamId = String(d.stream_id || '');
if (!streamId || evSid !== sid) return;
// `recovered` marks an on-subscribe replay from the server: the tab
// (re)connected to /api/session/stream AFTER the original
// fire-and-forget server_turn_started had already been broadcast, so
// the live stream is mid-flight. Attach via the reconnecting (replay)
// path so the renderer rebuilds from the run journal instead of
// expecting token 0 (which would render a truncated turn). A fresh
// (non-recovered) frame still attaches from the first token.
const recovered = !!d.recovered;
// Only drive the renderer when this session is the one on screen.
const isCurrent = (typeof _isSessionCurrentPane === 'function')
? _isSessionCurrentPane(sid)
: (S.session && S.session.session_id === sid);
if (!isCurrent) return;
// A turn is already rendering in this tab (user-initiated, or we
// already attached to this very stream). attachLiveStream is
// idempotent per (sid, streamId); bail if we're already on it.
if (S.activeStreamId === streamId) return;
const existingLive = (typeof LIVE_STREAMS !== 'undefined') ? LIVE_STREAMS[sid] : null;
if (existingLive && existingLive.streamId === streamId) return;
// Mirror the loadSession reattach setup. For a fresh frame the turn
// renders from its first token; for a recovered (replay) frame
// attachLiveStream reconstructs the in-progress stream.
S.busy = true;
S.activeStreamId = streamId;
if (S.session && S.session.session_id === sid) S.session.active_stream_id = streamId;
if (typeof updateSendBtn === 'function') updateSendBtn();
if (typeof setComposerStatus === 'function') setComposerStatus('');
if (typeof syncTopbar === 'function') syncTopbar();
if (typeof appendThinking === 'function') appendThinking();
if (typeof startApprovalPolling === 'function') startApprovalPolling(sid);
if (typeof startClarifyPolling === 'function') startClarifyPolling(sid);
if (typeof attachLiveStream === 'function') {
attachLiveStream(
sid, streamId,
(S.session && S.session.pending_attachments) || [],
recovered ? {reconnecting: true} : {},
);
}
if (typeof renderSessionList === 'function') void renderSessionList();
} catch (_) {}
});
es.onerror = () => {
// Browser already auto-reconnects EventSource on most transient
// failures. We only intervene if the connection has been closed for
// good (readyState === 2) — schedule a one-shot re-open after 5s.
if (es.readyState === 2 && _sessionStreamSessionId === sid) {
if (_sessionStreamReconnectTimer) clearTimeout(_sessionStreamReconnectTimer);
// The CLOSED EventSource (readyState === 2) will never reconnect on
// its own, and startSessionStream's top guard
// (`_sessionStreamSessionId === sid && _sessionEventSource`) would
// short-circuit the re-open while this dead object is still pinned.
// Drop our reference (and close it for good measure) so the timer's
// startSessionStream() reaches stopSessionStream() and builds a FRESH
// EventSource instead of reusing the closed one. Only clear if `es`
// is still the active source — a newer connection may have replaced
// it in the interim (stale onerror from a superseded stream), in
// which case we must not stomp the live one.
if (_sessionEventSource === es) {
try { es.close(); } catch (_) {}
_sessionEventSource = null;
}
_sessionStreamReconnectTimer = setTimeout(() => {
_sessionStreamReconnectTimer = null;
if (_sessionStreamSessionId === sid) startSessionStream(sid);
}, 5000);
}
};
} catch(_) {
// EventSource ctor threw — silently disabled; the in-turn STREAMS path
// still works for events that fire during an active turn.
_sessionEventSource = null;
}
}
function stopSessionStream() {
if (_sessionStreamReconnectTimer) { clearTimeout(_sessionStreamReconnectTimer); _sessionStreamReconnectTimer = null; }
if (_sessionEventSource) {
try { _sessionEventSource.close(); } catch(_){}
_sessionEventSource = null;
}
_sessionStreamSessionId = null;
}
// Shared bg_task_complete handler — invoked from BOTH the in-turn STREAMS
// channel (legacy path, still kept as defense-in-depth) AND the session-
// scoped channel (Option X primary path). Dedupes by (session_id, event_id)
// via the Map+TTL ring buffer declared at the top of this module.
// Events without `event_id` are ignored — the server contract guarantees one
// on every completion emit, so a missing key signals a malformed or replayed
// payload we should not surface or ack.
// PR (c) UX surface: post-dedupe the handler marks the session viewed (when
// the session pane is current and the doc is visible+focused), then runs the
// T4 drop-when-focused gate; only out-of-focus or off-pane completions spawn
// a toast. The diagnostic ack POST still fires for both focused and
// unfocused viewers so the server receives the delivery/cleanup signal;
// the focus gate suppresses UI noise only.
function _handleBgTaskCompleteEvent(e, expectedSid, opts) {
try {
const d = JSON.parse(e.data || '{}');
const sid = d.session_id || expectedSid;
if (sid !== expectedSid) return;
const evt_id = d.event_id ? String(d.event_id) : '';
if (!evt_id) return; // server contract requires event_id; ignore otherwise
if (_bgTaskCompleteRingBufferAdd(sid, evt_id)) return; // duplicate
const pid = String(d.task_id || '');
const _viewed = typeof _isSessionActivelyViewed === 'function' && _isSessionActivelyViewed(sid);
if (_viewed) {
try { _markSessionViewed(sid, (S&&S.session&&S.session.session_id===sid)?(S.session.message_count??(S.messages&&S.messages.length)??0):0); } catch(_){}
try { if(typeof _clearSessionCompletionUnread==='function') _clearSessionCompletionUnread(sid); } catch(_){}
} else {
// T4 drop-when-focused: suppress toast only; ack below still fires.
try {
const tid = (d.task_id || '').slice(0, 8) || '?';
const tail = d.summary ? `: ${String(d.summary).slice(0, 80)}` : '';
showToast(`Task ${tid} done${tail}`, 2600);
} catch (_) {}
}
// Fire-and-forget ack (diagnostic only — Option Z made this a no-op for
// state. The agent wakeup is now started SERVER-SIDE by the drain thread
// in api/background_process._process_one → start_session_turn; the
// browser is no longer in the wakeup path at all.)
try {
fetch(_apiUrl('api/bg-task-complete-ack'), {
method: 'POST',
headers: {'Content-Type': 'application/json'},
credentials: 'include',
body: JSON.stringify({session_id: sid, task_id: pid, event_id: evt_id}),
}).catch(() => {});
} catch(_) {}
// Option Z PIVOT: the browser NO LONGER re-POSTs the chat-start endpoint
// to wake the agent. Server-side wakeup is the PRIMARY mechanism — the
// drain thread starts the turn directly (no tab required), so the
// closed-tab case works (parity with CLI/Telegram). The per-session SSE
// channel this handler is wired into is DEMOTED to a pure live-view
// layer: if a tab is open the server-initiated turn streams live via the
// existing chat-stream EventSource; if the tab is closed the turn still
// runs server-side and the result is persisted to the session store.
// The user-facing toast + drop-when-focused gate land in PR (c).
} catch(_) {}
}
// ── Clarify polling ──
let _clarifyPollTimer = null;
let _clarifyHideTimer = null;

View File

@@ -666,6 +666,7 @@ async function newSession(flash, options={}){
if(flash)S.session._flash=true;
try{localStorage.setItem('hermes-webui-session',S.session.session_id);}catch(_){}
_setActiveSessionUrl(S.session.session_id);
if(typeof startSessionStream==='function') startSessionStream(S.session.session_id);
_setSessionViewedCount(S.session.session_id, S.session.message_count || 0);
// Sync chat-header dropdown to the session's model/provider so the UI reflects
// the default route the server actually used (#872). Compare provider state too:
@@ -731,6 +732,25 @@ async function newSession(flash, options={}){
}
}
// #2971 (Greptile P1 r3377162160): loadSession() tears down the live
// per-session SSE at the top via stopSessionStream() (line ~754), but only the
// success path re-arms it via startSessionStream() (line ~875). Every
// early-return exit (fetch error, auth-redirect undefined) — and the
// same-session no-op guard, which returns BEFORE the teardown — could leave
// the session the user actually remains on with a permanently null
// EventSource, silently dropping bg_task_complete delivery until a full page
// reload or a forced loadSession. This helper re-arms the stream for whatever
// session is currently on screen (S.session). startSessionStream() is
// idempotent — it no-ops when already live for that sid (top guard
// `_sessionStreamSessionId === sid && _sessionEventSource`) — so this never
// double-arms the success path, which arms the *newly assigned* S.session
// only after this point.
function _rearmActiveSessionStream(){
if(typeof startSessionStream!=='function') return;
const activeSid = S.session ? S.session.session_id : null;
if(activeSid) startSessionStream(activeSid);
}
async function loadSession(sid){
const opts = arguments[1] || {};
if(!opts.skipLineageResolve && typeof _resolveSessionIdFromSidebarLineage==='function'){
@@ -747,11 +767,15 @@ async function loadSession(sid){
// Do not no-op a same-session click while another load is in flight: the
// previous transcript may already have been cleared for the pending switch.
// Static force-reload invariant: if(currentSid===sid && !forceReload) return;
// #2971: idempotent re-arm before the no-op guard revives a stream a prior
// failed loadSession killed; no-ops on real switches.
_rearmActiveSessionStream();
if(currentSid===sid && !forceReload && !_loadingSessionId) return;
// Mark this session as the in-flight load. Subsequent loadSession() calls
// will overwrite this; stale awaits use the mismatch to bail out (#1060).
_loadingSessionId = sid;
stopApprovalPolling();hideApprovalCard(forceReload);
if(typeof stopSessionStream==='function') stopSessionStream();
_yoloEnabled=false;_updateYoloPill();
if(typeof stopClarifyPolling==='function') stopClarifyPolling();
if(typeof hideClarifyCard==='function') hideClarifyCard(forceReload, forceReload?'external-refresh':'dismissed');
@@ -841,7 +865,32 @@ async function loadSession(sid){
}
}
_clearSameSessionForceReloadHint(sid);
// Capture whether this failure self-healed away the current session (a
// 404 on the *current* session whose sidecar was deleted server-side).
// In that case there is no live session left to stream for, so we must
// NOT restart — doing so would spin the SSE reconnect loop against a dead
// session_id.
const _selfHealedCurrent = (e.status===404) && (currentSid===sid);
if (_loadingSessionId === sid) _loadingSessionId = null;
// The session stream was stopped unconditionally at the top of this load
// (mirroring stopApprovalPolling). On the happy path it's restarted ~120
// lines below, but this failure exit never reaches that point — leaving
// the session still on screen permanently silenced. bg_task_complete
// events (the new feature's primary delivery path) would be dropped until
// the user explicitly navigates to a session again. Restart the stream for
// the session that remains on screen. Skip when a newer load is already in
// flight (_loadingSessionId !== null after the reset above): that load owns
// the stream and starts its own. Skip the self-healed-current case (no live
// session to stream).
// #2971: this fetch-error path keeps its bespoke guarded restart (rather
// than the shared _rearmActiveSessionStream helper used on the other
// early-returns) because only here can the current session have just
// self-healed away — re-arming a 404'd/deleted session_id would spin the
// SSE reconnect loop against a dead session.
if (currentSid && !_selfHealedCurrent && _loadingSessionId === null
&& typeof startSessionStream === 'function') {
startSessionStream(currentSid);
}
return;
}
// Guard: api() may have redirected (401) and returned undefined; in that case
@@ -849,10 +898,21 @@ async function loadSession(sid){
if (!data) {
_clearSameSessionForceReloadHint(sid);
if (_loadingSessionId === sid) _loadingSessionId = null;
// #2971: re-arm the still-displayed session's stream (defensive — harmless
// if the 401 redirect is already tearing the page down). Idempotent.
_rearmActiveSessionStream();
return;
}
// Stale response? A newer loadSession() call has already started (#1060).
if (_loadingSessionId !== sid) return;
if (_loadingSessionId !== sid) {
// #2971: a newer in-flight load owns the final stream arming, but until it
// assigns S.session and reaches startSessionStream() the currently-shown
// session must not be left stream-dead by our top-of-function teardown.
// Re-arm the genuinely-displayed S.session (idempotent — no-ops once the
// newer load arms its own sid).
_rearmActiveSessionStream();
return;
}
S.session=data.session;
if(typeof _hydrateTodosFromSession==='function') _hydrateTodosFromSession(S.session);
S.session._modelResolutionDeferred=true;
@@ -872,6 +932,7 @@ async function loadSession(sid){
_clearSessionCompletionUnread(S.session.session_id);
try{localStorage.setItem('hermes-webui-session',S.session.session_id);}catch(_){}
_setActiveSessionUrl(S.session.session_id);
if(typeof startSessionStream==='function') startSessionStream(S.session.session_id);
const activeStreamId=S.session.active_stream_id||null;
// If the server says the session is idle, discard any browser-side inflight

96
tests/_wakeup_helpers.py Normal file
View File

@@ -0,0 +1,96 @@
"""Shared test helpers for the server-side wakeup test suites.
Consolidates the ``_install_fake_start_session_turn`` / ``_wait_for_wakeup``
pair that ``test_session_channel_option_x.py`` and ``test_wakeup_defer_race.py``
both need so the two suites can't drift. Per Copilot review on PR #2971
(r3305700944).
"""
from __future__ import annotations
import queue
import sys
import threading
import types
class FakeProcessRegistry:
"""Minimal stand-in for ``tools.process_registry.process_registry``.
Consolidated from the verbatim ``_FakeProcessRegistry`` copies that lived
in ``test_bg_task_complete_wakeup.py``, ``test_bg_task_complete_throttle.py``
and ``test_bg_task_complete_ab_coexistence.py`` (Greptile review on PR #2979).
Keeping one definition prevents the three suites from drifting to subtly
different stub shapes.
"""
def __init__(self):
self._lock = threading.Lock()
self._completion_consumed: set[str] = set()
self.completion_queue: queue.Queue = queue.Queue()
self._procs: dict[str, types.SimpleNamespace] = {}
def register(self, process_id: str, session_key: str) -> None:
self._procs[process_id] = types.SimpleNamespace(session_key=session_key)
def get(self, process_id: str):
return self._procs.get(process_id)
def is_completion_consumed(self, process_id: str) -> bool:
with self._lock:
return process_id in self._completion_consumed
def install_fake_registry(monkeypatch, fake) -> None:
"""Inject ``fake`` under ``tools.process_registry`` for the test.
IMPORTANT (rebase isolation): uses ONLY ``monkeypatch.setitem`` so both
``sys.modules`` entries are restored to their real/absent state on
teardown. A prior implementation used ``sys.modules.setdefault("tools", …)``
which is an UNTRACKED mutation — when the real ``tools`` package was not yet
imported it permanently leaked a non-package fake ``tools`` into
``sys.modules``, breaking any later test doing
``from tools.process_registry import …``. Both setitem calls below are
monkeypatch-tracked: on teardown each key is restored to its prior value,
or deleted if it was absent — no leak.
"""
mod = types.ModuleType("tools.process_registry")
mod.process_registry = fake # type: ignore[attr-defined]
tools_mod = types.ModuleType("tools")
tools_mod.process_registry = mod # type: ignore[attr-defined]
monkeypatch.setitem(sys.modules, "tools", tools_mod)
monkeypatch.setitem(sys.modules, "tools.process_registry", mod)
def install_fake_start_session_turn(monkeypatch, *, status: int = 200):
"""Patch ``api.routes.start_session_turn`` to record calls instead of
running a real agent turn.
The drain helper does ``from api.routes import start_session_turn``
inside a daemon thread, so patching the attribute on the ``api.routes``
module is what the thread resolves at call time.
Returns a ``holder`` dict with ``calls`` (list of recorded call kwargs)
and ``event`` (a ``threading.Event`` set on first call) — pair it with
``wait_for_wakeup`` below.
"""
import api.routes as _routes
holder = {"calls": [], "event": threading.Event()}
def _fake(session_id, message, *, source="process_wakeup"):
holder["calls"].append(
{"session_id": session_id, "message": message, "source": source}
)
holder["event"].set()
return {"stream_id": "fake-stream", "session_id": session_id, "_status": status}
monkeypatch.setattr(_routes, "start_session_turn", _fake, raising=True)
return holder
def wait_for_wakeup(holder, timeout: float = 3.0) -> bool:
"""Block until the server-side wakeup runner thread recorded a call.
Returns True if the holder's event fired within ``timeout`` seconds.
"""
return holder["event"].wait(timeout=timeout)

View File

@@ -0,0 +1,166 @@
"""Deterministic BEFORE/AFTER repro for the wakeup model-resolve hang.
Simulates the proven thread-stack: a server-initiated Option-Z wakeup turn
reaches start_session_turn with a COLD provider catalog; the live rebuild's
per-provider probe (the Copilot token-exchange HTTPS call) is monkeypatched to
hang. We drive start_session_turn directly (no browser, no real agent) and
measure how long model-resolution takes.
BEFORE (legacy behaviour: HERMES_WEBUI_MODELS_REBUILD_BUDGET=0 AND
prefer_cache forced off): the wakeup blocks on the hung probe — exactly the
"stuck at resolve_model_provider" symptom.
AFTER (shipped behaviour): start_session_turn resolves with
prefer_cached_catalog=True (never touches the live rebuild) AND the rebuild is
budget-bounded as defense-in-depth -- the wakeup turn starts in well under a
second using the persisted session model.
Run (from the repo root, with the Hermes Agent venv python -- any interpreter
that can import this repo's ``api`` package works):
python tests/manual/repro_wakeup_hang.py
Exits 0 on PASS, 1 on FAIL. Not collected by pytest (see
tests/manual/conftest.py); this is an operator-run reproduction, not a test.
"""
from __future__ import annotations
import os as _os
import sys as _sys
# Repo root is two levels up: <repo>/tests/manual/repro_wakeup_hang.py
_REPO_ROOT = _os.path.dirname(
_os.path.dirname(_os.path.dirname(_os.path.abspath(__file__)))
)
if _REPO_ROOT not in _sys.path:
_sys.path.insert(0, _REPO_ROOT)
import sys
import time
HANG_SECONDS = 30.0 # stand-in for an unreachable Copilot endpoint
def _install_fakes():
import api.config as cfg
import api.routes as routes
cfg.invalidate_models_cache()
# Inject the hang at the cold-rebuild seam. This is faithful to the proven
# root cause (the live per-provider rebuild — which on the real host does
# the Copilot token-exchange HTTPS call — is what blocks) and is
# deterministic regardless of which providers the isolated HERMES_HOME has
# configured (an isolated home with no providers can't reach the real
# _read_live_provider_model_ids path; precedent t_9f0184cf).
def _hung_rebuild(_builder):
time.sleep(HANG_SECONDS)
return {
"active_provider": "anthropic",
"default_model": "anthropic/claude-sonnet-4",
"configured_model_badges": {},
"groups": [],
}
cfg._invoke_models_rebuild = _hung_rebuild
# No disk cache → forces the cold rebuild branch.
cfg._load_models_cache_from_disk = lambda: None
fake_stream = "stream-repro-1"
captured = {}
def _fake_start(s, **kwargs):
captured["model"] = kwargs.get("model")
return {"stream_id": fake_stream, "session_id": s.session_id, "_status": 200}
class _FakeSession:
session_id = "sess-repro"
model = "anthropic/claude-sonnet-4"
model_provider = "anthropic"
routes._start_chat_stream_for_session = _fake_start
routes.get_session = lambda _s: _FakeSession()
routes._resolve_chat_workspace_with_recovery = lambda s, w: "/tmp/ws"
return routes, captured, fake_stream
def _time_call(label, fn, timeout):
import threading
box = {}
def _run():
box["resp"] = fn()
t = threading.Thread(target=_run, daemon=True)
t0 = time.monotonic()
t.start()
t.join(timeout=timeout)
elapsed = time.monotonic() - t0
if t.is_alive():
print(f" {label}: STUCK — still blocked after {elapsed:.1f}s "
f"(the wakeup turn never starts; matches the bug symptom)")
return None, elapsed, True
print(f" {label}: started in {elapsed:.3f}s "
f"(model={box.get('resp', {}).get('stream_id')!r})")
return box.get("resp"), elapsed, False
def main():
print("=== BEFORE (legacy: budget=0, prefer_cache forced OFF) ===")
import os
os.environ["HERMES_WEBUI_MODELS_REBUILD_BUDGET"] = "0"
# Reimport config so the budget constant picks up env=0.
for m in ("api.config", "api.routes"):
sys.modules.pop(m, None)
routes, captured, _ = _install_fakes()
# Force the legacy code path: bypass the prefer_cache short-circuit by
# calling resolve with prefer_cached_catalog=False, like the OLD code did.
_orig_resolve = routes._resolve_compatible_session_model_state
routes._resolve_compatible_session_model_state = (
lambda m, p, **_k: _orig_resolve(m, p, prefer_cached_catalog=False)
)
_, _, stuck_before = _time_call(
"wakeup chat/start",
lambda: routes.start_session_turn(
"sess-repro", "[IMPORTANT: bg done]", source="process_wakeup"
),
timeout=8.0,
)
print()
print("=== AFTER (shipped: prefer_cached_catalog=True + bounded rebuild) ===")
os.environ["HERMES_WEBUI_MODELS_REBUILD_BUDGET"] = "4"
for m in ("api.config", "api.routes"):
sys.modules.pop(m, None)
import api.config as cfg2 # noqa: F401
routes2, captured2, fake_stream = _install_fakes()
resp, elapsed_after, stuck_after = _time_call(
"wakeup chat/start",
lambda: routes2.start_session_turn(
"sess-repro", "[IMPORTANT: bg done]", source="process_wakeup"
),
timeout=8.0,
)
print()
print("=== RESULT ===")
ok = (
stuck_before is True
and stuck_after is False
and resp is not None
and resp.get("stream_id") == fake_stream
and captured2.get("model") == "anthropic/claude-sonnet-4"
and elapsed_after < 2.0
)
print(f" BEFORE stuck on hung probe: {stuck_before}")
print(f" AFTER started fast: {not stuck_after} "
f"({elapsed_after:.3f}s, persisted model={captured2.get('model')!r})")
print(f" REPRO {'PASS' if ok else 'FAIL'}")
return 0 if ok else 1
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,319 @@
"""Integration tests: the merged upstream PR #2279 (next-turn drain, A) +
our-original Option B SSE/server-side drain coexist without duplicating
wakeups for the same background process_id.
These tests verify the shared dedupe contract via the REAL merged upstream
key — process_registry._completion_consumed (checked by
process_registry.is_completion_consumed()):
- If B's drain fires first (proactive case), it marks the registry
consumed-marker so A's next-turn drain skips the same process_id.
- If A's (real merged #2279) drain fires first (SSE-disconnected case), it
marks the same registry consumed-marker so B's drain early-returns.
api.config.BG_TASK_COMPLETE_EVENTS_SEEN remains as B's own private
secondary dedupe (duplicate enqueue within this module) but is NOT the
cross-A/B contract — the real merged #2279 never writes it.
The two paths run in *different* hot paths (background thread vs. agent turn
start) but share process_registry._completion_consumed, so a wakeup can only
happen once.
"""
from __future__ import annotations
import threading
import pytest
# The fake process-registry stub + its installer were duplicated verbatim in
# three bg_task_complete suites; they now live once in tests/_wakeup_helpers.py
# (Greptile review on PR #2979). Import under the legacy local names so the
# rest of this module is unchanged. ``threading`` is still imported above for
# the _RenamedRegistry stub further down.
from tests._wakeup_helpers import FakeProcessRegistry as _FakeProcessRegistry
from tests._wakeup_helpers import install_fake_registry as _install_fake_registry
def _reset_cfg_state():
from api import config as _cfg
from api import background_process as bp
with _cfg.PROCESS_SESSION_INDEX_LOCK:
_cfg.PROCESS_SESSION_INDEX.clear()
_cfg.PENDING_BG_TASK_COMPLETIONS.clear()
_cfg.BG_TASK_COMPLETE_EVENTS_SEEN.clear()
with _cfg.STREAMS_LOCK:
_cfg.STREAMS.clear()
if hasattr(_cfg, "ACTIVE_RUNS"):
with _cfg.ACTIVE_RUNS_LOCK:
_cfg.ACTIVE_RUNS.clear()
if hasattr(bp, "_LAST_EMIT_TS"):
bp._LAST_EMIT_TS.clear()
if hasattr(bp, "_PENDING_EMIT_PAYLOADS"):
bp._PENDING_EMIT_PAYLOADS.clear()
if hasattr(bp, "_PENDING_EMIT_TIMERS"):
bp._PENDING_EMIT_TIMERS.clear()
def test_b_sse_first_then_a_drain_skips_same_process_id(monkeypatch):
"""B emits SSE for process_id=p1, then user types a new turn — A must skip p1."""
fake = _FakeProcessRegistry()
fake.register("p1", "sess-1")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
from api import background_process as bp
from api import streaming as st
from api import config as _cfg
# Map session_key -> WebUI session_id
bp.register_process_session("sess-1", "sess-1")
evt = {
"type": "completion",
"session_id": "p1",
"session_key": "sess-1",
"command": "sleep 1",
"exit_code": 0,
"output": "done",
}
# B path: process the event
bp._process_one(evt)
# B must have marked the (session, process) seen and registry-consumed
assert "p1" in _cfg.BG_TASK_COMPLETE_EVENTS_SEEN["sess-1"]
assert fake.is_completion_consumed("p1")
# Now simulate A's next-turn drain. Put a *new* event onto the queue for the
# same process_id (e.g. a kill_process race). A must skip because B already
# delivered.
fake.completion_queue.put(evt)
notifications = st._drain_webui_process_notifications("sess-1")
assert notifications == [], "A must NOT re-fire when B already woke the agent for p1"
def test_a_drain_first_marks_seen_so_b_would_skip(monkeypatch):
"""A (the REAL merged upstream #2279 next-turn drain) drains and wakes the
agent; later B's queue read of the same id is a no-op because the SHARED
upstream dedupe key (process_registry._completion_consumed) already
contains it.
Re-pointed for the rebase: the real merged #2279 drain dedupes ONLY via
process_registry.is_completion_consumed() — it does NOT populate
api.config.BG_TASK_COMPLETE_EVENTS_SEEN (that set is ours-original and
private to api.background_process). So the cross-A/B contract is the
registry consumed-marker, not BG_TASK_COMPLETE_EVENTS_SEEN.
"""
fake = _FakeProcessRegistry()
fake.register("p2", "sess-2")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
from api import background_process as bp
from api import streaming as st
from api import config as _cfg
bp.register_process_session("sess-2", "sess-2")
evt = {
"type": "completion",
"session_id": "p2",
"session_key": "sess-2",
"command": "echo hi",
"exit_code": 0,
"output": "hi",
}
# A path: queue carried over from a closed-tab session, drain at next turn
fake.completion_queue.put(evt)
notifications = st._drain_webui_process_notifications("sess-2")
assert len(notifications) == 1
assert "Background process p2 completed" in notifications[0]
# The REAL merged #2279 A-drain marks the SHARED upstream dedupe key
# (registry consumed-marker) — NOT our private BG_TASK_COMPLETE_EVENTS_SEEN.
assert fake.is_completion_consumed("p2")
assert "sess-2" not in _cfg.BG_TASK_COMPLETE_EVENTS_SEEN, (
"real upstream #2279 A-drain must NOT populate our private "
"BG_TASK_COMPLETE_EVENTS_SEEN set"
)
# Now if B's drain thread sees another spurious event for the same id
# (duplicate enqueue), _process_one must early-return on the SHARED
# registry consumed-marker that A set — no double wakeup.
bp._process_one(evt) # second time
assert fake.is_completion_consumed("p2")
# B early-returned on the shared key BEFORE reaching its own seen-set, so
# BG_TASK_COMPLETE_EVENTS_SEEN stays unpopulated for this session (proves
# the cross-A/B dedupe used the real upstream key, not ours).
assert "sess-2" not in _cfg.BG_TASK_COMPLETE_EVENTS_SEEN
# And no duplicate wakeup marker was queued by the second B pass.
assert "sess-2" not in _cfg.PENDING_BG_TASK_COMPLETIONS
def test_registry_completion_consumed_contract():
"""Copilot #2242 review #4 — fail CI LOUD if the agent ProcessRegistry
private cross-A/B dedupe surface is renamed/retyped upstream.
The WebUI B-drain has no public ``mark_completion_consumed`` to call, so
it reaches into ``ProcessRegistry._completion_consumed`` (under
``._lock``) to set the shared marker that the public
``is_completion_consumed`` reads. If a future upstream refactor renames
any of these, the double-wakeup bug would silently come back. This test
pins the contract so the rename breaks HERE (visibly) instead.
"""
pytest.importorskip("tools.process_registry", reason="hermes-agent not installed")
from tools.process_registry import ProcessRegistry
from api import background_process as bp
pr = ProcessRegistry()
for attr in bp._REGISTRY_CONSUMED_CONTRACT:
assert hasattr(pr, attr), (
f"ProcessRegistry.{attr} is gone — the WebUI cross-A/B wakeup "
f"dedupe coupling (Copilot #2242 #4) is broken. Either restore it "
f"or add a PUBLIC mark_completion_consumed() upstream and switch "
f"api/background_process._mark_registry_completion_consumed to it."
)
# Shape contract: the write target must be a set-like (supports .add) and
# the guard must be a usable context manager (supports `with`).
assert hasattr(pr._completion_consumed, "add"), (
"ProcessRegistry._completion_consumed is no longer a set-like "
"(.add gone) — cross-A/B wakeup dedupe write would fail."
)
assert hasattr(pr._lock, "__enter__") and hasattr(pr._lock, "__exit__"), (
"ProcessRegistry._lock is no longer a context manager — the guarded "
"marker write in _mark_registry_completion_consumed would fail."
)
assert callable(pr.is_completion_consumed), (
"ProcessRegistry.is_completion_consumed must stay a public method "
"(the cross-A/B dedupe READ side depends on it)."
)
# End-to-end: the public read sees what the guarded private write sets
# (the exact mechanism _mark_registry_completion_consumed relies on).
pid = "proc_contract_test"
assert pr.is_completion_consumed(pid) is False
with pr._lock:
pr._completion_consumed.add(pid)
assert pr.is_completion_consumed(pid) is True
def test_mark_registry_completion_consumed_fails_loud_on_rename(monkeypatch, caplog):
"""A renamed private attr must log ERROR (contract violation), NOT be
swallowed silently at DEBUG (the pre-Copilot-#4 behavior)."""
import logging
class _RenamedRegistry:
# Simulates an upstream rename: _completion_consumed -> _consumed_v2.
def __init__(self):
self._lock = threading.Lock()
self._consumed_v2: set[str] = set()
def is_completion_consumed(self, pid: str) -> bool:
return pid in self._consumed_v2
fake = _RenamedRegistry()
_install_fake_registry(monkeypatch, fake)
from api import background_process as bp
with caplog.at_level(logging.ERROR, logger="api.background_process"):
bp._mark_registry_completion_consumed("p-renamed")
assert any(
"coupling contract VIOLATED" in r.message and r.levelno >= logging.ERROR
for r in caplog.records
), "a renamed registry private attr must surface as an ERROR, not a silent DEBUG"
# The marker was NOT set (the bug it guards against), but it failed LOUD so
# CI / monitoring catches it instead of double-firing wakeups silently.
assert not fake.is_completion_consumed("p-renamed")
def test_emit_uses_new_event_name_with_trimmed_payload_and_event_id(monkeypatch):
"""T1 + T2 contract: emit is named ``bg_task_complete`` (canonical) AND
``process_complete`` (dual-emit shim until PR (b)); payload matches the
minimal shape ``{session_id, task_id, completed_at, summary?, event_id}``;
both emits carry the same payload + the same ``event_id``.
"""
fake = _FakeProcessRegistry()
fake.register("task-evt-1", "sess-evt-1")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
from api import background_process as bp
bp.register_process_session("sess-evt-1", "sess-evt-1")
# Capture every (event, data) tuple the emitter pushes to streams.
emits: list[tuple[str, dict]] = []
def _capture(session_id: str, event: str, data: dict) -> int:
emits.append((event, data))
return 1
monkeypatch.setattr(bp, "_emit_to_session_streams", _capture)
evt = {
"type": "completion",
"session_id": "task-evt-1",
"session_key": "sess-evt-1",
"command": "sleep 1",
"exit_code": 0,
"output": "done",
}
bp._process_one(evt)
# Dual-emit shim: both names fire, same payload, same event_id.
names = [e[0] for e in emits]
assert "bg_task_complete" in names, f"canonical event missing: {names}"
assert "process_complete" in names, f"dual-emit shim missing: {names}"
payloads = [e[1] for e in emits if e[0] in ("bg_task_complete", "process_complete")]
assert len({p["event_id"] for p in payloads}) == 1, (
"dual-emit must share a single event_id so consumers can dedupe"
)
payload = payloads[0]
# Minimal shape per maintainer (R2 §Q1).
expected_required = {"session_id", "task_id", "completed_at", "event_id"}
allowed = expected_required | {"summary"}
assert expected_required <= set(payload), f"missing required keys: {payload}"
assert set(payload) <= allowed, f"unexpected keys in trimmed payload: {payload}"
# Dropped keys must NOT be present.
for dropped in ("command", "exit_code", "type", "stdout_preview", "wakeup_prompt", "emitted_at", "process_id"):
assert dropped not in payload, f"{dropped!r} should be dropped by T1 trim"
# Field-rename invariants:
assert payload["session_id"] == "sess-evt-1"
assert payload["task_id"] == "task-evt-1" # was process_id
assert isinstance(payload["completed_at"], float) # was emitted_at
assert isinstance(payload["event_id"], str) and len(payload["event_id"]) >= 8
def test_event_id_is_unique_per_emit(monkeypatch):
"""T2: every emit gets a fresh event_id; two completions for two distinct
processes produce two distinct ids.
"""
fake = _FakeProcessRegistry()
fake.register("task-a", "sess-evt-2")
fake.register("task-b", "sess-evt-2")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
from api import background_process as bp
bp.register_process_session("sess-evt-2", "sess-evt-2")
monkeypatch.setattr(bp, "_EMIT_COALESCE_WINDOW_SECS", 0.0)
emits: list[tuple[str, dict]] = []
def _capture(session_id: str, event: str, data: dict) -> int:
emits.append((event, data))
return 1
monkeypatch.setattr(bp, "_emit_to_session_streams", _capture)
bp._process_one({"type": "completion", "session_id": "task-a", "session_key": "sess-evt-2", "exit_code": 0})
bp._process_one({"type": "completion", "session_id": "task-b", "session_key": "sess-evt-2", "exit_code": 0})
canonical_payloads = [d for ev, d in emits if ev == "bg_task_complete"]
assert len(canonical_payloads) == 2
assert canonical_payloads[0]["event_id"] != canonical_payloads[1]["event_id"]

View File

@@ -0,0 +1,215 @@
"""Structural assertions for the PR(c) UX surface on bg_task_complete.
Per P-bc §3.3 / §3.4: ``_handleBgTaskCompleteEvent`` in ``static/messages.js``
gains a toast surface and a T4 drop-when-focused gate stacked on top of the
PR(b) ring-buffer dedupe. The insertion order is contractual:
1. JSON parse + sid guard (existing)
2. ring-buffer dedup check (existing post-PR(b))
3. mark-as-seen + clear-unread bookkeeping (NEW)
4. T4 ``_isSessionActivelyViewed(sid)`` toast gate (NEW)
5. ``showToast(...)`` inside unfocused branch (NEW per Q-c-1)
6. diagnostic ack POST outside the T4 gate (existing)
We can't drive JS from pytest (the repo intentionally avoids a node/jsdom dep
per AGENTS.md), so this file does string-grep + relative-index assertions on
``static/messages.js`` — the same convention the rest of the WEBUI-SUB suite
uses. Each grep is precise so a behavioural regression (e.g. moving the ack
inside the focus gate, or leaking ``d.command`` / ``d.exit_code`` into the
toast copy per Rc-2) trips a hard failure.
"""
from __future__ import annotations
import re
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
def _read_messages_js() -> str:
return (REPO_ROOT / "static" / "messages.js").read_text()
def _handler_body() -> str:
"""Return the source slice of ``_handleBgTaskCompleteEvent`` start →
next top-level ``function`` declaration."""
js = _read_messages_js()
start = js.index("function _handleBgTaskCompleteEvent(")
# Next top-level function declaration after the handler.
rest = js[start + 1 :]
m = re.search(r"\n(function |// ──)", rest)
end = start + 1 + (m.start() if m else len(rest))
return js[start:end]
# ---------------------------------------------------------------------------
# Insertion-order contract
# ---------------------------------------------------------------------------
def _focus_gate_match(body: str):
"""Locate the T4 focus gate.
Supported shapes:
- legacy: ``if (_isSessionActivelyViewed(sid)) return;``
- current: ``const _viewed = ... _isSessionActivelyViewed(sid) ...;``
followed by ``if (_viewed) { ... } else { showToast(...) }``.
Returns an object exposing ``.start()`` / ``.end()`` over the gate block.
"""
# Direct legacy early-return form.
m = re.search(
r"if\s*\([^)]*_isSessionActivelyViewed\s*\(\s*sid\s*\)[^)]*\)\s*return",
body,
)
if m is not None:
return m
# Indirect form: const _viewed = ... _isSessionActivelyViewed(sid) ...;
# if (_viewed) { ... } else { ... } — brace-balance walk to find the gate block end.
flag_decl = re.search(
r"const\s+_viewed\s*=[^;]*_isSessionActivelyViewed\s*\(\s*sid\s*\)[^;]*;",
body,
)
if flag_decl is None:
return None
gate_head = re.search(r"if\s*\(\s*_viewed\s*\)\s*\{", body[flag_decl.end():])
if gate_head is None:
return None
open_abs = flag_decl.end() + gate_head.end() - 1 # index of '{'
depth = 0
close_abs = None
for i in range(open_abs, len(body)):
c = body[i]
if c == '{':
depth += 1
elif c == '}':
depth -= 1
if depth == 0:
close_abs = i
break
if close_abs is None:
return None
# If the gate has an `else`, include that branch because it owns the toast branch.
else_match = re.match(r"\s*else\s*\{", body[close_abs + 1 :])
if else_match is not None:
else_open_abs = close_abs + 1 + else_match.end() - 1
depth = 0
else_close_abs = None
for i in range(else_open_abs, len(body)):
c = body[i]
if c == '{':
depth += 1
elif c == '}':
depth -= 1
if depth == 0:
else_close_abs = i
break
if else_close_abs is None:
return None
close_abs = else_close_abs
class _M:
def __init__(self, s, e):
self._s, self._e = s, e
def start(self):
return self._s
def end(self):
return self._e
return _M(flag_decl.start(), close_abs + 1)
def test_focus_gate_is_after_ring_buffer_dedup():
"""T4 ``_isSessionActivelyViewed(sid)`` gate MUST appear AFTER the
``_bgTaskCompleteRingBufferAdd`` dedupe call so duplicates never reach the
focus gate (avoids touching mark-as-seen twice on a duplicate event)."""
body = _handler_body()
dedupe_idx = body.index("_bgTaskCompleteRingBufferAdd(sid, evt_id)")
gate_match = _focus_gate_match(body)
assert gate_match is not None, "T4 focus gate missing"
assert gate_match.start() > dedupe_idx, (
"T4 focus gate must follow the ring-buffer dedup check"
)
def test_mark_as_seen_is_after_dedup_and_inside_focus_gate():
"""Mark-as-seen + clear-unread bookkeeping MUST run after dedup and inside
the focused-viewer branch so an actively-viewed session clears its unread
counter even though the toast is suppressed."""
body = _handler_body()
dedupe_idx = body.index("_bgTaskCompleteRingBufferAdd(sid, evt_id)")
mark_idx = body.index("_markSessionViewed")
clear_idx = body.index("_clearSessionCompletionUnread")
gate_match = _focus_gate_match(body)
assert gate_match is not None
assert dedupe_idx < gate_match.start() <= mark_idx < gate_match.end(), (
"_markSessionViewed must sit after dedupe inside the T4 focus gate"
)
assert dedupe_idx < gate_match.start() <= clear_idx < gate_match.end(), (
"_clearSessionCompletionUnread must sit after dedupe inside the T4 focus gate"
)
def test_toast_call_is_inside_unfocused_gate_branch():
"""The ``showToast`` call MUST sit inside the T4 else/unfocused branch so a
focused viewer never sees a toast for the session they are watching."""
body = _handler_body()
gate_match = _focus_gate_match(body)
assert gate_match is not None
toast_idx = body.index("showToast(")
assert gate_match.start() < toast_idx < gate_match.end(), (
"showToast must be gated by the T4 drop-when-focused branch"
)
assert "} else {" in body[gate_match.start() : toast_idx], (
"showToast must be in the unfocused `else` branch, not the focused branch"
)
def test_ack_post_is_after_focus_gate_and_outside_toast_branch():
"""The diagnostic ack POST MUST run after the T4 gate and outside the toast
branch so both focused and unfocused viewers emit the server cleanup signal.
For unfocused viewers this preserves the existing toast-before-ack order."""
body = _handler_body()
gate_match = _focus_gate_match(body)
assert gate_match is not None
toast_idx = body.index("showToast(")
ack_idx = body.index("api/bg-task-complete-ack")
assert toast_idx < ack_idx, "unfocused toast must still precede diagnostic ack POST"
assert ack_idx > gate_match.end(), "diagnostic ack POST must live outside the T4 focus gate"
# ---------------------------------------------------------------------------
# Toast copy guards (Rc-2: minimal-payload-safe)
# ---------------------------------------------------------------------------
def _toast_block() -> str:
"""Slice the source from the toast comment to the showToast call's end."""
body = _handler_body()
# The toast block is the try { ... } that contains showToast(.
start = body.rindex("try", 0, body.index("showToast("))
# Find the matching close — toast block ends at the next "} catch (_) {}".
end_marker = body.index("} catch (_) {}", start)
return body[start : end_marker + len("} catch (_) {}")]
def test_toast_block_uses_only_minimal_payload_fields():
"""Per Rc-2 the toast copy may reference ONLY ``d.task_id`` and the
optional ``d.summary`` — never ``d.command`` or ``d.exit_code`` (those
fields are not guaranteed on the minimal payload shipped by the server)."""
block = _toast_block()
assert "d.task_id" in block, "toast must reference d.task_id"
assert "d.summary" in block, "toast must reference d.summary"
assert "d.command" not in block, "toast must NOT reference d.command (Rc-2)"
assert "d.exit_code" not in block, "toast must NOT reference d.exit_code (Rc-2)"
def test_toast_template_pins_copy():
"""The toast template (P-bc §3.3 Q-c-1 verbatim) wraps the task id in the
8-char prefix and falls back to ``''`` (empty tail — just ``Task <id> done``)
when ``d.summary`` is absent. Pin both literals so a future drift in copy is
caught loud."""
block = _toast_block()
assert "slice(0, 8)" in block
assert "slice(0, 80)" in block
assert "Task ${tid} done${tail}" in block
assert "2600" in block, "toast duration must be 2600ms per Q-c-1"

View File

@@ -0,0 +1,135 @@
"""Regression: ``loadSession`` must restart the session SSE stream on an early
failure exit (Greptile review on PR #2979).
Context
-------
``loadSession`` in ``static/sessions.js`` stops the per-session SSE stream
unconditionally near the top (mirroring ``stopApprovalPolling``):
if(typeof stopSessionStream==='function') stopSessionStream();
On the happy path it is restarted ~120 lines later at the success tail:
if(typeof startSessionStream==='function') startSessionStream(S.session.session_id);
But the metadata-fetch ``catch`` block (network error / 4xx / 5xx) returns
early WITHOUT reaching that restart. The session stream is the new feature's
primary delivery path for ``bg_task_complete`` events, so leaving it stopped
silently drops every completion event for the session still on screen until
the user explicitly navigates to a session again.
The fix restarts the stream for the session that remains on screen
(``currentSid``) inside the ``catch`` block, guarded so it does NOT fire when:
- a newer load is already in flight (``_loadingSessionId`` reset to a newer
sid owns the restart), or
- the failure self-healed away the current session (404 on the current
session) — there is no live session to stream for.
We can't drive JS from pytest (the repo intentionally avoids a node/jsdom dep
per AGENTS.md), so this file does string-grep + brace-balance assertions on
``static/sessions.js`` — the same convention the rest of the WEBUI suite uses.
"""
from __future__ import annotations
import re
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
def _read_sessions_js() -> str:
return (REPO_ROOT / "static" / "sessions.js").read_text()
def _load_session_body() -> str:
"""Return the source slice of ``async function loadSession(`` start →
next top-level ``function`` / ``async function`` declaration."""
js = _read_sessions_js()
start = js.index("async function loadSession(")
rest = js[start + 1 :]
m = re.search(r"\n(async function |function )", rest)
end = start + 1 + (m.start() if m else len(rest))
return js[start:end]
def _catch_block() -> str:
"""Return the metadata-fetch ``catch(e){ ... }`` slice within loadSession.
Anchors on the ``data = await api(`/api/session?...messages=0...`)`` try and
walks brace balance over the following ``catch (e) { ... }`` block.
"""
body = _load_session_body()
# The metadata fetch is the first `catch(` after the messages=0 api() call.
anchor = re.search(r"messages=0[^\n]*resolve_model=0", body)
assert anchor is not None, "metadata fetch (messages=0&resolve_model=0) not found"
cm = re.search(r"catch\s*\(\s*e\s*\)\s*\{", body[anchor.end():])
assert cm is not None, "catch(e){ for metadata fetch not found"
open_abs = anchor.end() + cm.end() - 1 # index of '{'
depth = 0
for i in range(open_abs, len(body)):
c = body[i]
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return body[open_abs : i + 1]
raise AssertionError("unbalanced catch block in loadSession")
def test_load_session_stops_stream_on_entry():
"""Sanity: the unconditional stop at the top of loadSession still exists —
it is the precondition that makes the restart-on-failure necessary."""
body = _load_session_body()
assert "stopSessionStream()" in body, (
"loadSession no longer stops the session stream on entry; the restart "
"guard's precondition has changed — re-derive this test."
)
def test_catch_restarts_session_stream_for_current_sid():
"""The metadata-fetch catch block must restart the session stream for the
session still on screen (currentSid)."""
catch = _catch_block()
m = re.search(r"startSessionStream\s*\(\s*currentSid\s*\)", catch)
assert m is not None, (
"loadSession's metadata-fetch catch block does not restart "
"startSessionStream(currentSid); bg_task_complete events would be "
"silently dropped for the on-screen session after a failed load."
)
def test_restart_is_guarded_against_newer_inflight_load():
"""The restart must be gated on ``_loadingSessionId === null`` so a newer
in-flight load (rapid session switch) owns the stream instead — the newer
load starts its own stream and must not be clobbered."""
catch = _catch_block()
# The guard and the restart call live in the same if-condition; assert the
# null-check precedes the startSessionStream(currentSid) call.
restart = re.search(r"startSessionStream\s*\(\s*currentSid\s*\)", catch)
assert restart is not None
guard = re.search(r"_loadingSessionId\s*===\s*null", catch[: restart.start()])
assert guard is not None, (
"restart of startSessionStream(currentSid) is not guarded by "
"_loadingSessionId === null; a newer in-flight load could be clobbered."
)
def test_restart_skipped_when_current_session_self_healed():
"""A 404 on the *current* session self-heals it away (clears localStorage +
URL). There is then no live session to stream for, so the restart must be
skipped to avoid spinning the SSE reconnect loop against a dead id."""
catch = _catch_block()
# A self-heal guard distinguishing the 404-on-current case must exist and
# gate the restart (negated in the restart condition).
assert re.search(r"_selfHealedCurrent", catch), (
"no _selfHealedCurrent guard found; a 404 on the current session would "
"wrongly restart a stream against a dead session id."
)
restart = re.search(r"startSessionStream\s*\(\s*currentSid\s*\)", catch)
assert restart is not None
neg_guard = re.search(r"!\s*_selfHealedCurrent", catch[: restart.start()])
assert neg_guard is not None, (
"restart is not gated on !_selfHealedCurrent; the self-healed-current "
"case would wrongly re-open a stream."
)

View File

@@ -0,0 +1,170 @@
"""Structural assertions for the bg_task_complete dedupe ring buffer.
Per P-bc §2.5: the WebUI consumer dedupe graduates from an unbounded ``Set``
keyed by ``task_id`` to a bounded ``Map``-backed ring buffer keyed by
``(session_id, event_id)``. The new structure carries:
* A 60-second TTL (``_BG_TASK_COMPLETE_TTL_MS = 60000``).
* A 256-entry soft cap (``_BG_TASK_COMPLETE_CAP = 256``).
* Lazy purge: every insert walks the Map in insertion order and drops
entries whose expiry has passed.
* Insertion-order eviction on overflow (oldest entry dropped first).
* A helper ``_bgTaskCompleteRingBufferAdd(sid, event_id)`` returning
``true`` on duplicate, ``false`` on first-seen.
We can't exercise JS at runtime from pytest (the repo intentionally avoids a
node/jsdom dep per AGENTS.md), so this file does structural / string-grep
assertions on ``static/messages.js`` — the same convention every other
WEBUI-SUB test uses. The grep targets are intentionally precise so a
behavioural regression (e.g. silently switching the dedupe key back to
``task_id`` or removing lazy purge) shows up as a hard test failure.
"""
from __future__ import annotations
import re
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
def _read_messages_js() -> str:
return (REPO_ROOT / "static" / "messages.js").read_text()
# ---------------------------------------------------------------------------
# Module-scope declarations
# ---------------------------------------------------------------------------
def test_ring_buffer_constants_declared_module_scope():
"""TTL and cap constants live at module scope, not buried inside a
function body, so the dedupe state is shared across every EventSource
(in-turn STREAMS + per-session SSE channel)."""
js = _read_messages_js()
assert "const _BG_TASK_COMPLETE_TTL_MS = 60000;" in js
assert "const _BG_TASK_COMPLETE_CAP = 256;" in js
# Map (not Set) carries the (key -> expiry) entries.
assert "const _bgTaskCompleteSeenIds = new Map();" in js
# Set form must be gone — D-b-2 replaced it.
assert "_bgTaskCompleteSeenIds = new Set()" not in js
assert "_seenProcessCompleteIds" not in js
def test_ring_buffer_helper_declared():
"""The add-and-dedupe helper exists and lives at module scope."""
js = _read_messages_js()
assert "function _bgTaskCompleteRingBufferAdd(sid, evt_id)" in js
# ---------------------------------------------------------------------------
# Helper-body internals
# ---------------------------------------------------------------------------
def _helper_source(js: str) -> str:
ix = js.index("function _bgTaskCompleteRingBufferAdd(sid, evt_id)")
# Helper body is short (~25-35 LOC); 2000 chars is comfortable headroom
# without bleeding into the next function.
return js[ix : ix + 2000]
def test_helper_ignores_missing_event_id():
"""Events without an event_id are ignored — server contract guarantees one.
Missing key returns ``true`` (treat as seen/skip) rather than ``false``
(proceed): if a future call site forgets the caller-side ``if (!evt_id)``
guard, an un-keyable completion is dropped instead of processed without a
dedupe key.
"""
body = _helper_source(_read_messages_js())
assert "if (!sid || !evt_id) return true;" in body
def test_helper_key_construction_uses_event_id_not_task_id():
"""Dedupe key is (session_id, event_id), the canonical contract surface."""
body = _helper_source(_read_messages_js())
# The composite key uses evt_id, not pid / task_id / process_id.
assert "const key = sid + '|' + evt_id;" in body
# Negative guards on the previous keying schemes.
assert "sid + '|' + pid" not in body
assert "sid + '|' + task_id" not in body
assert "process_id" not in body # legacy payload key, gone
def test_helper_lazy_purges_expired_entries():
"""Each add walks the Map in insertion order and drops expired entries.
Without lazy purge the soft cap eviction would silently drop live entries
while expired ones squat in the Map indefinitely."""
body = _helper_source(_read_messages_js())
assert "const now = Date.now();" in body
assert "for (const [k, exp] of _bgTaskCompleteSeenIds)" in body
assert "if (exp <= now)" in body
assert "_bgTaskCompleteSeenIds.delete(k);" in body
def test_helper_returns_true_on_duplicate():
"""Duplicate detection short-circuits — returns true without inserting."""
body = _helper_source(_read_messages_js())
assert "if (_bgTaskCompleteSeenIds.has(key)) return true;" in body
def test_helper_inserts_with_expiry_in_future():
"""First-seen entries are stamped with now + TTL_MS."""
body = _helper_source(_read_messages_js())
assert "_bgTaskCompleteSeenIds.set(key, now + _BG_TASK_COMPLETE_TTL_MS);" in body
def test_helper_soft_cap_evicts_oldest():
"""Soft cap enforcement uses insertion-order eviction (Map.keys().next())."""
body = _helper_source(_read_messages_js())
assert "while (_bgTaskCompleteSeenIds.size > _BG_TASK_COMPLETE_CAP)" in body
assert "_bgTaskCompleteSeenIds.keys().next().value" in body
assert "_bgTaskCompleteSeenIds.delete(firstKey);" in body
def test_helper_returns_false_on_first_seen():
"""First-seen path returns false so the caller proceeds to surface/ack."""
body = _helper_source(_read_messages_js())
# The trailing `return false;` of the helper.
assert re.search(r"return false;\s*\n\}", body), (
"helper must end with `return false;` after the soft-cap loop"
)
# ---------------------------------------------------------------------------
# Call-site integration in _handleBgTaskCompleteEvent
# ---------------------------------------------------------------------------
def _handler_source(js: str) -> str:
ix = js.index("function _handleBgTaskCompleteEvent")
return js[ix : ix + 2200]
def test_handler_calls_ring_buffer_helper():
"""The shared handler calls the helper with (sid, event_id) — not (sid, task_id)."""
body = _handler_source(_read_messages_js())
assert "_bgTaskCompleteRingBufferAdd(sid, evt_id)" in body
# Old call shape must be gone.
assert "_bgTaskCompleteSeenIds.has(dedupeKey)" not in body
assert "_bgTaskCompleteSeenIds.add(dedupeKey)" not in body
def test_handler_extracts_event_id_from_payload():
"""The handler pulls event_id out of the parsed payload."""
body = _handler_source(_read_messages_js())
assert "d.event_id" in body
# Missing event_id short-circuits before the dedupe / ack.
assert "if (!evt_id) return;" in body
def test_handler_dedup_runs_before_ack_post():
"""Dedupe gate must precede the fire-and-forget ack POST so duplicates
don't generate a flood of ack traffic."""
body = _handler_source(_read_messages_js())
dedupe_ix = body.index("_bgTaskCompleteRingBufferAdd(sid, evt_id)")
ack_ix = body.index("api/bg-task-complete-ack")
assert dedupe_ix < ack_ix, (
"ring-buffer dedupe must execute before the ack POST"
)

View File

@@ -0,0 +1,190 @@
"""T3 throttle tests for bg_task_complete emits.
The backend emits a canonical ``bg_task_complete`` SSE frame and a temporary
``process_complete`` alias for the same payload. T3 adds a per-session 1s
coalesce gate around that dual emit so rapid completion bursts do not flood a
live WebUI tab; the deferred emit must carry the latest payload from the burst.
"""
from __future__ import annotations
# The fake process-registry stub + its installer were duplicated verbatim in
# three bg_task_complete suites; they now live once in tests/_wakeup_helpers.py
# (Greptile review on PR #2979). Import under the legacy local names so the
# rest of this module is unchanged.
from tests._wakeup_helpers import FakeProcessRegistry as _FakeProcessRegistry
from tests._wakeup_helpers import install_fake_registry as _install_fake_registry
def _reset_state(bp):
from api import config as _cfg
with _cfg.PROCESS_SESSION_INDEX_LOCK:
_cfg.PROCESS_SESSION_INDEX.clear()
_cfg.PENDING_BG_TASK_COMPLETIONS.clear()
_cfg.BG_TASK_COMPLETE_EVENTS_SEEN.clear()
if hasattr(_cfg, "DEFERRED_PROCESS_WAKEUPS"):
with _cfg.DEFERRED_PROCESS_WAKEUPS_LOCK:
_cfg.DEFERRED_PROCESS_WAKEUPS.clear()
with _cfg.STREAMS_LOCK:
_cfg.STREAMS.clear()
if hasattr(_cfg, "ACTIVE_RUNS"):
with _cfg.ACTIVE_RUNS_LOCK:
_cfg.ACTIVE_RUNS.clear()
# T3 module-level throttle state.
if hasattr(bp, "_LAST_EMIT_TS"):
bp._LAST_EMIT_TS.clear()
if hasattr(bp, "_PENDING_EMIT_PAYLOADS"):
bp._PENDING_EMIT_PAYLOADS.clear()
if hasattr(bp, "_PENDING_EMIT_TIMERS"):
bp._PENDING_EMIT_TIMERS.clear()
class _FakeClock:
def __init__(self):
self.now = 1000.0
def time(self) -> float:
return self.now
def advance(self, seconds: float) -> None:
self.now += seconds
class _ManualTimer:
def __init__(self, timers: list["_ManualTimer"], delay: float, callback, args=()):
self.timers = timers
self.delay = delay
self.callback = callback
self.args = args
self.daemon = False
self.cancelled = False
self.started = False
def start(self):
self.started = True
self.timers.append(self)
def cancel(self):
self.cancelled = True
def fire(self):
if not self.cancelled:
self.callback(*self.args)
def _install_emit_harness(monkeypatch, *, session_id: str = "sess-throttle"):
from api import background_process as bp
fake = _FakeProcessRegistry()
_install_fake_registry(monkeypatch, fake)
_reset_state(bp)
bp.register_process_session(session_id, session_id)
emits: list[tuple[str, dict]] = []
def _capture_emit(sid: str, event: str, data: dict) -> int:
emits.append((event, dict(data)))
return 1
monkeypatch.setattr(bp, "_emit_to_session_streams", _capture_emit)
monkeypatch.setattr(bp, "_start_server_side_wakeup_turn", lambda sid, prompt: None)
monkeypatch.setattr(bp, "_session_has_active_turn", lambda sid: False)
clock = _FakeClock()
monkeypatch.setattr(bp.time, "time", clock.time)
timers: list[_ManualTimer] = []
def _timer_factory(delay, callback, args=(), kwargs=None):
assert kwargs in (None, {})
return _ManualTimer(timers, delay, callback, args)
monkeypatch.setattr(bp.threading, "Timer", _timer_factory)
return bp, fake, emits, clock, timers
def _process_completion(bp, fake, task_id: str, session_id: str = "sess-throttle") -> None:
fake.register(task_id, session_id)
bp._process_one(
{
"type": "completion",
"session_id": task_id,
"session_key": session_id,
"command": f"echo {task_id}",
"exit_code": 0,
"output": task_id,
}
)
def _canonical_payloads(emits: list[tuple[str, dict]]) -> list[dict]:
return [payload for event, payload in emits if event == "bg_task_complete"]
def test_rapid_bg_task_complete_emits_coalesce_to_immediate_plus_one_deferred(monkeypatch):
bp, fake, emits, _clock, timers = _install_emit_harness(monkeypatch)
for idx in range(10):
_process_completion(bp, fake, f"task-{idx}")
# First emit is immediate. The rest of the burst is represented by one
# deferred timer; cancelled timer instances may remain in the manual list.
assert len(_canonical_payloads(emits)) == 1
live_timers = [timer for timer in timers if not timer.cancelled]
assert len(live_timers) == 1
live_timers[0].fire()
canonical_payloads = _canonical_payloads(emits)
assert len(canonical_payloads) <= 2
assert [payload["task_id"] for payload in canonical_payloads] == ["task-0", "task-9"]
def test_bg_task_complete_emits_two_seconds_apart_all_fire(monkeypatch):
bp, fake, emits, clock, timers = _install_emit_harness(monkeypatch)
for idx in range(3):
_process_completion(bp, fake, f"spaced-{idx}")
clock.advance(2.0)
assert [payload["task_id"] for payload in _canonical_payloads(emits)] == [
"spaced-0",
"spaced-1",
"spaced-2",
]
assert not [timer for timer in timers if not timer.cancelled]
def test_coalesced_bg_task_complete_payload_replace_uses_latest_payload(monkeypatch):
bp, fake, emits, _clock, timers = _install_emit_harness(monkeypatch)
_process_completion(bp, fake, "first")
_process_completion(bp, fake, "middle")
_process_completion(bp, fake, "latest")
live_timers = [timer for timer in timers if not timer.cancelled]
assert len(live_timers) == 1
live_timers[0].fire()
canonical_payloads = _canonical_payloads(emits)
assert [payload["task_id"] for payload in canonical_payloads] == ["first", "latest"]
assert canonical_payloads[-1]["summary"].endswith("latest completed (exit_code=0).")
def test_outside_window_arrival_flushes_immediately_even_with_pending_timer(monkeypatch):
bp, fake, emits, clock, timers = _install_emit_harness(monkeypatch)
_process_completion(bp, fake, "first")
clock.advance(0.2)
_process_completion(bp, fake, "pending")
assert [payload["task_id"] for payload in _canonical_payloads(emits)] == ["first"]
assert len([timer for timer in timers if not timer.cancelled]) == 1
clock.advance(1.1)
_process_completion(bp, fake, "outside-window")
assert [payload["task_id"] for payload in _canonical_payloads(emits)] == [
"first",
"outside-window",
]
assert not [timer for timer in timers if not timer.cancelled]

View File

@@ -0,0 +1,323 @@
"""Wakeup tests for the renamed ``bg_task_complete`` SSE event.
This file replaces the legacy ``test_process_complete_wakeup.py`` after the
R2 §Q1 / Q4 contract update with the maintainer:
- Q1: the canonical SSE event is now ``bg_task_complete`` carrying the
minimal ``{session_id, task_id, completed_at, summary?, event_id}``
payload (the legacy ``process_complete`` name is dual-emitted under
PR (a) only as a 1-PR-cycle compatibility shim and is removed in
PR (b)).
- Q4: each emit must carry a fresh server-side ``event_id`` so the WebUI
can build a consumer-side TTL ring buffer for cross-disconnect
dedupe in a follow-up PR.
The tests below cover the wakeup-emit hot path end to end:
1. A completion event flowing through ``_process_one`` produces the
canonical ``bg_task_complete`` SSE emission with the trimmed payload
and a non-empty ``event_id``.
2. The same call also emits the legacy ``process_complete`` shim with the
same payload + same ``event_id``, so a consumer running an old
listener still wakes exactly once.
3. When ``_process_one`` runs while no per-session emit-coalesce window
is pending, the event is emitted immediately (i.e. wakeup is not
dropped by the throttle gate on a single completion).
4. The previous file name ``tests/test_process_complete_wakeup.py`` must
remain absent in the BACKEND-tier slice.
"""
from __future__ import annotations
import os
# The fake process-registry stub + its installer were duplicated verbatim in
# three bg_task_complete suites; they now live once in tests/_wakeup_helpers.py
# (Greptile review on PR #2979). Import under the legacy local names so the
# rest of this module is unchanged.
from tests._wakeup_helpers import FakeProcessRegistry as _FakeProcessRegistry
from tests._wakeup_helpers import install_fake_registry as _install_fake_registry
def _reset_cfg_state():
from api import config as _cfg
from api import background_process as bp
with _cfg.PROCESS_SESSION_INDEX_LOCK:
_cfg.PROCESS_SESSION_INDEX.clear()
_cfg.PENDING_BG_TASK_COMPLETIONS.clear()
_cfg.BG_TASK_COMPLETE_EVENTS_SEEN.clear()
with _cfg.STREAMS_LOCK:
_cfg.STREAMS.clear()
if hasattr(_cfg, "ACTIVE_RUNS"):
with _cfg.ACTIVE_RUNS_LOCK:
_cfg.ACTIVE_RUNS.clear()
if hasattr(bp, "_LAST_EMIT_TS"):
bp._LAST_EMIT_TS.clear()
if hasattr(bp, "_PENDING_EMIT_PAYLOADS"):
bp._PENDING_EMIT_PAYLOADS.clear()
if hasattr(bp, "_PENDING_EMIT_TIMERS"):
bp._PENDING_EMIT_TIMERS.clear()
def _capture_emits(monkeypatch):
"""Replace the per-session emit fan-out with a capturing list."""
from api import background_process as bp
emits: list[tuple[str, dict]] = []
def _capture(session_id: str, event: str, data: dict) -> int:
emits.append((event, data))
return 1
monkeypatch.setattr(bp, "_emit_to_session_streams", _capture)
# Run the coalesce gate in pass-through mode so a single completion
# exercises the immediate-emit branch (the throttle behaviour itself is
# covered exhaustively by tests/test_bg_task_complete_throttle.py).
monkeypatch.setattr(bp, "_EMIT_COALESCE_WINDOW_SECS", 0.0)
return emits
def test_bg_task_complete_wakeup_emits_canonical_event_with_event_id(monkeypatch):
"""``_process_one`` emits the canonical ``bg_task_complete`` SSE event
with the R2 §Q1 trimmed payload and a fresh server-side ``event_id``
(R2 §Q4).
"""
fake = _FakeProcessRegistry()
fake.register("task-wakeup-1", "sess-wakeup-1")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
from api import background_process as bp
bp.register_process_session("sess-wakeup-1", "sess-wakeup-1")
emits = _capture_emits(monkeypatch)
monkeypatch.setattr(bp, "_start_server_side_wakeup_turn", lambda *_args, **_kwargs: None)
evt = {
"type": "completion",
"session_id": "task-wakeup-1",
"session_key": "sess-wakeup-1",
"command": "sleep 1",
"exit_code": 0,
"output": "done",
}
bp._process_one(evt)
names = [e[0] for e in emits]
assert "bg_task_complete" in names, (
f"canonical bg_task_complete emit missing: {names}"
)
canonical_payloads = [d for ev, d in emits if ev == "bg_task_complete"]
assert canonical_payloads, "no canonical bg_task_complete payload captured"
payload = canonical_payloads[0]
expected_required = {"session_id", "task_id", "completed_at", "event_id"}
allowed = expected_required | {"summary"}
assert expected_required <= set(payload), (
f"missing required keys in bg_task_complete payload: {payload}"
)
assert set(payload) <= allowed, (
f"unexpected keys in trimmed bg_task_complete payload: {payload}"
)
# The legacy/dropped keys must NOT survive the T1 trim.
for dropped in (
"command",
"exit_code",
"type",
"stdout_preview",
"wakeup_prompt",
"emitted_at",
"process_id",
):
assert dropped not in payload, (
f"{dropped!r} should be dropped from bg_task_complete payload"
)
# Field-rename invariants.
assert payload["session_id"] == "sess-wakeup-1"
assert payload["task_id"] == "task-wakeup-1"
assert isinstance(payload["completed_at"], float)
# R2 §Q4: ``event_id`` is a non-empty string (uuid4().hex => 32 chars).
assert isinstance(payload["event_id"], str)
assert len(payload["event_id"]) >= 8
class _FakeHandler:
"""Minimal handler stub for exercising ``handle_post`` directly.
Mirrors the pattern in ``tests/test_issue1909_csp_report_only.py`` —
captures status + response headers + body without spinning a real HTTP
server.
"""
def __init__(self, body: bytes = b"{}", headers: dict | None = None):
import io as _io
self.headers = {
"Content-Length": str(len(body)),
"Content-Type": "application/json",
**(headers or {}),
}
self.rfile = _io.BytesIO(body)
self.wfile = _io.BytesIO()
self.client_address = ("127.0.0.1", 12345)
self.status: int | None = None
self.sent_headers: dict[str, str] = {}
def send_response(self, status: int) -> None:
self.status = status
def send_header(self, key: str, value: str) -> None:
self.sent_headers[key] = value
def end_headers(self) -> None:
pass
def test_legacy_process_complete_ack_returns_410_gone_with_x_replaced_by():
"""T1 deprecation alias: the old ``/api/process-complete-ack`` POST path
must return HTTP 410 Gone and an ``X-Replaced-By`` header pointing at
``/api/bg-task-complete-ack`` (V-a-final criterion #6 + D-a-fix item #1).
"""
import json as _json
from urllib.parse import urlparse
import api.routes as routes
handler = _FakeHandler()
parsed = urlparse("/api/process-complete-ack")
result = routes.handle_post(handler, parsed)
assert result is True, "deprecated ack endpoint must claim the request"
assert handler.status == 410, (
f"expected HTTP 410 Gone for deprecated ack path, got {handler.status}"
)
assert handler.sent_headers.get("X-Replaced-By") == "/api/bg-task-complete-ack", (
f"X-Replaced-By header missing or wrong: {handler.sent_headers}"
)
body = handler.wfile.getvalue()
# Body is gzip-wrapped only if Accept-Encoding allowed it; the fake
# handler does not set Accept-Encoding so the body is plain JSON.
payload = _json.loads(body.decode("utf-8"))
assert payload.get("replaced_by") == "/api/bg-task-complete-ack"
assert "gone" in payload.get("error", "").lower()
def test_bg_task_complete_ack_marks_process_id_alias_deprecated(monkeypatch):
"""The diagnostic ack endpoint keeps ``process_id`` as a transitional
request alias, but makes that legacy usage visible via ``Deprecation``.
"""
import json as _json
import types as _types
import api.routes as routes
monkeypatch.setattr(
routes,
"get_session",
lambda sid: _types.SimpleNamespace(session_id=sid),
)
handler = _FakeHandler()
routes._handle_bg_task_complete_ack(
handler,
{"session_id": "sess-legacy-alias", "process_id": "proc-legacy-1"},
)
assert handler.status == 200
assert handler.sent_headers.get("Deprecation") == "true"
payload = _json.loads(handler.wfile.getvalue().decode("utf-8"))
assert payload["task_id"] == "proc-legacy-1"
def test_bg_task_complete_ack_marks_mixed_process_id_presence_deprecated(monkeypatch):
"""If a request still includes ``process_id``, surface the transitional
alias even when the canonical ``task_id`` is also present.
"""
import json as _json
import types as _types
import api.routes as routes
monkeypatch.setattr(
routes,
"get_session",
lambda sid: _types.SimpleNamespace(session_id=sid),
)
handler = _FakeHandler()
routes._handle_bg_task_complete_ack(
handler,
{
"session_id": "sess-mixed-alias",
"task_id": "task-canonical-1",
"process_id": "proc-legacy-1",
},
)
assert handler.status == 200
assert handler.sent_headers.get("Deprecation") == "true"
payload = _json.loads(handler.wfile.getvalue().decode("utf-8"))
assert payload["task_id"] == "task-canonical-1"
def test_bg_task_complete_ack_canonical_task_id_has_no_deprecation_header(monkeypatch):
"""Canonical ``task_id`` requests should not be marked deprecated."""
import types as _types
import api.routes as routes
monkeypatch.setattr(
routes,
"get_session",
lambda sid: _types.SimpleNamespace(session_id=sid),
)
handler = _FakeHandler()
routes._handle_bg_task_complete_ack(
handler,
{"session_id": "sess-canonical", "task_id": "task-canonical-1"},
)
assert handler.status == 200
assert "Deprecation" not in handler.sent_headers
def test_bg_task_complete_ack_empty_process_id_alias_has_no_deprecation_header(monkeypatch):
"""An empty transitional alias key should not signal real legacy usage."""
import types as _types
import api.routes as routes
monkeypatch.setattr(
routes,
"get_session",
lambda sid: _types.SimpleNamespace(session_id=sid),
)
handler = _FakeHandler()
routes._handle_bg_task_complete_ack(
handler,
{
"session_id": "sess-canonical-empty-alias",
"task_id": "task-canonical-1",
"process_id": "",
},
)
assert handler.status == 200
assert "Deprecation" not in handler.sent_headers
def test_old_process_complete_wakeup_test_file_is_absent():
"""The legacy filename ``tests/test_process_complete_wakeup.py`` must
remain absent on this branch — the rename is part of the BACKEND-tier
T1 contract and is required by V-a-final criterion #9.
"""
here = os.path.dirname(os.path.abspath(__file__))
legacy = os.path.join(here, "test_process_complete_wakeup.py")
assert not os.path.exists(legacy), (
f"legacy {legacy!r} must not exist after the T1 rename"
)

View File

@@ -131,7 +131,7 @@ def test_576_restore_happens_after_load_session():
def test_585_get_available_models_calls_reload_config():
"""api/config.py: get_available_models() must do a mtime-based reload check."""
config_src = (REPO_ROOT / "api" / "config.py").read_text(encoding="utf-8")
fn_start = config_src.find("def get_available_models()")
fn_start = config_src.find("def get_available_models(")
assert fn_start != -1, "get_available_models not found"
fn_body_end = config_src.find('"""', config_src.find('"""', fn_start + 30) + 3) + 3
# Must check mtime before reading config

View File

@@ -0,0 +1,59 @@
"""Live-rebuild-budget warning rate-limit — Q-2979-A3.
Per Copilot discussion_r3305864400 the budget-exceeded warning at
api/config.py is potentially high-volume: a hung upstream probe or a sustained
burst of cold callers could flood the log at warning level. The fix wraps the
warning with ``_should_warn_budget``: the FIRST hit in a cooldown window logs
at warning, subsequent hits in the same window log at info — so the signal is
retained but the volume is bounded.
"""
from __future__ import annotations
import time
def test_should_warn_budget_first_call_returns_true():
from api import config as cfg
# Isolate per-test state.
cfg._BUDGET_WARN_STATE.pop("unit-test-reason-A", None)
assert cfg._should_warn_budget("unit-test-reason-A", cooldown_s=300.0) is True
def test_should_warn_budget_inside_cooldown_returns_false():
from api import config as cfg
cfg._BUDGET_WARN_STATE.pop("unit-test-reason-B", None)
assert cfg._should_warn_budget("unit-test-reason-B", cooldown_s=300.0) is True
# Second hit within cooldown — must be False (caller should demote to info).
assert cfg._should_warn_budget("unit-test-reason-B", cooldown_s=300.0) is False
assert cfg._should_warn_budget("unit-test-reason-B", cooldown_s=300.0) is False
def test_should_warn_budget_after_cooldown_returns_true_again():
from api import config as cfg
cfg._BUDGET_WARN_STATE.pop("unit-test-reason-C", None)
assert cfg._should_warn_budget("unit-test-reason-C", cooldown_s=0.05) is True
assert cfg._should_warn_budget("unit-test-reason-C", cooldown_s=0.05) is False
time.sleep(0.1)
# Cooldown elapsed — warning level resumes.
assert cfg._should_warn_budget("unit-test-reason-C", cooldown_s=0.05) is True
def test_should_warn_budget_distinct_reasons_have_independent_windows():
from api import config as cfg
for k in ("unit-test-reason-D1", "unit-test-reason-D2"):
cfg._BUDGET_WARN_STATE.pop(k, None)
assert cfg._should_warn_budget("unit-test-reason-D1", cooldown_s=300.0) is True
# A different reason MUST get its own first-hit warning even while D1 is
# still inside cooldown.
assert cfg._should_warn_budget("unit-test-reason-D2", cooldown_s=300.0) is True
# Both are now inside cooldown — both demote to info.
assert cfg._should_warn_budget("unit-test-reason-D1", cooldown_s=300.0) is False
assert cfg._should_warn_budget("unit-test-reason-D2", cooldown_s=300.0) is False

View File

@@ -0,0 +1,553 @@
"""Option Z live-view + SSE backpressure regression tests.
Two defects fixed on top of 481ddb9 (feat/process-complete-event-isla):
Defect B — server-initiated wakeup turn is not shown live (needs refresh).
Option Z starts the wakeup turn server-side via start_session_turn →
_start_chat_stream_for_session, which only emits the turn's token/tool/
stream_end frames to STREAMS[stream_id]. No browser EventSource is ever
attached to that stream (the browser only opens /api/chat/stream when IT
POSTs /api/chat/start). The per-session SSE channel only carried
bg_task_complete, never a signal to attach. Fix: when a process_wakeup
turn starts, emit a lightweight `server_turn_started` {stream_id} frame
onto SESSION_CHANNELS[session_id]; the open tab reuses its existing
chat-stream renderer (attachLiveStream) to attach to that stream_id.
Defect A — SSE thread exhaustion with multiple tabs.
server.py QuietHTTPServer(ThreadingHTTPServer) = one OS thread per
connection, no pool cap. A slow/backgrounded tab whose TCP recv window
is full makes handler.wfile.write()/flush() block forever → the worker
thread is pinned for the whole connection lifetime. Fix: a socket-level
SSE write deadline converts the indefinite block into socket.timeout
(== TimeoutError on py3.10+, already in routes._CLIENT_DISCONNECT_ERRORS)
so the handler loop breaks, `finally` unsubscribes, the thread is
released, and the channel reaper can reclaim it.
"""
from __future__ import annotations
import socket
import threading
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
# ---------------------------------------------------------------------------
# Defect B — server-initiated turn fans `server_turn_started` to SessionChannel
# ---------------------------------------------------------------------------
def test_server_turn_streams_to_session_channel(monkeypatch):
"""A process_wakeup turn that successfully starts must emit a
`server_turn_started` {stream_id} frame onto a subscribed SessionChannel
so an open tab can attach its existing renderer to the server-created
stream. Closed-tab path is unaffected (no subscriber → no-op)."""
from api import background_process as bp
import api.routes as routes
sid = "sess-optz-liveview-fanout"
fake_stream_id = "stream-optz-fanout-1"
# Patch the heavy turn-start core so the test stays unit-fast: pretend a
# turn started and return the same dict shape the real function returns.
def _fake_start_chat_stream_for_session(s, **kwargs):
return {"stream_id": fake_stream_id, "session_id": s.session_id, "_status": 200}
class _FakeSession:
session_id = sid
model = "test-model"
model_provider = None
monkeypatch.setattr(
routes, "_start_chat_stream_for_session", _fake_start_chat_stream_for_session, raising=True
)
monkeypatch.setattr(routes, "get_session", lambda _sid: _FakeSession(), raising=True)
monkeypatch.setattr(
routes, "_resolve_chat_workspace_with_recovery", lambda s, w: "/tmp/ws", raising=True
)
monkeypatch.setattr(
routes,
"_resolve_compatible_session_model_state",
lambda m, p, **_kw: ("test-model", None, False),
raising=True,
)
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
try:
resp = routes.start_session_turn(sid, "[IMPORTANT: bg done]", source="process_wakeup")
assert resp.get("stream_id") == fake_stream_id
event_name, data = q.get(timeout=2.0)
assert event_name == "server_turn_started", (
"server-initiated turn must fan a server_turn_started frame onto "
"the per-session live-view channel"
)
assert data["stream_id"] == fake_stream_id
assert data["session_id"] == sid
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
def test_server_turn_no_session_channel_is_noop(monkeypatch):
"""Closed-tab path: no SessionChannel exists → start_session_turn must
NOT create one and must still return the started stream (server-side
wakeup, the Option Z headline, is unaffected)."""
from api import background_process as bp
import api.routes as routes
sid = "sess-optz-liveview-notab"
fake_stream_id = "stream-optz-notab-1"
def _fake_start_chat_stream_for_session(s, **kwargs):
return {"stream_id": fake_stream_id, "session_id": s.session_id, "_status": 200}
class _FakeSession:
session_id = sid
model = "test-model"
model_provider = None
monkeypatch.setattr(
routes, "_start_chat_stream_for_session", _fake_start_chat_stream_for_session, raising=True
)
monkeypatch.setattr(routes, "get_session", lambda _sid: _FakeSession(), raising=True)
monkeypatch.setattr(
routes, "_resolve_chat_workspace_with_recovery", lambda s, w: "/tmp/ws", raising=True
)
monkeypatch.setattr(
routes,
"_resolve_compatible_session_model_state",
lambda m, p, **_kw: ("test-model", None, False),
raising=True,
)
assert bp.get_session_channel(sid) is None
resp = routes.start_session_turn(sid, "[IMPORTANT: bg done]", source="process_wakeup")
assert resp.get("stream_id") == fake_stream_id
# Must not have auto-created a channel just to fan a frame nobody hears.
assert bp.get_session_channel(sid) is None
# ---------------------------------------------------------------------------
# Defect A — SSE write deadline drops a stuck writer / releases the thread
# ---------------------------------------------------------------------------
def test_sse_write_deadline_helper_sets_socket_timeout():
from api.streaming import _sse_set_write_deadline, SSE_WRITE_DEADLINE_SECONDS
recorded = {}
class _FakeConn:
def settimeout(self, v):
recorded["timeout"] = v
class _FakeHandler:
connection = _FakeConn()
h = _FakeHandler()
_sse_set_write_deadline(h)
assert recorded["timeout"] == SSE_WRITE_DEADLINE_SECONDS
_sse_set_write_deadline(h, 7.5)
assert recorded["timeout"] == 7.5
def test_sse_write_deadline_helper_never_raises():
"""A handler without a usable connection must not blow up the SSE setup."""
from api.streaming import _sse_set_write_deadline
class _NoConn:
connection = None
class _Broken:
@property
def connection(self):
raise RuntimeError("boom")
_sse_set_write_deadline(_NoConn()) # no exception
_sse_set_write_deadline(_Broken()) # no exception
_sse_set_write_deadline(object()) # no exception
def test_sse_write_timeout_drops_slow_subscriber():
"""Behavioural: a SessionChannel subscriber whose SSE writer raises
socket.timeout (the stuck-tab signal a write deadline produces) results
in the channel being unsubscribed and the worker released — modelled by
running the exact loop/break/finally contract the route uses.
socket.timeout is TimeoutError on py3.10+, which is in
routes._CLIENT_DISCONNECT_ERRORS, so the route's existing
`except _CLIENT_DISCONNECT_ERRORS:` already handles it once a deadline
is set. This test pins that contract.
"""
from api import background_process as bp
from api.routes import _CLIENT_DISCONNECT_ERRORS
assert socket.timeout in (_CLIENT_DISCONNECT_ERRORS) or issubclass(
socket.timeout, _CLIENT_DISCONNECT_ERRORS
), "socket.timeout must be catchable by the SSE route's disconnect handler"
sid = "sess-optz-stuck-writer"
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
assert ch.subscriber_count() == 1
released = threading.Event()
def _route_like_loop():
# Mirror _handle_session_sse_stream's loop+finally exactly.
try:
while True:
ch.emit("server_turn_started", {"stream_id": "x"})
_evt = q.get(timeout=1.0)
# Simulate handler.wfile.write hitting the write deadline:
raise socket.timeout("timed out")
except _CLIENT_DISCONNECT_ERRORS:
pass
finally:
ch.unsubscribe(q)
released.set()
t = threading.Thread(target=_route_like_loop, daemon=True)
t.start()
assert released.wait(timeout=3.0), "stuck-writer handler did not release"
t.join(timeout=2.0)
assert ch.subscriber_count() == 0, "stuck subscriber was not dropped"
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
# ---------------------------------------------------------------------------
# Source-grep wiring guards
# ---------------------------------------------------------------------------
def test_all_sse_endpoints_set_write_deadline():
src = (REPO_ROOT / "api" / "routes.py").read_text()
assert "_sse_set_write_deadline" in src
# Every SSE handler must arm the deadline. Count call sites — there are
# 6 long-lived SSE endpoints (chat-stream, terminal, gateway, approval,
# clarify, session).
assert src.count("_sse_set_write_deadline(handler") >= 6, (
"all 6 SSE endpoints must arm the write deadline"
)
def test_streaming_exports_write_deadline_api():
from api import streaming
assert hasattr(streaming, "_sse_set_write_deadline")
assert hasattr(streaming, "SSE_WRITE_DEADLINE_SECONDS")
assert isinstance(streaming.SSE_WRITE_DEADLINE_SECONDS, (int, float))
def test_sse_write_deadline_env_override(monkeypatch):
import importlib
from api import streaming
monkeypatch.setenv("HERMES_SSE_WRITE_DEADLINE", "7.25")
try:
reloaded = importlib.reload(streaming)
assert reloaded.SSE_WRITE_DEADLINE_SECONDS == 7.25
finally:
monkeypatch.delenv("HERMES_SSE_WRITE_DEADLINE", raising=False)
importlib.reload(streaming)
def test_start_session_turn_emits_server_turn_started():
src = (REPO_ROOT / "api" / "routes.py").read_text()
assert "server_turn_started" in src
# Must use the non-creating accessor so the closed-tab path stays a no-op.
assert "get_session_channel" in src
def test_frontend_attaches_renderer_on_server_turn_started():
js = (REPO_ROOT / "static" / "messages.js").read_text()
assert "server_turn_started" in js
# Must reuse the existing chat-stream render path, not hand-roll a 2nd one.
assert "attachLiveStream" in js
# ---------------------------------------------------------------------------
# Root cause (open-tab live-view): lost fire-and-forget server_turn_started
# The fan-out in start_session_turn is SessionChannel.emit with NO replay
# buffer — a tab whose /api/session/stream subscriber is momentarily absent
# at the emit instant (transient SSE drop, reverse-proxy idle-timeout,
# browser connection-pool starvation) misses the frame permanently and the
# server-initiated wakeup never renders live (the user must hard-refresh).
# The server-side wakeup itself ran + persisted fine; ONLY the live-view
# was lost. Fix: on (re)subscribe to /api/session/stream, if the session
# has a live run RIGHT NOW, replay a synthetic server_turn_started
# {recovered: True} to that new subscriber so the open tab self-heals.
#
# Reproduced deterministically with Playwright on the real instance: with
# the per-session EventSource force-closed at the exact wakeup-emit instant
# (no subscriber), `sleep 15`'s wakeup turn did NOT render live; a hard
# refresh showed it WAS persisted (proving server-side wakeup works and
# only live-view was broken). See workspace/liveview-open-tab-fix.md §1.
# ---------------------------------------------------------------------------
def test_active_stream_id_for_session_returns_live_run_stream():
"""The on-subscribe recovery lookup must return the live run's stream_id
when the session has an ACTIVE_RUNS row, and None when idle."""
from api import background_process as bp, config as cfg
sid = "sess-recover-lookup"
stream_id = "stream-recover-lookup-1"
assert bp.active_stream_id_for_session(sid) is None # idle → None
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
try:
assert bp.active_stream_id_for_session(sid) == stream_id
# Unrelated session must not match.
assert bp.active_stream_id_for_session("sess-other") is None
finally:
cfg.ACTIVE_RUNS.pop(stream_id, None)
assert bp.active_stream_id_for_session(sid) is None # cleaned up → None
def test_session_sse_on_subscribe_recovers_lost_server_turn_started():
"""Behavioural contract: a tab that (re)subscribes to the per-session
channel AFTER the fire-and-forget server_turn_started was already
broadcast (so it missed the original frame) must still receive a
recovery server_turn_started for the in-flight stream — modelled by
running the exact recovery block the route uses.
This is the root-cause fix: without it, a momentarily-absent subscriber
loses the frame permanently and the open tab never renders the
server-initiated wakeup turn live (needs a hard refresh).
"""
from api import background_process as bp, config as cfg
sid = "sess-recover-onsub"
stream_id = "stream-recover-onsub-1"
# Simulate: server-side wakeup turn IS live (ACTIVE_RUNS row exists), but
# the original server_turn_started broadcast already happened and reached
# NO subscriber (the tab's EventSource was momentarily down).
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe() # tab (re)connects NOW, after the lost broadcast
try:
# Exactly the route's on-subscribe recovery logic
# (_handle_session_sse_stream): look up the live run and replay.
recover_stream_id = bp.active_stream_id_for_session(sid)
assert recover_stream_id == stream_id
recovery_frame = {
"session_id": sid,
"stream_id": recover_stream_id,
"source": "subscribe_recovery",
"recovered": True,
}
# The route _sse()s this directly to the new subscriber's connection;
# the contract under test is "a freshly-subscribed tab gets a
# server_turn_started for the in-flight stream so it can attach".
assert recovery_frame["stream_id"] == stream_id
assert recovery_frame["recovered"] is True
assert recovery_frame["session_id"] == sid
# And when the session is idle (no live run) the recovery is a no-op
# — no spurious attach frame for a session with nothing running.
cfg.ACTIVE_RUNS.pop(stream_id, None)
assert bp.active_stream_id_for_session(sid) is None
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
cfg.ACTIVE_RUNS.pop(stream_id, None)
def test_session_sse_handler_wires_on_subscribe_recovery():
"""Source-grep: the per-session SSE handler must perform on-subscribe
recovery via active_stream_id_for_session and emit a recovered
server_turn_started, AFTER subscribing (so it can't race the original)."""
src = (REPO_ROOT / "api" / "routes.py").read_text()
assert "active_stream_id_for_session" in src
# The recovery must be inside the session SSE handler and use the
# recovered marker so the frontend uses the replay attach path.
handler_ix = src.index("def _handle_session_sse_stream")
handler_src = src[handler_ix:handler_ix + 6000]
assert "active_stream_id_for_session" in handler_src
assert '"recovered": True' in handler_src
assert "server_turn_started" in handler_src
# Recovery CALL must come AFTER the channel subscription so a frame emitted
# between subscribe and recovery is still caught by the queue (no lost-frame
# gap). The handler subscribes via the atomic ``subscribe_to_session_channel``
# helper (TOCTOU-safe get-or-create+subscribe under one lock); assert on that
# call site vs the recovery call site ``= active_stream_id_for_session(``.
assert handler_src.index("subscribe_to_session_channel(") < handler_src.index(
"= active_stream_id_for_session("
)
def test_frontend_recovered_frame_uses_reconnecting_attach():
"""The frontend server_turn_started handler must honour `recovered`:
a recovered (replay) frame attaches via the reconnecting path so the
renderer rebuilds the in-progress stream from the run journal instead
of expecting token 0 (which would render a truncated turn)."""
js = (REPO_ROOT / "static" / "messages.js").read_text()
assert "recovered" in js
h_ix = js.index("addEventListener('server_turn_started'")
h_src = js[h_ix:h_ix + 1600]
assert "d.recovered" in h_src
assert "reconnecting" in h_src
# Still reuses the single renderer — no second hand-rolled stream.
assert "attachLiveStream" in h_src
# ---------------------------------------------------------------------------
# Copilot review #3 — _emit_to_session_streams owner-unknown broadcast
# Resolution: skip non-matching AND owner-unknown streams on the STREAMS
# loop (rely solely on SESSION_CHANNELS for cross-turn live-view, which the
# repro proved is the sole authoritative carrier post Option X/Z). Removes
# the cross-session-leak surface Copilot flagged.
# ---------------------------------------------------------------------------
def test_emit_to_session_streams_skips_owner_unknown_stream():
"""Copilot #3: a STREAMS entry with NO ACTIVE_RUNS row (owner unknown)
must NOT receive the event on the STREAMS loop — the old code broadcast
to it, relying on every frontend consumer to filter by session_id (a
fragile cross-session leak). The per-session SessionChannel still
delivers (the authoritative cross-turn live-view path)."""
from api import background_process as bp, config as cfg
sid = "sess-copilot3-skip"
unknown_stream_id = "stream-copilot3-no-active-run"
leaked: list = []
class _FakeStreamChannel:
def put_nowait(self, item):
leaked.append(item)
with cfg.STREAMS_LOCK:
cfg.STREAMS[unknown_stream_id] = _FakeStreamChannel()
# Deliberately NO cfg.ACTIVE_RUNS row for unknown_stream_id → owner unknown.
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
try:
emitted = bp._emit_to_session_streams(sid, "bg_task_complete", {"session_id": sid})
# The owner-unknown STREAMS entry must NOT have been written to.
assert leaked == [], (
"owner-unknown stream must be skipped (no cross-session broadcast)"
)
# The per-session SessionChannel still delivered (authoritative path).
ev, data = q.get(timeout=2.0)
assert ev == "bg_task_complete"
assert data["session_id"] == sid
assert emitted >= 1
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
with cfg.STREAMS_LOCK:
cfg.STREAMS.pop(unknown_stream_id, None)
def test_emit_to_session_streams_still_delivers_to_matching_owner():
"""Regression guard for the Copilot #3 change: an owner-KNOWN stream
whose session matches MUST still receive the event on the STREAMS loop
(in-turn defense-in-depth path is preserved)."""
from api import background_process as bp, config as cfg
sid = "sess-copilot3-match"
stream_id = "stream-copilot3-match-1"
received: list = []
class _FakeStreamChannel:
def put_nowait(self, item):
received.append(item)
with cfg.STREAMS_LOCK:
cfg.STREAMS[stream_id] = _FakeStreamChannel()
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
try:
bp._emit_to_session_streams(sid, "bg_task_complete", {"session_id": sid})
assert received, "owner-matching stream must still receive in-turn delivery"
assert received[0][0] == "bg_task_complete"
ev, _data = q.get(timeout=2.0)
assert ev == "bg_task_complete"
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
with cfg.STREAMS_LOCK:
cfg.STREAMS.pop(stream_id, None)
cfg.ACTIVE_RUNS.pop(stream_id, None)
def test_emit_to_session_streams_does_not_leak_to_other_session_owner():
"""Cross-session isolation: a stream owned by a DIFFERENT session must
never receive this session's event (unchanged behavior, pinned)."""
from api import background_process as bp, config as cfg
sid = "sess-copilot3-self"
other_sid = "sess-copilot3-other"
other_stream_id = "stream-copilot3-other-1"
leaked: list = []
class _FakeStreamChannel:
def put_nowait(self, item):
leaked.append(item)
with cfg.STREAMS_LOCK:
cfg.STREAMS[other_stream_id] = _FakeStreamChannel()
cfg.ACTIVE_RUNS[other_stream_id] = {"session_id": other_sid}
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
try:
bp._emit_to_session_streams(sid, "bg_task_complete", {"session_id": sid})
assert leaked == [], "must not leak to a different session's stream"
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
with cfg.STREAMS_LOCK:
cfg.STREAMS.pop(other_stream_id, None)
cfg.ACTIVE_RUNS.pop(other_stream_id, None)
def test_emit_to_session_streams_skip_unknown_owner_documented_in_source():
"""Source-grep: the Copilot #3 resolution must be the skip-unknown-owner
form (`if owner_sid != session_id: continue`), not the old
broadcast-on-unknown fallback (`if owner_sid and owner_sid != ...`)."""
src = (REPO_ROOT / "api" / "background_process.py").read_text()
fn_ix = src.index("def _emit_to_session_streams")
fn_src = src[fn_ix:fn_ix + 2600]
assert "if owner_sid != session_id:" in fn_src
assert "if owner_sid and owner_sid != session_id:" not in fn_src
assert "Copilot review #3" in fn_src
# ---------------------------------------------------------------------------
# event_id contract surface (post-rename to bg_task_complete)
# ---------------------------------------------------------------------------
#
# The #2242 thread Q4 reply pins the consumer dedupe key on
# `(session_id, event_id)`. The handler in static/messages.js MUST treat
# event_id as mandatory and MUST NOT surface or ack an event missing one.
# ---------------------------------------------------------------------------
def test_frontend_handler_requires_event_id_to_surface():
"""Source-grep: _handleBgTaskCompleteEvent ignores events without event_id."""
js = (REPO_ROOT / "static" / "messages.js").read_text()
fn_ix = js.index("function _handleBgTaskCompleteEvent")
fn_src = js[fn_ix:fn_ix + 1800]
# event_id is extracted from the payload.
assert "d.event_id" in fn_src
# Missing event_id short-circuits before any dedupe / ack.
assert "if (!evt_id) return;" in fn_src or "if(!evt_id) return;" in fn_src
# Dedupe goes through the ring buffer helper, keyed by (sid, event_id).
assert "_bgTaskCompleteRingBufferAdd(sid, evt_id)" in fn_src
# Ack body carries event_id so server can correlate.
assert "event_id: evt_id" in fn_src

View File

@@ -1060,10 +1060,15 @@ def test_session_model_display_resolver_is_read_only(monkeypatch):
"""Read-path model resolution must not mutate or save the session."""
import api.routes as routes
# Accept **kwargs: the read-only display resolver now opts into the
# cache-only catalog via get_available_models(prefer_cache=True) so the
# hot GET /api/session path never triggers the cold live provider rebuild
# (multi-tab streaming interlock RCA). The stub must mirror the real
# signature; the contract under test here is read-only-ness, not arity.
monkeypatch.setattr(
routes,
"get_available_models",
lambda: {
lambda **_kw: {
"active_provider": "openai-codex",
"default_model": "gpt-5.4-mini",
},

View File

@@ -406,33 +406,61 @@ def test_approval_respond_approves_from_gateway_queues_when_pending_empty() -> N
def test_chat_start_route_selects_adapter_only_when_flag_enabled():
routes = importlib.import_module("api.routes")
src = (routes.Path(__file__).parent.parent / "api" / "routes.py").read_text(encoding="utf-8")
# NOTE: T-2979-fix factored the adapter-selection block out of
# _handle_chat_start into the shared `_start_run` helper (used by both
# /api/chat/start and start_session_turn — Q-2979-A2 / Copilot
# r3305864087/r3305864173). Scan the helper body for the contract; the
# route body only needs to delegate to it.
helper_idx = src.index("def _start_run(")
helper_body = src[helper_idx:src.index("def start_session_turn(", helper_idx)]
start_idx = src.index("def _handle_chat_start")
start_body = src[start_idx:src.index("def _resolve_chat_workspace_with_recovery", start_idx)]
assert "runtime_adapter_enabled()" in start_body
assert "runtime_adapter_runner_enabled()" in start_body
assert "build_runtime_adapter(" in start_body
assert "legacy_adapter_factory=_legacy_adapter_factory" in start_body
assert "runner_client_factory=_runtime_runner_client_factory" in start_body
assert "LegacyJournalRuntimeAdapter" in start_body
assert "_start_chat_stream_for_session(" in start_body
assert "HERMES_WEBUI_RUNTIME_ADAPTER" not in start_body, "route should use runtime_adapter_enabled(), not inline env checks"
# Contract enforced in the shared helper:
assert "runtime_adapter_enabled()" in helper_body
assert "runtime_adapter_runner_enabled()" in helper_body
assert "build_runtime_adapter(" in helper_body
assert "legacy_adapter_factory=_legacy_adapter_factory" in helper_body
assert "runner_client_factory=_runtime_runner_client_factory" in helper_body
assert "LegacyJournalRuntimeAdapter" in helper_body
assert "_start_chat_stream_for_session(" in helper_body
# Route delegates to the helper instead of inlining env checks:
assert "_start_run(" in start_body
assert "HERMES_WEBUI_RUNTIME_ADAPTER" not in start_body, "route should use runtime_adapter_enabled() via _start_run, not inline env checks"
assert "HERMES_WEBUI_RUNTIME_ADAPTER" not in helper_body, "helper should use runtime_adapter_enabled(), not inline env checks"
def test_runner_local_chat_start_selection_does_not_fallback_to_legacy():
routes = importlib.import_module("api.routes")
src = (routes.Path(__file__).parent.parent / "api" / "routes.py").read_text(encoding="utf-8")
# See note in test_chat_start_route_selects_adapter_only_when_flag_enabled
# — adapter selection moved into the shared `_start_run` helper.
helper_idx = src.index("def _start_run(")
helper_body = src[helper_idx:src.index("def start_session_turn(", helper_idx)]
start_idx = src.index("def _handle_chat_start")
start_body = src[start_idx:src.index("def _resolve_chat_workspace_with_recovery", start_idx)]
flag_branch = "if runtime_adapter_enabled() or runtime_adapter_runner_enabled():"
assert flag_branch in start_body
assert "except NotImplementedError as exc:" in start_body
assert 'return j(handler, {"error": str(exc)}, status=501)' in start_body
assert flag_branch in helper_body
assert "except NotImplementedError as exc:" in helper_body
# The helper returns {"error": str(exc), "_status": 501}; the route then
# maps that onto the legacy j(handler, {...}, status=501) response shape
# to keep the public contract identical to pre-refactor behavior.
assert 'return {"error": str(exc), "_status": 501}' in helper_body
assert 'return j(handler, {"error": response["error"]}, status=501)' in start_body
assert "runner-local chat backend is not configured" in src
adapter_branch = start_body[start_body.index(flag_branch):start_body.index("else:", start_body.index(flag_branch))]
# The adapter branch inside the helper still calls _start_chat_stream_for_session
# through the _legacy_start_run delegate before the trailing legacy-direct
# fallthrough (the function returns the legacy direct call when the flag
# is off — no `else:` keyword anymore since each branch returns).
adapter_branch_start = helper_body.index(flag_branch)
# Slice up to the final (post-flag) return _start_chat_stream_for_session
# — there are two occurrences: one inside _legacy_start_run, one at the
# fallthrough; we want both inside the branch slice.
fallthrough = helper_body.rindex("return _start_chat_stream_for_session(")
adapter_branch = helper_body[adapter_branch_start:fallthrough]
assert "_start_chat_stream_for_session(" in adapter_branch, "legacy-journal delegate should still call the legacy path"
assert "runtime_adapter_runner_enabled()" in adapter_branch
assert "runtime_adapter_runner_enabled()" in adapter_branch or "runtime_adapter_runner_enabled()" in helper_body
def test_chat_start_adapter_path_preserves_legacy_response_shape():

View File

@@ -0,0 +1,921 @@
"""Tests for the persistent per-session SSE channel (Option X).
Verifies the SessionChannel registry + reaper + dual-emit + endpoint wiring
that bridges the cross-turn bg_task_complete delivery gap (between agent
turns STREAMS is torn down, so the session-scoped channel is the only live
surface).
Companion to t_98368bd0 implementation plan. Structural (source-grep) checks
plus pure-function tests for the SessionChannel class and reaper logic.
"""
from __future__ import annotations
import time
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[1]
# ---------------------------------------------------------------------------
# Module surface
# ---------------------------------------------------------------------------
def test_background_process_exports_session_channel_api():
from api import background_process as bp
for name in (
"SessionChannel",
"SESSION_CHANNELS",
"SESSION_CHANNELS_LOCK",
"get_or_create_session_channel",
"subscribe_to_session_channel",
"get_session_channel",
"start_session_channel_reaper",
"stop_session_channel_reaper",
):
assert hasattr(bp, name), f"missing: {name}"
def test_config_exports_session_channel_ttl_constants():
from api import config as cfg
assert isinstance(cfg.SESSION_CHANNEL_IDLE_TTL_SECS, int)
assert cfg.SESSION_CHANNEL_IDLE_TTL_SECS == 14400 # 4 hours per spec
assert isinstance(cfg.SESSION_CHANNEL_SUBSCRIBER_GRACE_SECS, int)
assert cfg.SESSION_CHANNEL_SUBSCRIBER_GRACE_SECS == 60
# ---------------------------------------------------------------------------
# SessionChannel: subscribe / emit / unsubscribe lifecycle
# ---------------------------------------------------------------------------
def test_session_channel_subscriber_lifecycle():
"""subscribe → emit → unsubscribe with no leak."""
from api.background_process import SessionChannel
ch = SessionChannel("sess-1")
q1 = ch.subscribe()
q2 = ch.subscribe()
assert ch.subscriber_count() == 2
delivered = ch.emit("bg_task_complete", {"hello": 1})
assert delivered == 2
assert q1.get_nowait() == ("bg_task_complete", {"hello": 1})
assert q2.get_nowait() == ("bg_task_complete", {"hello": 1})
ch.unsubscribe(q1)
assert ch.subscriber_count() == 1
# Re-emit goes to remaining sub only
ch.emit("bg_task_complete", {"hello": 2})
assert q2.get_nowait() == ("bg_task_complete", {"hello": 2})
ch.unsubscribe(q2)
assert ch.subscriber_count() == 0
def test_session_channel_emit_with_full_buffer_drops_silently():
"""A slow tab whose queue is full doesn't block other subscribers."""
from api.background_process import SessionChannel
ch = SessionChannel("sess-full")
q_slow = ch.subscribe(maxsize=1)
q_fast = ch.subscribe(maxsize=16)
# Fill the slow queue
q_slow.put_nowait(("filler", {}))
delivered = ch.emit("bg_task_complete", {"x": 1})
# fast receives, slow dropped
assert delivered == 1
assert q_fast.get_nowait() == ("bg_task_complete", {"x": 1})
# ---------------------------------------------------------------------------
# Reaper: subscribers-empty grace + idle TTL cap
# ---------------------------------------------------------------------------
def test_session_channel_reaper_keeps_live_subscriber():
"""A channel with at least one subscriber must NEVER be collected."""
from api.background_process import SessionChannel
ch = SessionChannel("sess-live")
ch.subscribe()
# Far in the future, with live subscriber → not collected
assert ch.reaper_should_collect(time.time() + 1_000_000) is False
def test_session_channel_reaper_grace_period():
"""No subscribers + grace expired → collect."""
from api.background_process import SessionChannel
from api import config as cfg
ch = SessionChannel("sess-grace")
q = ch.subscribe()
ch.unsubscribe(q)
# Just dropped — within grace → keep
assert ch.reaper_should_collect(time.time()) is False
# Past grace → collect
later = time.time() + cfg.SESSION_CHANNEL_SUBSCRIBER_GRACE_SECS + 1
assert ch.reaper_should_collect(later) is True
def test_session_channel_reaper_idle_ttl_cap():
"""Channel older than idle TTL with no subscribers → collect."""
from api.background_process import SessionChannel
from api import config as cfg
ch = SessionChannel("sess-zombie")
# Force created_at far in the past, no subscriber drop tracked
ch.created_at = time.time() - (cfg.SESSION_CHANNEL_IDLE_TTL_SECS + 100)
ch.last_subscriber_drop_at = None
assert ch.subscriber_count() == 0
assert ch.reaper_should_collect(time.time()) is True
def test_session_channel_reaper_idle_ttl_held_by_live_subscriber():
"""Even past idle TTL, a live subscriber keeps the channel alive."""
from api.background_process import SessionChannel
from api import config as cfg
ch = SessionChannel("sess-zombie-live")
ch.created_at = time.time() - (cfg.SESSION_CHANNEL_IDLE_TTL_SECS + 100)
ch.subscribe() # someone IS listening
assert ch.reaper_should_collect(time.time()) is False
def test_reaper_collects_via_registry():
"""The reaper loop iterates SESSION_CHANNELS and removes collected entries."""
from api import background_process as bp, config as cfg
sid = "sess-reaper-integration"
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
ch.unsubscribe(q)
# Push the drop time far enough back to trigger grace collection
ch.last_subscriber_drop_at = time.time() - (cfg.SESSION_CHANNEL_SUBSCRIBER_GRACE_SECS + 5)
# Drive one iteration of the reaper's body directly
now = time.time()
with bp.SESSION_CHANNELS_LOCK:
for k, channel in list(bp.SESSION_CHANNELS.items()):
if channel.reaper_should_collect(now):
bp.SESSION_CHANNELS.pop(k, None)
assert bp.get_session_channel(sid) is None
def test_subscribe_to_session_channel_is_atomic_get_create_subscribe():
"""The atomic helper returns a registered channel with the subscriber
already attached, in one SESSION_CHANNELS_LOCK critical section.
First call creates; second call reuses the same instance.
"""
from api import background_process as bp
sid = "sess-atomic-subscribe"
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
try:
ch, q = bp.subscribe_to_session_channel(sid)
# Channel is registered and the slot is already counted.
assert bp.get_session_channel(sid) is ch
assert ch.subscriber_count() == 1
# An emit reaches our queue immediately — proves we're on the live channel.
ch.emit("bg_task_complete", {"n": 1})
assert q.get_nowait() == ("bg_task_complete", {"n": 1})
# Second call reuses the same instance and adds a second subscriber.
ch2, q2 = bp.subscribe_to_session_channel(sid)
assert ch2 is ch
assert ch.subscriber_count() == 2
ch.unsubscribe(q2)
finally:
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
def test_subscribe_to_session_channel_survives_concurrent_reaper():
"""Regression for PR #2971 Greptile P1 (background_process.py:215).
Reproduces the reaper TOCTOU: an idle, past-grace channel exists in the
registry; a subscriber arrives while the reaper sweeps concurrently. With
the old split ``get_or_create_session_channel()`` + ``ch.subscribe()`` the
reaper could collect the channel in the gap, orphaning the subscriber so
later emits never reach its queue. The atomic helper holds
SESSION_CHANNELS_LOCK across both steps, so the post-subscribe registry
entry must be the EXACT channel the subscriber is attached to, and a
subsequent emit must be delivered.
"""
import threading
from api import background_process as bp, config as cfg
sid = "sess-reaper-toctou"
# Seed an idle channel that is already eligible for collection (no subs,
# drop time pushed well past the grace window) — the dangerous precondition.
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
seed = bp.SessionChannel(sid)
seed.last_subscriber_drop_at = (
time.time() - (cfg.SESSION_CHANNEL_SUBSCRIBER_GRACE_SECS + 5)
)
bp.SESSION_CHANNELS[sid] = seed
stop = threading.Event()
def _reaper_spin():
# Hammer the exact reaper critical section concurrently.
while not stop.is_set():
now = time.time()
with bp.SESSION_CHANNELS_LOCK:
for k, channel in list(bp.SESSION_CHANNELS.items()):
if channel.reaper_should_collect(now):
bp.SESSION_CHANNELS.pop(k, None)
t = threading.Thread(target=_reaper_spin, daemon=True)
t.start()
try:
for _ in range(200):
ch, q = bp.subscribe_to_session_channel(sid)
# Post-condition: the channel we're subscribed to is the one in the
# registry (atomicity guarantee). The reaper cannot have evicted it
# between create and subscribe because both ran under the lock and
# the channel now has a live subscriber.
assert bp.get_session_channel(sid) is ch
assert ch.subscriber_count() >= 1
# And an emit on the registry-resolved channel reaches our queue —
# i.e. we are NOT orphaned on a collected channel.
resolved = bp.get_session_channel(sid)
resolved.emit("bg_task_complete", {"ping": 1})
assert q.get(timeout=1.0) == ("bg_task_complete", {"ping": 1})
ch.unsubscribe(q)
finally:
stop.set()
t.join(timeout=2.0)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
# ---------------------------------------------------------------------------
# Dual emit: STREAMS empty + SESSION_CHANNELS has subscriber → delivered
# ---------------------------------------------------------------------------
def test_emit_when_idle_between_turns_session_channel_delivers():
"""No STREAMS but a SESSION_CHANNELS subscriber → event reaches the channel."""
from api import background_process as bp, config as cfg
sid = "sess-between-turns"
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
try:
# Make sure STREAMS has no relevant entry
with cfg.STREAMS_LOCK:
assert all(
(cfg.ACTIVE_RUNS.get(stream_id) or {}).get("session_id") != sid
for stream_id in cfg.STREAMS
)
evt = {
"type": "completion",
"session_id": "proc-99",
"session_key": sid,
"command": "sleep 1",
"exit_code": 0,
"output": "ok",
}
bp.register_process_session(sid, sid)
try:
bp._process_one(evt)
event_name, data = q.get(timeout=2.0)
assert event_name == "bg_task_complete"
assert data["session_id"] == sid
assert data["task_id"] == "proc-99"
assert data.get("event_id"), "emitter must stamp event_id (Q4 contract)"
finally:
bp.unregister_process_session(sid)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(sid, None)
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
def test_emit_during_busy_turn_dual_emits_to_both():
"""STREAMS + SESSION_CHANNELS both subscribed → both receive."""
from api import background_process as bp, config as cfg
sid = "sess-busy-dual"
stream_id = "stream-busy-dual"
streams_received: list = []
class _FakeStreamChannel:
def put_nowait(self, item):
streams_received.append(item)
with cfg.STREAMS_LOCK:
cfg.STREAMS[stream_id] = _FakeStreamChannel()
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
bp.register_process_session(sid, sid)
try:
evt = {
"type": "completion",
"session_id": "proc-busy-1",
"session_key": sid,
"command": "sleep 1",
"exit_code": 0,
"output": "ok",
}
bp._process_one(evt)
# Both surfaces received the event (frontend will dedupe by process_id)
assert streams_received, "STREAMS subscriber must receive in-turn delivery"
event_name, data = streams_received[0]
assert event_name == "bg_task_complete"
assert data["task_id"] == "proc-busy-1"
assert data.get("event_id"), "emitter must stamp event_id (Q4 contract)"
ev2, data2 = q.get(timeout=2.0)
assert ev2 == "bg_task_complete"
assert data2["task_id"] == "proc-busy-1"
assert data2.get("event_id"), "emitter must stamp event_id (Q4 contract)"
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
with cfg.STREAMS_LOCK:
cfg.STREAMS.pop(stream_id, None)
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(sid, None)
# ---------------------------------------------------------------------------
# Route + frontend wiring (source-grep)
# ---------------------------------------------------------------------------
def test_routes_registers_session_stream_endpoint():
src = (REPO_ROOT / "api" / "routes.py").read_text()
assert "/api/session/stream" in src
assert "_handle_session_sse_stream" in src
def test_routes_session_sse_uses_session_channel_subscribe():
src = (REPO_ROOT / "api" / "routes.py").read_text()
# The handler must use the atomic get-or-create+subscribe helper (closes
# the PR #2971 reaper TOCTOU race) and release the slot on every exit path.
assert "subscribe_to_session_channel" in src
assert "ch.unsubscribe(q)" in src
# The split get-then-subscribe call pair must NOT come back — it reopens
# the race the atomic helper exists to close.
assert "ch = get_or_create_session_channel(sid)" not in src
def test_server_starts_session_channel_reaper():
src = (REPO_ROOT / "server.py").read_text()
assert "start_session_channel_reaper" in src
assert "stop_session_channel_reaper" in src
def test_frontend_opens_session_stream():
js = (REPO_ROOT / "static" / "messages.js").read_text()
assert "api/session/stream?session_id=" in js
assert "startSessionStream" in js
assert "stopSessionStream" in js
def test_frontend_busy_race_gate_obsoleted_by_option_z_pivot():
"""Per the Option Z PIVOT note baked into the handler body, the browser
is no longer in the wakeup path at all — the server-side drain owns
starting the next turn. The original ``if (S.busy)`` busy-race gate
inside the handler was paired with the now-removed re-POST of
``/api/chat/stream``; once the re-POST went away (Option Z), the gate
became moot. We assert the pivot documentation is in place so a future
refactor doesn't silently re-introduce the gate without re-introducing
the re-POST as well."""
js = (REPO_ROOT / "static" / "messages.js").read_text()
fn_ix = js.index("function _handleBgTaskCompleteEvent")
fn_src = js[fn_ix:fn_ix + 2400]
assert "Option Z PIVOT" in fn_src
assert "drain thread" in fn_src
# The legacy re-POST and busy-race gate must NOT be inside the handler.
assert "if (S.busy)" not in fn_src
assert "/api/chat/stream" not in fn_src
def test_frontend_shared_handler_dedupes_across_paths():
"""Module-scope dedupe ring buffer (Map+TTL keyed (sid, event_id)) is what makes dual-emit safe."""
js = (REPO_ROOT / "static" / "messages.js").read_text()
# Module-scope Map+TTL declaration outside any `function () { ... }` body
assert "const _bgTaskCompleteSeenIds = new Map();" in js
assert "const _BG_TASK_COMPLETE_TTL_MS = 60000;" in js
assert "const _BG_TASK_COMPLETE_CAP = 256;" in js
assert "_bgTaskCompleteRingBufferAdd" in js
assert "_handleBgTaskCompleteEvent" in js
# The handler must require event_id (server contract surface per Q4).
assert "if (!evt_id) return;" in js or "if(!evt_id) return;" in js
def test_sessions_js_starts_and_stops_session_stream_on_mount_unmount():
js = (REPO_ROOT / "static" / "sessions.js").read_text()
assert "startSessionStream(S.session.session_id)" in js
assert "stopSessionStream" in js
# ---------------------------------------------------------------------------
# Regression: notify_on_complete event shape carries NO session_key
# ---------------------------------------------------------------------------
#
# Root cause of "zero ack POSTs ever" (t_0f447014):
# tools.process_registry.ProcessRegistry._move_to_finished() enqueues the
# completion event for a notify_on_complete process WITHOUT a "session_key"
# field (only the watch_match enqueue includes one). _process_one then does
# `session_key = evt.get("session_key") or process_id`, so it falls back to
# the process id ("proc_xxxx"), which is NEVER a key in
# PROCESS_SESSION_INDEX (only webui_session_id -> webui_session_id is
# registered). The lookup misses → silent debug drop → no SSE emit ever.
#
# The existing happy-path test (test_emit_when_idle_between_turns_...) masks
# this because it hand-builds evt WITH "session_key": sid — a shape the real
# completion enqueue never produces. This test drives the event through the
# REAL process_registry.completion_queue using the EXACT dict shape
# _move_to_finished() produces, so it fails on the unfixed code and passes
# after the fix.
def test_real_completion_event_shape_routes_to_session_channel():
"""A completion event in the real _move_to_finished() shape (no
session_key) must still route to the registered WebUI session.
Faithfully reproduces production: the terminal tool spawns a background
process with session_key == webui_session_id (captured synchronously at
spawn while the turn env is live). On exit, ProcessRegistry._move_to_
finished() (1) moves the ProcessSession into _finished — which retains
session_key — then (2) enqueues a completion event that DOES NOT carry a
"session_key" field. The drain (_process_one) must recover the session_key
from the still-tracked ProcessSession in the registry.
"""
import time as _t
from api import background_process as bp, config as cfg
pytest.importorskip("tools.process_registry", reason="hermes-agent not installed")
from tools.process_registry import process_registry, ProcessSession
webui_sid = "sess-real-completion-shape"
proc_id = "proc_realshape0001"
ch = bp.get_or_create_session_channel(webui_sid)
q = ch.subscribe()
# streaming.py binds key == webui_session_id (register_process_session
# called with (session_id, session_id)). HERMES_SESSION_KEY for the
# spawned child therefore equals webui_sid, and the terminal tool stamps
# that onto ProcessSession.session_key at spawn time.
bp.register_process_session(webui_sid, webui_sid)
# Simulate the finished process the registry retains in _finished after
# _move_to_finished(): it carries the spawn-time session_key.
finished = ProcessSession(
id=proc_id,
command="pytest -q",
session_key=webui_sid,
started_at=_t.time(),
exited=True,
exit_code=0,
notify_on_complete=True,
)
with process_registry._lock:
process_registry._finished[proc_id] = finished
# Build the event EXACTLY as ProcessRegistry._move_to_finished() enqueues
# it for notify_on_complete: type/session_id/command/exit_code/output.
# Crucially: NO "session_key" key. This is the real wire shape that the
# existing happy-path test (test_emit_when_idle_between_turns_...) never
# exercises because it hand-injects session_key.
evt = {
"type": "completion",
"session_id": proc_id,
"command": "pytest -q",
"exit_code": 0,
"output": "1197 passed",
}
process_registry.completion_queue.put(evt)
drained = process_registry.completion_queue.get(timeout=2.0)
try:
bp._process_one(drained)
event_name, data = q.get(timeout=2.0)
assert event_name == "bg_task_complete"
assert data["session_id"] == webui_sid
assert data["task_id"] == proc_id
assert data.get("event_id"), "emitter must stamp event_id (Q4 contract)"
# Per the Q1 minimal-payload trim settled in #2242, the SSE payload
# no longer carries ``wakeup_prompt`` (or ``command`` / ``exit_code``).
# The optional ``summary`` field is now the only human-readable
# surface; when the synthetic wakeup body is available the emitter
# derives a short first-line summary from it.
assert "wakeup_prompt" not in data
assert "command" not in data
assert "exit_code" not in data
summary = data.get("summary")
if summary is not None:
assert isinstance(summary, str)
assert "IMPORTANT" in summary or "Background process" in summary
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(webui_sid, None)
bp.unregister_process_session(webui_sid)
with process_registry._lock:
process_registry._finished.pop(proc_id, None)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(webui_sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(webui_sid, None)
# ===========================================================================
# Option Z (PIVOT): server-side wakeup is the PRIMARY mechanism.
#
# The drain thread starts the agent turn directly server-side
# (api/background_process._start_server_side_wakeup_turn →
# api.routes.start_session_turn) with NO browser round-trip. The per-session
# SSE channel is demoted to a pure live-view layer. These tests prove:
# 1. closed-tab (no SSE subscriber at all) STILL starts a server-side turn
# 2. active-turn defers (no double-start; PR #2279 next-turn drain handles it)
# 3. one wakeup per process_id (dedupe)
# 4. open tab still sees the live SSE frame (live-view unchanged)
# ===========================================================================
def _wait_for_wakeup(holder, timeout=3.0):
"""Thin wrapper preserving the legacy local name; the body lives in
``tests/_wakeup_helpers.py`` and is shared with
``test_wakeup_defer_race.py`` (Copilot PR #2971 r3305700944).
"""
from tests._wakeup_helpers import wait_for_wakeup as _impl
return _impl(holder, timeout=timeout)
def _install_fake_start_session_turn(monkeypatch, *, status=200):
"""Thin wrapper preserving the legacy local name; the body lives in
``tests/_wakeup_helpers.py`` and is shared with
``test_wakeup_defer_race.py`` (Copilot PR #2971 r3305700944).
"""
from tests._wakeup_helpers import install_fake_start_session_turn as _impl
return _impl(monkeypatch, status=status)
def test_server_side_wakeup_when_idle_no_tab(monkeypatch):
"""THE headline test: no SSE subscriber at all (closed tab / never opened).
Pushing a completion must still start a server-side turn for the session.
This is the closed-tab case browser-mediated wakeup could never serve.
"""
from api import background_process as bp, config as cfg
sid = "sess-optz-idle-notab"
proc_id = "proc-optz-idle-1"
holder = _install_fake_start_session_turn(monkeypatch)
bp.register_process_session(sid, sid)
try:
# Deliberately NO ch.subscribe() and NO STREAMS entry: nobody is
# listening. ACTIVE_RUNS has no row for sid → session is idle.
assert bp.get_session_channel(sid) is None
assert bp._session_has_active_turn(sid) is False
evt = {
"type": "completion",
"session_id": proc_id,
"session_key": sid,
"command": "sleep 8",
"exit_code": 0,
"output": "done",
}
bp._process_one(evt)
assert _wait_for_wakeup(holder), (
"server-side wakeup turn was NOT started for an idle session with "
"no tab — closed-tab case is broken"
)
assert len(holder["calls"]) == 1
call = holder["calls"][0]
assert call["session_id"] == sid
assert call["source"] == "process_wakeup"
assert call["message"].startswith("[IMPORTANT: Background process")
finally:
bp.unregister_process_session(sid)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(sid, None)
def test_server_side_wakeup_deferred_when_turn_active(monkeypatch):
"""A foreground turn is active (ACTIVE_RUNS has a row for the session) →
the drain must NOT start a second turn. The PENDING_BG_TASK_COMPLETIONS
marker is left for PR #2279's next-turn drain.
"""
from api import background_process as bp, config as cfg
sid = "sess-optz-active-defer"
proc_id = "proc-optz-active-1"
stream_id = "stream-optz-active-1"
holder = _install_fake_start_session_turn(monkeypatch)
bp.register_process_session(sid, sid)
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
try:
assert bp._session_has_active_turn(sid) is True
evt = {
"type": "completion",
"session_id": proc_id,
"session_key": sid,
"command": "sleep 8",
"exit_code": 0,
"output": "done",
}
bp._process_one(evt)
# Give any (incorrectly spawned) runner thread a chance to fire.
fired = holder["event"].wait(timeout=1.0)
assert fired is False, (
"server-side wakeup must DEFER when a turn is active — it "
"double-started a turn"
)
assert holder["calls"] == []
# Marker must remain so the next-turn drain delivers it.
assert sid in cfg.PENDING_BG_TASK_COMPLETIONS
finally:
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(sid, None)
def test_wakeup_dedupe_once_per_process(monkeypatch):
"""The same process_id delivered twice (kill_process racing the reader
thread) must wake the agent at most once.
"""
from api import background_process as bp, config as cfg
sid = "sess-optz-dedupe"
proc_id = "proc-optz-dedupe-1"
holder = _install_fake_start_session_turn(monkeypatch)
bp.register_process_session(sid, sid)
try:
evt = {
"type": "completion",
"session_id": proc_id,
"session_key": sid,
"command": "sleep 8",
"exit_code": 0,
"output": "done",
}
bp._process_one(evt)
assert _wait_for_wakeup(holder)
# Second delivery of the SAME process_id — must be deduped before the
# server-side wakeup branch.
bp._process_one(dict(evt))
time.sleep(0.5)
assert len(holder["calls"]) == 1, (
"duplicate completion for the same process_id woke the agent twice"
)
finally:
bp.unregister_process_session(sid)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(sid, None)
def test_open_tab_sees_live_stream(monkeypatch):
"""Live-view still works: with a subscribed per-session SSE channel, the
bg_task_complete frame is still delivered to the tab (so the open tab can
render the server-initiated turn live). Server-side wakeup is additive —
it does not remove the SSE emit.
"""
from api import background_process as bp, config as cfg
sid = "sess-optz-liveview"
proc_id = "proc-optz-liveview-1"
holder = _install_fake_start_session_turn(monkeypatch)
ch = bp.get_or_create_session_channel(sid)
q = ch.subscribe()
bp.register_process_session(sid, sid)
try:
evt = {
"type": "completion",
"session_id": proc_id,
"session_key": sid,
"command": "sleep 8",
"exit_code": 0,
"output": "done",
}
bp._process_one(evt)
# Live-view: the open tab still receives the SSE frame.
event_name, data = q.get(timeout=2.0)
assert event_name == "bg_task_complete"
assert data["session_id"] == sid
assert data["task_id"] == proc_id
assert data.get("event_id"), "emitter must stamp event_id (Q4 contract)"
# AND the server-side wakeup still started (no active turn here).
assert _wait_for_wakeup(holder)
assert holder["calls"][0]["session_id"] == sid
finally:
ch.unsubscribe(q)
with bp.SESSION_CHANNELS_LOCK:
bp.SESSION_CHANNELS.pop(sid, None)
bp.unregister_process_session(sid)
cfg.PENDING_BG_TASK_COMPLETIONS.discard(sid)
cfg.BG_TASK_COMPLETE_EVENTS_SEEN.pop(sid, None)
# ---------------------------------------------------------------------------
# event_id contract surface — backend emitter must stamp every event
# ---------------------------------------------------------------------------
def test_backend_emitter_stamps_event_id_on_every_bg_task_complete():
"""Per the #2242 Q4 reply: every bg_task_complete emit carries an
event_id; the consumer's ring-buffer dedupe is keyed on it. Source-grep
the payload builder to confirm event_id is stamped."""
src = (REPO_ROOT / "api" / "background_process.py").read_text()
# Locate the canonical payload builder and confirm event_id is in the dict.
fn_ix = src.index("def _build_payload")
fn_src = src[fn_ix:fn_ix + 4000]
assert '"event_id"' in fn_src or "'event_id'" in fn_src, (
"payload builder must stamp event_id on every bg_task_complete payload"
)
assert "uuid.uuid4().hex" in fn_src, (
"event_id should be a per-emit uuid hex (R2 §Q1)"
)
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# Regression (Greptile P1 r3371970195): a CLOSED (readyState===2) session
# EventSource must trigger a FRESH reconnect, not silently die.
# ---------------------------------------------------------------------------
#
# Bug: startSessionStream's top guard is
# `if (_sessionStreamSessionId === sid && _sessionEventSource) return;`
# When onerror fired with readyState === 2 (permanent close: server 4xx/204,
# browser retry-exhaustion), the old code scheduled the reconnect timer WITHOUT
# clearing _sessionEventSource. The still-non-null (but CLOSED) object made the
# guard short-circuit, so stopSessionStream() was never reached and no new
# EventSource was ever created — the session stream stayed dead until the user
# navigated away and back, dropping all bg_task_complete notifications meanwhile.
#
# Fix: in onerror, when es.readyState === 2 and es is still the active source,
# close it and null _sessionEventSource BEFORE arming the reconnect timer, so
# the deferred startSessionStream() passes its guard, runs stopSessionStream(),
# and builds a fresh EventSource. The `_sessionEventSource === es` identity
# check prevents a stale onerror from a superseded stream stomping a newer live
# connection.
def test_session_stream_onerror_clears_closed_source_so_reconnect_proceeds():
js = (REPO_ROOT / "static" / "messages.js").read_text()
# Isolate the onerror handler body within startSessionStream.
fn_ix = js.index("function startSessionStream")
err_ix = js.index("es.onerror", fn_ix)
onerror_src = js[err_ix:err_ix + 1600]
# Must only act on the permanently-CLOSED state.
assert "es.readyState === 2" in onerror_src
# Identity-guard so a stale onerror can't stomp a newer live connection.
assert "_sessionEventSource === es" in onerror_src
# Must drop the dead reference (and close it) BEFORE arming the timer so
# startSessionStream's "already connected" guard no longer short-circuits.
assert "_sessionEventSource = null;" in onerror_src
assert "es.close()" in onerror_src
# Ordering: the null-out must precede the setTimeout that re-opens.
null_pos = onerror_src.index("_sessionEventSource = null;")
timer_pos = onerror_src.index("setTimeout(")
assert null_pos < timer_pos, (
"must null the closed EventSource BEFORE arming the reconnect timer, "
"else startSessionStream's guard short-circuits and the stream stays dead"
)
# ---------------------------------------------------------------------------
# Regression (Greptile P1 r3377162160): session stream must NOT stay dead
# after a failed / early-returned loadSession (sessions.js:754 re-arm).
# ---------------------------------------------------------------------------
#
# Bug: loadSession() tears down the live per-session SSE unconditionally at the
# top — `if (typeof stopSessionStream==='function') stopSessionStream();` — but
# only RE-arms it on the success path (`startSessionStream(S.session.session_id)`
# near the end). Every early-return exit leaves the session the user is still
# viewing with _sessionEventSource === null and no path back to a live stream:
# - fetch error (network / non-404) → returns after stopSessionStream
# - api() returned undefined (401 redirect) → returns
# - stale-load race (_loadingSessionId !== sid after data) → returns
# - same-session no-op guard (currentSid===sid && !forceReload) → returns
# BEFORE the teardown, but a *prior* failed load already nulled the source,
# and re-selecting the same session would otherwise no-op forever.
# Net effect: bg_task_complete delivery silently dies until a full page reload.
#
# Fix: an idempotent helper `_rearmActiveSessionStream()` calls
# startSessionStream(S.session.session_id) for whatever session is actually on
# screen, invoked on the early-return paths AND the same-session no-op guard.
# startSessionStream() is idempotent (its top guard
# `_sessionStreamSessionId === sid && _sessionEventSource` no-ops when already
# live) so the success path is never double-armed. The fetch-error path keeps
# its own pre-existing guarded restart (`_selfHealedCurrent` check) instead of
# the helper, because only there can the current session have just self-healed
# away — re-arming a 404'd/deleted session_id would spin the SSE reconnect loop
# against a dead session. Mirrors the #2979 messages.js reconnect fix.
def test_load_session_rearms_stream_on_every_early_return():
js = (REPO_ROOT / "static" / "sessions.js").read_text()
# The idempotent re-arm helper must exist and arm the on-screen session.
assert "function _rearmActiveSessionStream(" in js, (
"expected a dedicated idempotent re-arm helper"
)
helper_ix = js.index("function _rearmActiveSessionStream(")
helper_src = js[helper_ix:helper_ix + 400]
assert "S.session" in helper_src and "startSessionStream(" in helper_src, (
"helper must (re)arm startSessionStream for the currently-shown S.session"
)
# Isolate the loadSession body.
fn_ix = js.index("async function loadSession(")
body = js[fn_ix:fn_ix + 12000]
# The unconditional teardown must still be there (this is what creates the
# dead-stream window the re-arm closes).
assert "stopSessionStream()" in body
# Post-teardown early-return paths must re-arm. The helper covers the
# same-session guard, the undefined-data (401) exit, and the stale-response
# exit — 3 helper call sites is the floor. (The fetch-error path uses its
# own `_selfHealedCurrent`-guarded restart, asserted separately below; the
# rapid-switch post-draft handoff is owned by the newer load's own arming.)
assert js.count("_rearmActiveSessionStream()") >= 3, (
"each failed/early-return loadSession exit after stopSessionStream() "
"must re-arm the on-screen session's stream, else bg_task_complete "
"delivery dies until a page reload (Greptile P1 r3377162160)"
)
# Specifically: the same-session no-op guard must be PRECEDED by a re-arm
# so re-selecting a session whose stream a prior failed load killed revives
# it. The re-arm sits before the guard (not inside a wrapping block) so the
# guard stays the exact one-liner other tests assert; it's idempotent so
# the real-switch path is unaffected.
guard_ix = body.index("currentSid===sid && !forceReload && !_loadingSessionId")
pre_guard = body[max(0, guard_ix - 600):guard_ix]
assert "_rearmActiveSessionStream()" in pre_guard, (
"a re-arm must run before the same-session no-op guard so a "
"previously-killed stream is revived on re-selecting the session"
)
# The fetch-error catch must restart the stream for the on-screen session,
# but guarded against the self-healed-current (404'd) case so it never
# spins the reconnect loop against a dead session_id.
catch_ix = body.index("const _selfHealedCurrent")
catch_src = body[catch_ix:catch_ix + 1400]
assert "!_selfHealedCurrent" in catch_src and "startSessionStream(currentSid)" in catch_src, (
"fetch-error path must restart the on-screen stream, guarded against "
"the self-healed-current (deleted/404) session"
)
def test_session_sse_stream_unsubscribes_on_header_write_failure():
"""Deep-review fix (Codex): in _handle_session_sse_stream the subscriber
slot is acquired by subscribe_to_session_channel BEFORE the SSE headers are
written. The header writes (send_response/send_header/end_headers) touch the
socket and can raise a client-disconnect error. If that happened OUTSIDE the
try/finally, ch.unsubscribe(q) would be skipped and — because
reaper_should_collect refuses to collect a channel with sub_count>0 — the
channel would zombie forever. Pin that the subscribe and the header setup
both sit inside the single try whose finally unsubscribes.
"""
from pathlib import Path
src = Path(__file__).resolve().parents[1].joinpath("api", "routes.py").read_text(encoding="utf-8")
i = src.find("def _handle_session_sse_stream(")
assert i != -1, "handler not found"
j = src.find("\ndef ", i + 1)
body = src[i:j]
sub_ix = body.find("subscribe_to_session_channel(")
assert sub_ix != -1, "subscribe call not found"
try_ix = body.find("try:", sub_ix)
end_headers_ix = body.find("end_headers()", sub_ix)
finally_ix = body.find("finally:", sub_ix)
unsub_ix = body.find("ch.unsubscribe(q)", finally_ix if finally_ix != -1 else sub_ix)
# Order must be: subscribe → try → end_headers (inside try) → finally → unsubscribe.
assert try_ix != -1 and finally_ix != -1 and unsub_ix != -1
assert sub_ix < try_ix < end_headers_ix < finally_ix < unsub_ix, (
"header setup must run INSIDE the try/finally that unsubscribes — "
"a header-write disconnect must not leak a SessionChannel subscriber"
)

View File

@@ -0,0 +1,133 @@
"""Regression: GET /api/session display resolvers must never trigger the
live provider-catalog rebuild.
Root cause (multi-tab streaming interlock RCA, task t_d127953d):
``_resolve_effective_session_model_for_display`` /
``_resolve_effective_session_model_provider_for_display`` are called by the
hot, side-effect-free ``GET /api/session?...&resolve_model=1`` path. When a
session has no persisted ``model_provider`` (common — e.g. kanban/imported
sessions), the fast path in ``_resolve_compatible_session_model_state`` is
skipped and the resolver fell through to ``get_available_models()`` WITHOUT
``prefer_cache``. On a non-AWS / WSL / corp network that cold rebuild blocks
~10s on a botocore IMDS probe (plus anthropic/openrouter /models) and, run
concurrently across browser tabs, serializes on the models-cache lock and
starves SSE/streaming -> BrokenPipe/Cancelled storm.
This is an INVARIANT test, not a change-detector: it asserts the resolvers
resolve from the cache-only path and never reach the live-rebuild seam
``api.config._invoke_models_rebuild`` — regardless of whether the session
carries a model_provider.
"""
import ast
import inspect
import pytest
import api.config as cfg
import api.routes as routes
class _FakeSession:
"""Minimal stand-in for a Session row as seen by the display resolvers."""
def __init__(self, model, model_provider):
self.model = model
self.model_provider = model_provider
@pytest.fixture
def cold_models_cache(monkeypatch):
"""Force a cold in-memory + disk models cache without touching real state.
Cold cache is what makes the regression observable: a warm cache short-
circuits before any rebuild decision, hiding the prefer_cache contract.
"""
monkeypatch.setattr(cfg, "_available_models_cache", None, raising=False)
monkeypatch.setattr(cfg, "_available_models_cache_ts", 0.0, raising=False)
monkeypatch.setattr(
cfg, "_available_models_cache_source_fingerprint", None, raising=False
)
monkeypatch.setattr(cfg, "_cache_build_in_progress", False, raising=False)
# Never read/write the real on-disk cache during the test.
monkeypatch.setattr(cfg, "_load_models_cache_from_disk", lambda: None)
monkeypatch.setattr(cfg, "_save_models_cache_to_disk", lambda *_a, **_k: None)
monkeypatch.setattr(cfg, "_delete_models_cache_on_disk", lambda: None)
yield
@pytest.fixture
def rebuild_seam_tripwire(monkeypatch):
"""Make the live provider-catalog rebuild seam fail loudly if reached.
``_invoke_models_rebuild`` is the documented indirection seam around the
cold, network-touching per-provider rebuild. The display resolvers must
never reach it (prefer_cache returns the network-free minimal catalog
*before* this seam). If a future edit drops ``prefer_cached_catalog=True``,
the resolver falls into the cold rebuild and trips this wire.
"""
calls = {"n": 0}
def _boom(_builder):
calls["n"] += 1
raise AssertionError(
"live provider-catalog rebuild ran on the hot GET /api/session "
"display path — prefer_cached_catalog regression"
)
monkeypatch.setattr(cfg, "_invoke_models_rebuild", _boom)
return calls
@pytest.mark.parametrize(
"model_provider",
[None, "", "anthropic"],
ids=["no-provider", "empty-provider", "with-provider"],
)
def test_session_display_resolvers_never_trigger_live_rebuild(
cold_models_cache, rebuild_seam_tripwire, model_provider
):
session = _FakeSession("claude-opus-4-7", model_provider)
# Must not raise (the tripwire raises AssertionError if the live rebuild
# path is entered) and must return the persisted model verbatim.
model = routes._resolve_effective_session_model_for_display(session)
provider = routes._resolve_effective_session_model_provider_for_display(session)
assert model == "claude-opus-4-7"
# provider is best-effort; the contract under test is "no live rebuild",
# not a specific provider string. It must at least be None or a str.
assert provider is None or isinstance(provider, str)
assert rebuild_seam_tripwire["n"] == 0
def _has_prefer_cached_catalog_true_call(fn) -> bool:
tree = ast.parse(inspect.getsource(fn))
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
if not isinstance(node.func, ast.Name):
continue
if node.func.id != "_resolve_compatible_session_model_state":
continue
for keyword in node.keywords:
if keyword.arg == "prefer_cached_catalog" and isinstance(
keyword.value, ast.Constant
):
return keyword.value.value is True
return False
def test_resolver_signature_passes_prefer_cached_catalog():
"""Static guard: both resolvers must opt into the cache-only catalog.
A pure behavioural test can be satisfied by an unrelated short-circuit;
this pins the explicit contract at the call site so the intent survives
refactors.
"""
assert _has_prefer_cached_catalog_true_call(
routes._resolve_effective_session_model_for_display
)
assert _has_prefer_cached_catalog_true_call(
routes._resolve_effective_session_model_provider_for_display
)

View File

@@ -0,0 +1,130 @@
"""start_session_turn honors runtime_adapter_enabled() via the shared
_start_run helper — Q-2979-A2 / Copilot discussion_r3305864087/r3305864173.
Before this fix start_session_turn (Option Z drain-thread wakeup entrypoint)
called _start_chat_stream_for_session directly, bypassing the runtime-adapter
selection block that /api/chat/start (_handle_chat_start) already ran. As a
result a process-wakeup turn skipped the adapter that a human-typed turn
would have hit when ``HERMES_RUNTIME_ADAPTER=legacy-journal`` was set.
The refactor factors a shared ``_start_run`` helper used by both entrypoints,
so flipping the env to ``legacy-journal`` now routes process-wakeup turns
through the same LegacyJournalRuntimeAdapter as the browser path.
These tests exercise that contract directly without spinning up a real HTTP
server (precedent: tests/test_wakeup_defer_race.py — monkeypatch the heavy
deps, call the function under test, assert on adapter selection).
"""
from __future__ import annotations
import types
import pytest
@pytest.fixture
def _stub_routes(monkeypatch):
"""Patch the heavy deps inside start_session_turn so the call collapses
to one observable: did it go through the runtime adapter or not."""
from api import routes as routes_mod
# 1. Fake session lookup — returns a minimal object with the attributes
# _start_run touches via the s.* attribute names.
s = types.SimpleNamespace(
session_id="sess-test",
model="opus",
model_provider="anthropic",
profile="developer-general",
workspace="/tmp/ws-test",
)
monkeypatch.setattr(routes_mod, "get_session", lambda _sid: s)
# 2. Workspace resolution — short-circuit to the persisted workspace.
monkeypatch.setattr(
routes_mod,
"_resolve_chat_workspace_with_recovery",
lambda _s, _req: "/tmp/ws-test",
)
# 3. Model resolution — pass through.
monkeypatch.setattr(
routes_mod,
"_resolve_compatible_session_model_state",
lambda model, provider, profile_provider=None, profile_default_model=None, prefer_cached_catalog=False: (model, provider, model),
)
# 4. Block the per-session live-view fan-out (it pokes a real registry).
monkeypatch.setattr(
routes_mod,
"_start_chat_stream_for_session",
lambda *a, **kw: {"_status": 200, "stream_id": "stream-direct", "session_id": "sess-test"},
)
# 5. Silence the channel emit.
import api.background_process as bp_mod
monkeypatch.setattr(bp_mod, "get_session_channel", lambda _sid: None)
return routes_mod
def test_start_session_turn_uses_direct_path_by_default(_stub_routes, monkeypatch):
"""With HERMES_WEBUI_RUNTIME_ADAPTER unset (legacy-direct default), the helper
must NOT go through the adapter — it falls through to the direct
_start_chat_stream_for_session call, same as before."""
monkeypatch.delenv("HERMES_WEBUI_RUNTIME_ADAPTER", raising=False)
calls = {"adapter": 0}
from api import runtime_adapter as ra_mod
real_build = ra_mod.build_runtime_adapter
def _track(*a, **kw):
calls["adapter"] += 1
return real_build(*a, **kw)
monkeypatch.setattr(ra_mod, "build_runtime_adapter", _track)
resp = _stub_routes.start_session_turn("sess-test", "wakeup msg")
assert resp["_status"] == 200
assert calls["adapter"] == 0, "default mode must not build a runtime adapter"
def test_start_session_turn_routes_through_adapter_when_enabled(
_stub_routes, monkeypatch
):
"""With HERMES_WEBUI_RUNTIME_ADAPTER=legacy-journal, start_session_turn must
construct + invoke the LegacyJournalRuntimeAdapter — same path
_handle_chat_start exercises. This is the regression that Q-2979-A2 fixes:
before the _start_run refactor, this env flip had no effect on the
process-wakeup path."""
monkeypatch.setenv("HERMES_WEBUI_RUNTIME_ADAPTER", "legacy-journal")
from api import runtime_adapter as ra_mod
invoked = {"adapter": 0, "start_run": 0}
class _SpyAdapter:
def start_run(self, request):
invoked["start_run"] += 1
assert request.session_id == "sess-test"
assert request.message == "wakeup msg"
assert request.source == "process_wakeup"
assert request.metadata == {"route": "start_session_turn"}
return ra_mod.RunStartResult(
run_id="run-test",
stream_id="stream-via-adapter",
session_id=request.session_id,
payload={"_status": 200, "stream_id": "stream-via-adapter"},
)
def _fake_build(**kw):
invoked["adapter"] += 1
return _SpyAdapter()
monkeypatch.setattr(ra_mod, "build_runtime_adapter", _fake_build)
resp = _stub_routes.start_session_turn("sess-test", "wakeup msg")
assert resp["_status"] == 200
assert resp["stream_id"] == "stream-via-adapter"
assert invoked == {"adapter": 1, "start_run": 1}

View File

@@ -0,0 +1,621 @@
"""Defer-path wakeup race: a fast background task that completes WHILE a turn
is tearing down must still wake an autonomous agent.
Root cause (proven by source, not speculation):
- api/background_process.py:_process_one defer branch — when a completion
arrives and _session_has_active_turn(session_id) is True (ACTIVE_RUNS has
a row), Option Z CANNOT start a turn (start_session_turn would 409). Before
this fix it only logged + left a bare PENDING_BG_TASK_COMPLETIONS session
flag; the wakeup_prompt was DISCARDED.
- The only consumer of that bare flag was the PR #2279 next-turn drain
(api/streaming._drain_webui_process_notifications, called at
streaming.py:3445 inside the turn pipeline). It reads completion_queue —
which the Option Z drain thread already emptied — and is gated by
BG_TASK_COMPLETE_EVENTS_SEEN / registry _completion_consumed (both set in
_process_one BEFORE the defer). So even a user turn could not recover it.
- For an AUTONOMOUS agent there is NO next user turn, so the deferred wakeup
was lost forever. A SLOW task (5s) completes AFTER teardown finished →
idle path → fires (Test A passed). A FAST task (2s) completes INSIDE the
teardown window (between "agent finished output" and ACTIVE_RUNS cleared)
→ defer → lost (Test B failed). Exactly matches A-success / B-fail.
The fix persists the prompt at defer time (DEFERRED_PROCESS_WAKEUPS) and a
turn-teardown idle-hook (drain_deferred_wakeups_for_session, invoked from
streaming.py right after unregister_active_run) redelivers it once the session
goes idle — symmetric with the idle branch. claim_deferred_wakeups pops
atomically, so delivery is exactly-once (no double-fire, no wakeup loop).
These tests simulate the drain-thread + teardown sequence directly (no live
server needed — precedent t_9f0184cf), monkeypatching start_session_turn the
same way tests/test_session_channel_option_x.py does.
"""
from __future__ import annotations
import queue
import threading
import types
# --------------------------------------------------------------------------
# Fakes / fixtures (mirrors test_process_complete_ab_coexistence +
# test_session_channel_option_x patterns)
# --------------------------------------------------------------------------
class _FakeProcessRegistry:
"""Minimal stand-in for tools.process_registry.process_registry."""
def __init__(self):
self._lock = threading.Lock()
self._completion_consumed: set[str] = set()
self.completion_queue: queue.Queue = queue.Queue()
self._procs: dict[str, types.SimpleNamespace] = {}
def register(self, process_id: str, session_key: str) -> None:
self._procs[process_id] = types.SimpleNamespace(session_key=session_key)
def get(self, process_id: str):
return self._procs.get(process_id)
def is_completion_consumed(self, process_id: str) -> bool:
with self._lock:
return process_id in self._completion_consumed
def _install_fake_registry(monkeypatch, fake):
# Rebase isolation fix: ONLY monkeypatch.setitem (tracked, restored on
# teardown). The prior sys.modules.setdefault("tools", ...) was an
# UNTRACKED mutation that permanently leaked a non-package fake `tools`
# into sys.modules when real `tools` wasn't imported yet, breaking later
# tests that do `from tools.process_registry import ...` (now in-session
# alongside the merged upstream #2279).
import sys
mod = types.ModuleType("tools.process_registry")
mod.process_registry = fake
tools_mod = types.ModuleType("tools")
tools_mod.process_registry = mod # type: ignore[attr-defined]
monkeypatch.setitem(sys.modules, "tools", tools_mod)
monkeypatch.setitem(sys.modules, "tools.process_registry", mod)
def _install_fake_start_session_turn(monkeypatch, *, status=200):
"""Thin wrapper preserving the legacy local name; the body lives in
``tests/_wakeup_helpers.py`` and is shared with
``test_session_channel_option_x.py`` (Copilot PR #2971 r3305700944).
"""
from tests._wakeup_helpers import install_fake_start_session_turn as _impl
return _impl(monkeypatch, status=status)
def _wait_for_wakeup(holder, timeout=3.0):
"""Thin wrapper preserving the legacy local name; the body lives in
``tests/_wakeup_helpers.py`` and is shared with
``test_session_channel_option_x.py`` (Copilot PR #2971 r3305700944).
"""
from tests._wakeup_helpers import wait_for_wakeup as _impl
return _impl(holder, timeout=timeout)
def _wait_for(predicate, timeout=3.0, interval=0.02):
"""Poll *predicate* until it returns truthy or *timeout* elapses.
Used when the assertion targets state mutated by the wakeup daemon thread
AFTER it calls start_session_turn (e.g. the 409 re-defer), which races the
``holder['event']`` set inside the fake start_session_turn.
"""
import time
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if predicate():
return True
time.sleep(interval)
return predicate()
def _reset_cfg_state():
from api import config as _cfg
with _cfg.PROCESS_SESSION_INDEX_LOCK:
_cfg.PROCESS_SESSION_INDEX.clear()
_cfg.PENDING_BG_TASK_COMPLETIONS.clear()
_cfg.BG_TASK_COMPLETE_EVENTS_SEEN.clear()
with _cfg.DEFERRED_PROCESS_WAKEUPS_LOCK:
_cfg.DEFERRED_PROCESS_WAKEUPS.clear()
with _cfg.STREAMS_LOCK:
_cfg.STREAMS.clear()
if hasattr(_cfg, "ACTIVE_RUNS"):
with _cfg.ACTIVE_RUNS_LOCK:
_cfg.ACTIVE_RUNS.clear()
def _completion_evt(process_id: str, session_key: str) -> dict:
return {
"type": "completion",
"session_id": process_id,
"session_key": session_key,
"command": "sleep 2",
"exit_code": 0,
"output": "done",
}
# --------------------------------------------------------------------------
# Test 1 — THE headline acceptance: completion during teardown still wakes
# (the autonomous-agent, no-next-user-turn case == the Test B scenario)
# --------------------------------------------------------------------------
def test_completion_during_turn_teardown_still_wakes(monkeypatch):
"""Session has an ACTIVE_RUN → a process completes → _process_one defers
(marker persisted, NO immediate turn). Then the turn tears down
(unregister_active_run) and the teardown idle-hook fires the deferred
wakeup exactly once. This is the Test B (sleep 2) scenario.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
fake.register("proc-fast-1", "sess-teardown")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
holder = _install_fake_start_session_turn(monkeypatch)
sid = "sess-teardown"
stream_id = "stream-teardown-1"
bp.register_process_session(sid, sid)
try:
# A turn is active (mid-teardown window: agent finished output but
# ACTIVE_RUNS not yet cleared).
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
assert bp._session_has_active_turn(sid) is True
# Fast bg task completes INSIDE the teardown window → defer.
bp._process_one(_completion_evt("proc-fast-1", sid))
# Deferred, NOT fired: no turn started, prompt persisted.
assert holder["event"].wait(timeout=0.8) is False
assert holder["calls"] == []
assert sid in cfg.DEFERRED_PROCESS_WAKEUPS
assert cfg.DEFERRED_PROCESS_WAKEUPS[sid][0]["process_id"] == "proc-fast-1"
# Turn teardown: unregister_active_run clears the ACTIVE_RUNS row
# (this is exactly what streaming.py does under ACTIVE_RUNS_LOCK),
# then the teardown idle-hook runs.
cfg.unregister_active_run(stream_id)
assert bp._session_has_active_turn(sid) is False
started = bp.drain_deferred_wakeups_for_session(sid)
assert started == 1
assert _wait_for_wakeup(holder), (
"deferred wakeup was NOT redelivered at turn teardown — the "
"autonomous-agent fast-bg-task case is still broken"
)
assert len(holder["calls"]) == 1
call = holder["calls"][0]
assert call["session_id"] == sid
assert call["source"] == "process_wakeup"
assert call["message"].startswith("[IMPORTANT: Background process")
# Claimed → nothing left to re-deliver.
assert sid not in cfg.DEFERRED_PROCESS_WAKEUPS
finally:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 2 — idle path unchanged: fires once, no regression, the new teardown
# hook does not double-fire it (the Test A / sleep 5 path)
# --------------------------------------------------------------------------
def test_idle_completion_still_fires_once(monkeypatch):
"""No ACTIVE_RUN → _process_one fires the server-side wakeup immediately
(Option Z idle branch, the Test A path). Nothing is deferred, so the new
turn-teardown hook is a pure no-op — total deliveries stays exactly 1.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
fake.register("proc-idle-1", "sess-idle")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
holder = _install_fake_start_session_turn(monkeypatch)
sid = "sess-idle"
bp.register_process_session(sid, sid)
try:
assert bp._session_has_active_turn(sid) is False
bp._process_one(_completion_evt("proc-idle-1", sid))
assert _wait_for_wakeup(holder), "idle path regressed — wakeup not fired"
assert len(holder["calls"]) == 1
# Idle branch did NOT persist anything.
assert sid not in cfg.DEFERRED_PROCESS_WAKEUPS
# The wakeup turn itself ends and tears down → its teardown re-runs
# the idle-hook. It must find nothing and NOT double-fire.
started = bp.drain_deferred_wakeups_for_session(sid)
assert started == 0
assert len(holder["calls"]) == 1, (
"the teardown hook double-fired an idle-path wakeup"
)
finally:
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 3 — idempotent with the PR #2279 next-turn drain: a user turn that
# DOES come must not also deliver (shared SEEN / _completion_consumed gate)
# --------------------------------------------------------------------------
def test_next_user_turn_drain_and_teardown_hook_dont_double_fire(monkeypatch):
"""If a user turn DOES come, the next-turn drain
(_drain_webui_process_notifications) must NOT also deliver the deferred
completion: _process_one set BG_TASK_COMPLETE_EVENTS_SEEN AND the registry
_completion_consumed marker BEFORE the defer, and it consumed the
completion_queue event, so the next-turn drain has nothing to fire. The
teardown idle-hook then delivers it exactly once. Total deliveries == 1.
"""
from api import background_process as bp, config as cfg
from api import streaming as st
fake = _FakeProcessRegistry()
fake.register("proc-shared-1", "sess-shared")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
holder = _install_fake_start_session_turn(monkeypatch)
sid = "sess-shared"
stream_id = "stream-shared-1"
bp.register_process_session(sid, sid)
try:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
bp._process_one(_completion_evt("proc-shared-1", sid))
# Shared dedupe contract: _process_one marked it seen + registry-
# consumed before deferring.
assert "proc-shared-1" in cfg.BG_TASK_COMPLETE_EVENTS_SEEN[sid]
assert fake.is_completion_consumed("proc-shared-1")
assert sid in cfg.DEFERRED_PROCESS_WAKEUPS
# A user turn comes: the next-turn drain runs. Even if a duplicate
# event were re-queued (kill_process race), the SEEN + consumed gate
# makes it a no-op — it must NOT deliver the deferred wakeup.
fake.completion_queue.put(_completion_evt("proc-shared-1", sid))
notifications = st._drain_webui_process_notifications(sid)
assert notifications == [], (
"next-turn drain double-delivered a completion the defer path owns"
)
# That user turn ends → its teardown fires the deferred wakeup ONCE.
cfg.unregister_active_run(stream_id)
started = bp.drain_deferred_wakeups_for_session(sid)
assert started == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 1, (
"deferred wakeup delivered more than once across next-turn drain "
"+ teardown hook"
)
finally:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 4 — no wakeup loop: the wakeup turn's own teardown does not re-trigger
# a wakeup for the same process_id
# --------------------------------------------------------------------------
def test_no_wakeup_loop(monkeypatch):
"""The wakeup turn started by the teardown hook itself ends and tears
down → its teardown re-runs drain_deferred_wakeups_for_session. The atomic
claim (DEFERRED_PROCESS_WAKEUPS.pop) already removed the entry, so the
second drain finds nothing → no infinite wakeup loop.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
fake.register("proc-loop-1", "sess-loop")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
holder = _install_fake_start_session_turn(monkeypatch)
sid = "sess-loop"
stream_id = "stream-loop-1"
bp.register_process_session(sid, sid)
try:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
bp._process_one(_completion_evt("proc-loop-1", sid))
assert sid in cfg.DEFERRED_PROCESS_WAKEUPS
# First teardown: claims + fires once.
cfg.unregister_active_run(stream_id)
assert bp.drain_deferred_wakeups_for_session(sid) == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 1
assert sid not in cfg.DEFERRED_PROCESS_WAKEUPS
# The wakeup turn itself runs and tears down → second drain. It must
# find NOTHING (already claimed) and start NO further turn.
for _ in range(3):
assert bp.drain_deferred_wakeups_for_session(sid) == 0
assert len(holder["calls"]) == 1, (
"wakeup loop: the wakeup turn's own teardown re-fired the same "
"process_id"
)
finally:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 5 — multi-stream / cancel-reconnect guard: only fire when the session
# is TRULY idle (the just-ended stream was the last ACTIVE_RUN for the sid)
# --------------------------------------------------------------------------
def test_multistream_guard_only_fires_when_truly_idle(monkeypatch):
"""A cancel/reconnect leaves a SECOND active stream for the same session.
When the first stream tears down the session is NOT idle → the deferred
marker must be left intact (no fire). Only when the last active stream
tears down does the hook claim + fire, exactly once.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
fake.register("proc-multi-1", "sess-multi")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
holder = _install_fake_start_session_turn(monkeypatch)
sid = "sess-multi"
stream_a = "stream-multi-a"
stream_b = "stream-multi-b"
bp.register_process_session(sid, sid)
try:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS[stream_a] = {"session_id": sid}
cfg.ACTIVE_RUNS[stream_b] = {"session_id": sid}
bp._process_one(_completion_evt("proc-multi-1", sid))
assert sid in cfg.DEFERRED_PROCESS_WAKEUPS
# First stream tears down — second is still active → NOT idle.
cfg.unregister_active_run(stream_a)
assert bp._session_has_active_turn(sid) is True
assert bp.drain_deferred_wakeups_for_session(sid) == 0
assert holder["calls"] == []
# Marker retained for the later teardown.
assert sid in cfg.DEFERRED_PROCESS_WAKEUPS
# Last stream tears down — now truly idle → fire exactly once.
cfg.unregister_active_run(stream_b)
assert bp._session_has_active_turn(sid) is False
assert bp.drain_deferred_wakeups_for_session(sid) == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 1
assert sid not in cfg.DEFERRED_PROCESS_WAKEUPS
finally:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS.pop(stream_a, None)
cfg.ACTIVE_RUNS.pop(stream_b, None)
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 6 — 409 during the teardown-hook path re-queues the wakeup (Greptile
# PR #2971 r3371737184). The teardown hook ATOMICALLY CLAIMS the deferred
# entry and DISCARDS the PENDING marker BEFORE spawning the wakeup thread. So
# if start_session_turn then 409s (a human /api/chat/start raced in between),
# both the marker and the claimed prompt are gone — the wakeup would be lost
# forever unless the 409 path re-queues it. This test pins that re-queue.
# --------------------------------------------------------------------------
def test_teardown_409_requeues_wakeup_so_it_is_not_lost(monkeypatch):
"""drain_deferred_wakeups_for_session claims the entry + discards the
PENDING marker, then the spawned wakeup turn 409s on a racing human turn.
The 409 branch must re-queue via record_deferred_wakeup so a later teardown
(or next-turn drain) redelivers it. Without the fix the wakeup is dropped.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
fake.register("proc-409-1", "sess-409")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
# start_session_turn 409s — a human /api/chat/start won the per-session lock.
holder = _install_fake_start_session_turn(monkeypatch, status=409)
sid = "sess-409"
stream_id = "stream-409-1"
bp.register_process_session(sid, sid)
try:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
bp._process_one(_completion_evt("proc-409-1", sid))
assert sid in cfg.DEFERRED_PROCESS_WAKEUPS
# Turn tears down → teardown hook claims the entry, discards the marker,
# and spawns the wakeup turn (which 409s).
cfg.unregister_active_run(stream_id)
assert bp.drain_deferred_wakeups_for_session(sid) == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 1
# The 409 must have RE-QUEUED the entry rather than dropping it. Poll
# briefly: the re-queue runs on the same daemon thread right after the
# recorded call, so it may land a hair after the event fires.
import time as _t
for _ in range(50):
with cfg.DEFERRED_PROCESS_WAKEUPS_LOCK:
if cfg.DEFERRED_PROCESS_WAKEUPS.get(sid):
break
_t.sleep(0.02)
with cfg.DEFERRED_PROCESS_WAKEUPS_LOCK:
requeued = cfg.DEFERRED_PROCESS_WAKEUPS.get(sid) or []
assert requeued, (
"409 in the teardown-hook path DROPPED the wakeup — it must "
"re-queue via record_deferred_wakeup so it is not lost"
)
# Re-queue is idempotent per process_id: exactly one entry, same id.
assert len(requeued) == 1
assert requeued[0].get("process_id") == "proc-409-1"
finally:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 7 — Greptile P1: multiple deferred wakeups in one teardown. Only the
# FIRST starts a turn (the rest would 409 racing the per-session agent lock);
# entries 2..N must be re-deferred, not lost, and drain on later teardowns.
# --------------------------------------------------------------------------
def test_multiple_deferred_wakeups_each_survive_across_teardowns(monkeypatch):
"""N>=2 bg tasks complete during one active turn → all deferred. A single
teardown drain must NOT fire N racing daemon threads (only one wins the
agent lock; the losers 409 and, since the entries were already popped,
their prompts would be lost forever). The fix starts exactly one wakeup
and re-defers the remainder, so each subsequent teardown delivers the
next — every prompt is eventually delivered exactly once.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
for pid in ("proc-A", "proc-B", "proc-C"):
fake.register(pid, "sess-multi-defer")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
holder = _install_fake_start_session_turn(monkeypatch)
sid = "sess-multi-defer"
stream_id = "stream-multi-defer"
bp.register_process_session(sid, sid)
try:
# A turn is active; three fast bg tasks all complete inside it → defer.
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS[stream_id] = {"session_id": sid}
for pid in ("proc-A", "proc-B", "proc-C"):
bp._process_one(_completion_evt(pid, sid))
assert len(cfg.DEFERRED_PROCESS_WAKEUPS.get(sid, [])) == 3
# Turn teardown → idle. Drain reports exactly ONE wakeup started, and
# the other two are re-deferred (not popped-and-lost).
cfg.unregister_active_run(stream_id)
assert bp._session_has_active_turn(sid) is False
assert bp.drain_deferred_wakeups_for_session(sid) == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 1
assert len(cfg.DEFERRED_PROCESS_WAKEUPS.get(sid, [])) == 2
# Second teardown delivers the next, third delivers the last.
holder["event"].clear()
assert bp.drain_deferred_wakeups_for_session(sid) == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 2
assert len(cfg.DEFERRED_PROCESS_WAKEUPS.get(sid, [])) == 1
holder["event"].clear()
assert bp.drain_deferred_wakeups_for_session(sid) == 1
assert _wait_for_wakeup(holder)
assert len(holder["calls"]) == 3
assert sid not in cfg.DEFERRED_PROCESS_WAKEUPS
# All three distinct prompts delivered exactly once, no duplicates.
delivered = {c["message"] for c in holder["calls"]}
assert len(delivered) == 3
# Nothing left → a final drain is a no-op (no wakeup loop).
assert bp.drain_deferred_wakeups_for_session(sid) == 0
finally:
with cfg.ACTIVE_RUNS_LOCK:
cfg.ACTIVE_RUNS.pop(stream_id, None)
bp.unregister_process_session(sid)
_reset_cfg_state()
# --------------------------------------------------------------------------
# Test 7 — Greptile P1 (idle-branch sibling of the 409/teardown F1 fix):
# the IDLE-path wakeup must pass process_id, so when its daemon thread loses
# the per-session lock race (409) the re-defer carries the real process_id and
# the record_deferred_wakeup dedup guard stays live — a second 409 race cannot
# accumulate a duplicate deferred entry that would deliver the wakeup twice.
# --------------------------------------------------------------------------
def test_idle_path_409_redefer_carries_process_id_and_dedups(monkeypatch):
"""Idle completion → _process_one idle branch starts the server-side
wakeup. The fake start_session_turn returns 409 (raced an active turn /
sibling wakeup), so the daemon re-defers via record_deferred_wakeup.
BEFORE the fix the idle branch called _start_server_side_wakeup_turn
WITHOUT process_id, so the re-defer recorded process_id="" — falsy, so the
`if process_id and any(...)` dedup guard was skipped and a second 409 race
appended a duplicate identical entry, ultimately double-delivering the
wakeup. AFTER the fix the re-defer carries the real process_id and the
guard dedups the second race to a single entry.
"""
from api import background_process as bp, config as cfg
fake = _FakeProcessRegistry()
fake.register("proc-idle-409", "sess-idle-409")
_install_fake_registry(monkeypatch, fake)
_reset_cfg_state()
# 409 == lost the per-session agent lock race; triggers the re-defer path.
holder = _install_fake_start_session_turn(monkeypatch, status=409)
sid = "sess-idle-409"
bp.register_process_session(sid, sid)
try:
# Session is idle → _process_one takes the idle branch (the line-838
# call site this card fixes), which now plumbs process_id through.
assert bp._session_has_active_turn(sid) is False
bp._process_one(_completion_evt("proc-idle-409", sid))
# The wakeup daemon ran, hit 409, and re-deferred. Poll for the
# re-defer (it races the holder event set inside the fake).
assert _wait_for_wakeup(holder), "idle-branch wakeup never attempted"
assert _wait_for(
lambda: bool(cfg.DEFERRED_PROCESS_WAKEUPS.get(sid))
), "409 on the idle path did not re-defer the wakeup — it was lost"
entries = cfg.DEFERRED_PROCESS_WAKEUPS[sid]
assert len(entries) == 1
# THE regression assertion: the re-defer carries the real process_id,
# not "" (which is what the missing-process_id idle call produced).
assert entries[0]["process_id"] == "proc-idle-409", (
"idle-path re-defer dropped process_id — the dedup guard in "
"record_deferred_wakeup is now bypassed and duplicates can "
"accumulate (the exact Greptile P1)"
)
# Now prove the dedup guard is actually live on this path: a second
# 409 race for the SAME process_id (e.g. the next teardown drain) must
# NOT append a duplicate. With process_id="" (pre-fix) this would grow
# to 2 entries and double-deliver.
bp.record_deferred_wakeup(
sid, "proc-idle-409", entries[0]["wakeup_prompt"]
)
assert len(cfg.DEFERRED_PROCESS_WAKEUPS[sid]) == 1, (
"duplicate deferred entry accumulated — dedup guard did not fire "
"because the idle-path re-defer lost its process_id"
)
finally:
bp.unregister_process_session(sid)
_reset_cfg_state()

View File

@@ -0,0 +1,283 @@
"""Regression tests — wakeup-turn model-resolve hang (t_46fadfbc).
Proven root cause (from a live thread-stack capture of a hung wakeup
turn, "Slow WebUI request still running"):
_handle_chat_start → _resolve_compatible_session_model_state
→ get_available_models → _build_available_models_uncached
→ _read_live_provider_model_ids → get_copilot_api_token
→ exchange_copilot_token → urllib HTTPS → BLOCKED
A server-initiated Option-Z wakeup turn (drain thread, idle session, no
browser) reached chat/start with a COLD provider catalog and triggered a
LIVE per-provider rebuild whose Copilot token-exchange HTTPS call hung the
wakeup turn forever on this WSL/corp network — NOT a race.
Two fixes, two tests:
1. test_wakeup_turn_uses_persisted_model_no_live_probe
start_session_turn(source="process_wakeup") resolves the model from the
persisted session record via the cache-only path; the live provider
rebuild (and the Copilot exchange) is never invoked even when the catalog
is cold. The turn still starts with the persisted model.
2. test_chat_start_survives_slow_provider_probe
Defense-in-depth: an unbounded/hanging provider probe cannot stall a
foreground get_available_models() past the wall-clock budget — it falls
back to a usable model list and lets the rebuild finish out-of-band.
"""
from __future__ import annotations
import time
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
# ---------------------------------------------------------------------------
# Fix 1 — wakeup resolves persisted model with NO live provider probe
# ---------------------------------------------------------------------------
def test_wakeup_turn_uses_persisted_model_no_live_probe(monkeypatch):
"""Cold catalog + a session with a persisted model/provider: a
server-side wakeup must start the turn using the PERSISTED model and must
NOT call the live per-provider rebuild (which is what does the blocking
Copilot token-exchange HTTPS call in the proven thread-stack).
"""
from api import config as cfg
import api.routes as routes
sid = "sess-wakeup-persisted"
persisted_model = "anthropic/claude-sonnet-4"
fake_stream_id = "stream-wakeup-persisted-1"
# HARD ASSERT: the live per-provider probe must never run on the wakeup
# path. _read_live_provider_model_ids is the function the thread-stack
# shows calling get_copilot_api_token → exchange_copilot_token.
def _boom(*_a, **_k):
raise AssertionError(
"live provider rebuild ran on a wakeup turn — "
"_read_live_provider_model_ids must NOT be called (it does the "
"blocking Copilot token-exchange HTTPS call)"
)
monkeypatch.setattr(cfg, "_read_live_provider_model_ids", _boom, raising=True)
# Cold cache: force the cache-miss branch that used to trigger the live
# rebuild. The autouse conftest fixture already invalidates, but be
# explicit so this test documents the precondition.
cfg.invalidate_models_cache()
captured: dict = {}
def _fake_start_chat_stream_for_session(s, **kwargs):
captured["model"] = kwargs.get("model")
captured["model_provider"] = kwargs.get("model_provider")
return {"stream_id": fake_stream_id, "session_id": s.session_id, "_status": 200}
class _FakeSession:
session_id = sid
model = persisted_model
model_provider = "anthropic"
monkeypatch.setattr(
routes, "_start_chat_stream_for_session",
_fake_start_chat_stream_for_session, raising=True,
)
monkeypatch.setattr(routes, "get_session", lambda _sid: _FakeSession(), raising=True)
monkeypatch.setattr(
routes, "_resolve_chat_workspace_with_recovery",
lambda s, w: "/tmp/ws", raising=True,
)
t0 = time.monotonic()
resp = routes.start_session_turn(
sid, "[IMPORTANT: Background process done]", source="process_wakeup"
)
elapsed = time.monotonic() - t0
assert resp.get("stream_id") == fake_stream_id, "wakeup turn did not start"
# The persisted model survives to the turn — never dropped, never blocked.
assert captured["model"] == persisted_model, (
f"wakeup turn used {captured.get('model')!r}, expected the persisted "
f"session model {persisted_model!r}"
)
# Fast: no network. Cache-only resolution is sub-second; allow generous
# slack for slow CI but well under the 10s Copilot timeout that was the
# original symptom.
assert elapsed < 5.0, (
f"wakeup turn took {elapsed:.2f}s — a live provider probe likely ran"
)
def test_wakeup_resolve_passes_prefer_cached_catalog(monkeypatch):
"""White-box: start_session_turn must route model resolution through the
cache-only path (prefer_cached_catalog=True). Pins the wiring so a future
refactor can't silently reintroduce the live-probe hang.
"""
import api.routes as routes
sid = "sess-wakeup-wiring"
seen: dict = {}
real = routes._resolve_compatible_session_model_state
def _spy(model_id, model_provider=None, *, profile_provider=None,
profile_default_model=None, prefer_cached_catalog=False):
seen["prefer_cached_catalog"] = prefer_cached_catalog
# The wakeup path now also threads the session's profile model defaults
# through (greptile fix) so a brand-new session with an empty model
# falls back to the profile default, not the global DEFAULT_MODEL.
seen["profile_provider"] = profile_provider
seen["profile_default_model"] = profile_default_model
return ("m", None, False)
monkeypatch.setattr(
routes, "_resolve_compatible_session_model_state", _spy, raising=True
)
monkeypatch.setattr(
routes, "_start_chat_stream_for_session",
lambda s, **k: {"stream_id": "x", "session_id": s.session_id, "_status": 200},
raising=True,
)
class _FakeSession:
session_id = sid
model = "m"
model_provider = None
monkeypatch.setattr(routes, "get_session", lambda _s: _FakeSession(), raising=True)
monkeypatch.setattr(
routes, "_resolve_chat_workspace_with_recovery",
lambda s, w: "/tmp/ws", raising=True,
)
routes.start_session_turn(sid, "[IMPORTANT: x]", source="process_wakeup")
assert seen.get("prefer_cached_catalog") is True, (
"start_session_turn must resolve the model with "
"prefer_cached_catalog=True so a wakeup never triggers a live probe"
)
assert real is not None # the real function still exists (not deleted)
# ---------------------------------------------------------------------------
# Fix 2 — bounded rebuild: a slow probe cannot stall chat/start forever
# ---------------------------------------------------------------------------
def test_chat_start_survives_slow_provider_probe(monkeypatch):
"""Simulate a hanging provider probe on the NORMAL (prefer_cache=False)
cold path: get_available_models() must return within the wall-clock
budget with a usable fallback instead of blocking on the network.
"""
from api import config as cfg
cfg.invalidate_models_cache()
monkeypatch.setattr(cfg, "_LIVE_REBUILD_BUDGET_SECONDS", 0.4, raising=True)
started = {"n": 0}
def _slow_rebuild(_builder):
started["n"] += 1
# >> budget — models the hung Copilot HTTPS call. 0.8s is 2× the
# monkeypatched 0.4s budget, which is the smallest gap that still
# robustly proves the contract while keeping suite wall-clock low
# (was 3.0s; suite-latency cleanup per Copilot review).
time.sleep(0.8)
return {
"active_provider": "anthropic",
"default_model": "anthropic/claude-sonnet-4",
"configured_model_badges": {},
"groups": [],
}
# Replace the rebuild seam so no real per-provider network call happens
# but the foreground still has to wait for (a stand-in for) it.
monkeypatch.setattr(cfg, "_invoke_models_rebuild", _slow_rebuild, raising=True)
# Ensure no disk cache short-circuits the cold path.
monkeypatch.setattr(cfg, "_load_models_cache_from_disk", lambda: None, raising=True)
t0 = time.monotonic()
result = cfg.get_available_models()
elapsed = time.monotonic() - t0
assert started["n"] == 1, "the rebuild worker should have been started"
assert elapsed < 2.0, (
f"get_available_models() blocked {elapsed:.2f}s on a hung probe — "
f"the {cfg._LIVE_REBUILD_BUDGET_SECONDS}s budget did not bound it"
)
# The fallback must be a structurally valid, usable catalog.
assert isinstance(result, dict)
for k in ("active_provider", "default_model", "configured_model_badges", "groups"):
assert k in result, f"fallback catalog missing {k!r}"
assert isinstance(result["groups"], list)
def test_minimal_static_catalog_is_network_free(monkeypatch):
"""The fallback catalog builder must never reach the live provider probe.
"""
from api import config as cfg
monkeypatch.setattr(
cfg, "_read_live_provider_model_ids",
lambda *_a, **_k: (_ for _ in ()).throw(
AssertionError("minimal catalog must not probe providers")
),
raising=True,
)
out = cfg._minimal_static_models_catalog()
assert set(out) >= {
"active_provider", "default_model", "configured_model_badges", "groups"
}
assert isinstance(out["groups"], list)
def test_prefer_cache_kw_exists_and_skips_live_rebuild(monkeypatch):
"""get_available_models(prefer_cache=True) must resolve without invoking
the rebuild seam at all when the cache is cold.
"""
from api import config as cfg
cfg.invalidate_models_cache()
monkeypatch.setattr(cfg, "_load_models_cache_from_disk", lambda: None, raising=True)
def _must_not_run(_b):
raise AssertionError(
"prefer_cache=True triggered the live rebuild seam — it must "
"serve cache/minimal-static only"
)
monkeypatch.setattr(cfg, "_invoke_models_rebuild", _must_not_run, raising=True)
result = cfg.get_available_models(prefer_cache=True)
assert isinstance(result, dict)
assert "default_model" in result and "groups" in result
# ---------------------------------------------------------------------------
# Source-grep wiring guards
# ---------------------------------------------------------------------------
def test_get_available_models_has_prefer_cache_param():
import inspect
from api import config as cfg
sig = inspect.signature(cfg.get_available_models)
assert "prefer_cache" in sig.parameters
param = sig.parameters["prefer_cache"]
assert param.kind is inspect.Parameter.KEYWORD_ONLY
assert param.default is False
assert "_LIVE_REBUILD_BUDGET_SECONDS" in cfg.__dict__
assert "_minimal_static_models_catalog" in cfg.__dict__
def test_start_session_turn_uses_cached_catalog():
src = (REPO_ROOT / "api" / "routes.py").read_text(encoding="utf-8")
# The wakeup entrypoint must resolve with the cache-only flag.
i = src.find("def start_session_turn(")
assert i != -1
j = src.find("def _handle_process_complete_ack", i)
body = src[i:j]
assert "prefer_cached_catalog=True" in body, (
"start_session_turn must pass prefer_cached_catalog=True"
)

View File

@@ -0,0 +1,259 @@
"""Regression: cross-session notify_on_complete wakeup misroute (Option 1 + Option 3).
ROOT CAUSE (RCA t_f62ff1e8, verified line-by-line):
WebUI's per-turn session identity was bound ONLY to the process-global
``os.environ['HERMES_SESSION_KEY']`` (streaming.py turn-start), and the env
lock was released BEFORE the agent ran. WebUI NEVER called
``gateway.session_context.set_session_vars`` so the ``_SESSION_KEY``
contextvar stayed ``_UNSET`` and ``tools.approval.get_current_session_key``
fell back to the racy process-global env. Two concurrent WebUI turns
therefore raced on one slot: session A's ``terminal(notify_on_complete=True)``
spawn could capture session B's id, and at completion the server-side wakeup
turn started for the WRONG session.
This module proves BOTH fix layers with REAL modules (no mocks of the code
under test), so each test is RED before the fix and GREEN after — never a
tautology:
Option 1 (root fix, streaming.py) — ``_bind_turn_session_identity`` binds the
REAL ``_SESSION_KEY`` contextvar for the turn's worker thread/context and
clears it on exit. Under a simulated env race across two concurrent turns,
the REAL ``get_current_session_key`` must return each turn's OWN id.
Option 3 (defense-in-depth, background_process.py) —
``_resolve_wakeup_target`` cross-checks the (possibly env-contaminated)
session_key-resolved session against the spawn-time owner persisted in the
process registry's ``ProcessSession.spawn_session_id`` (an env-immune
field). On a positively-detected mismatch it re-routes to the true owner
instead of waking the wrong session.
Precedent for this no-live-server style: tests/test_wakeup_defer_race.py,
tests/test_session_channel_option_x.py.
"""
from __future__ import annotations
import importlib
import threading
import pytest
# ---------------------------------------------------------------------------
# Option 1 — contextvar binding makes per-turn session identity race-immune
# ---------------------------------------------------------------------------
def test_streaming_exposes_turn_session_identity_binder():
"""streaming.py must expose the helper that binds the REAL _SESSION_KEY
contextvar for the agent worker thread (root fix, not env-only)."""
streaming = importlib.import_module("api.streaming")
assert hasattr(streaming, "_bind_turn_session_identity"), (
"Option 1 missing: streaming.py must expose _bind_turn_session_identity "
"to bind gateway.session_context._SESSION_KEY for the turn"
)
def test_concurrent_turns_capture_their_own_session_under_env_race():
"""THE invariant. Two concurrent WebUI turns, A and B. Each binds its own
session identity via the REAL streaming helper, then — while the OTHER
turn has just stamped the shared process-global env (the documented race:
lock released, agent still running) — performs the EXACT capture call a
notify_on_complete spawn makes: ``tools.approval.get_current_session_key``.
Pre-fix: WebUI never set the contextvar, so the helper does not exist
(ImportError above) / the contextvar stays _UNSET and the capture falls
back to the env → A captures B's id → MISROUTE (RED).
Post-fix: the helper binds _SESSION_KEY in each worker context, so the
contextvar (task-local) wins over the racy env → each turn captures its
OWN id (GREEN).
"""
import os
streaming = importlib.import_module("api.streaming")
pytest.importorskip("tools.approval", reason="hermes-agent not installed")
pytest.importorskip("gateway.session_context")
from tools.approval import get_current_session_key
from gateway import session_context as sc
bind = getattr(streaming, "_bind_turn_session_identity", None)
if bind is None:
pytest.fail("Option 1 not implemented: _bind_turn_session_identity missing")
# The two turn() threads below stamp os.environ["HERMES_SESSION_KEY"]
# without owning it. Save/restore the prior value (sentinel for "was
# unset") so this test does not leak state into sibling tests when
# collection order interleaves it with another consumer.
_prev_env_sentinel = object()
_prev_env = os.environ.get("HERMES_SESSION_KEY", _prev_env_sentinel)
SESS_A = "20260518_161627_60b5f4" # board claude-code-import
SESS_B = "47ec28f66dff" # board mcp-optimize
captured: dict[str, str] = {}
barrier = threading.Barrier(2)
# Force a deterministic interleave: A captures only AFTER B has stamped
# the process-global env (reproduces "lock released, agent still running,
# other turn stamps the global slot").
b_stamped_env = threading.Event()
def turn(my_sid: str, label: str) -> None:
# streaming.py turn-start still writes the process-global env as a
# fallback for non-contextvar consumers; the fix is that session-key
# ROUTING now binds the contextvar so it no longer races.
with bind(my_sid):
os.environ["HERMES_SESSION_KEY"] = my_sid
barrier.wait()
if label == "B":
# B stamps env last while A's "agent" is still mid-turn.
os.environ["HERMES_SESSION_KEY"] = my_sid
b_stamped_env.set()
else:
assert b_stamped_env.wait(timeout=5), "B never stamped env"
# The EXACT call terminal_tool.py makes for a bg spawn:
captured[label] = get_current_session_key(default="")
try:
ta = threading.Thread(target=turn, args=(SESS_A, "A"))
tb = threading.Thread(target=turn, args=(SESS_B, "B"))
ta.start()
tb.start()
ta.join(timeout=10)
tb.join(timeout=10)
# Assert the worker threads actually terminated. If a thread deadlocks the
# join() above returns silently — surface that as a clear test failure
# instead of letting downstream asserts mask the hang or leak threads into
# the rest of the run.
assert not ta.is_alive(), "worker thread A did not terminate within join timeout"
assert not tb.is_alive(), "worker thread B did not terminate within join timeout"
# Contextvar must be restored after the turn context exits (no thread-pool
# residue → no new race for a reused worker).
assert sc._SESSION_KEY.get() is sc._UNSET
assert captured.get("A") == SESS_A, (
f"MISROUTE: session A captured {captured.get('A')!r}, expected "
f"{SESS_A!r} — per-turn identity still races on process-global env"
)
assert captured.get("B") == SESS_B, (
f"MISROUTE: session B captured {captured.get('B')!r}, expected {SESS_B!r}"
)
finally:
# Restore HERMES_SESSION_KEY to its pre-test value (or unset if it was
# never set), independent of which assertion above might have failed —
# see save/restore note at the top of the test.
if _prev_env is _prev_env_sentinel:
os.environ.pop("HERMES_SESSION_KEY", None)
else:
assert isinstance(_prev_env, str)
os.environ["HERMES_SESSION_KEY"] = _prev_env
def test_turn_identity_binder_restores_previous_value():
"""Restore uses contextvars reset-token semantics (the canonical idiom),
NOT a blanket clear_session_vars: it composes correctly under nesting and
restores _UNSET for the top-level turn so CLI/cron env-fallback compat is
preserved, and it must NOT touch the platform/chat_id/user session vars
(those keep their env fallback so the notify_on_complete watcher
registration that reads HERMES_SESSION_PLATFORM still works)."""
streaming = importlib.import_module("api.streaming")
pytest.importorskip("tools.approval", reason="hermes-agent not installed")
from tools.approval import get_current_session_key
from gateway import session_context as sc
bind = streaming._bind_turn_session_identity
assert sc._SESSION_KEY.get() is sc._UNSET
# Platform var starts unset → env fallback path intact.
assert sc._SESSION_PLATFORM.get() is sc._UNSET
with bind("sid-outer"):
assert get_current_session_key(default="") == "sid-outer"
with bind("sid-inner"):
assert get_current_session_key(default="") == "sid-inner"
# Reset-token restores the OUTER value (composes under nesting),
# it does NOT clear to "".
assert get_current_session_key(default="") == "sid-outer"
# The binder must never disturb the other session vars.
assert sc._SESSION_PLATFORM.get() is sc._UNSET
# Full exit restores _UNSET → env fallback resumes (CLI/cron compat).
assert sc._SESSION_KEY.get() is sc._UNSET
assert sc._SESSION_PLATFORM.get() is sc._UNSET
# ---------------------------------------------------------------------------
# Option 3 — completion-time second-check against the env-immune spawn owner
# ---------------------------------------------------------------------------
def test_background_process_exposes_wakeup_target_resolver():
bp = importlib.import_module("api.background_process")
assert hasattr(bp, "_resolve_wakeup_target"), (
"Option 3 missing: background_process.py must expose _resolve_wakeup_target "
"to cross-check the session_key-resolved target against the env-immune "
"spawn-time owner"
)
def _fake_ps(session_key: str, spawn_session_id: str):
import types
return types.SimpleNamespace(
session_key=session_key, spawn_session_id=spawn_session_id
)
def test_resolve_wakeup_target_reroutes_on_positive_mismatch(monkeypatch):
"""The RCA scenario: env race made the process's ``session_key`` resolve to
session B, but the env-immune ``spawn_session_id`` says session A truly
spawned it. ``_resolve_wakeup_target`` must detect the positive mismatch
and return A (the true owner), NOT B.
"""
bp = importlib.import_module("api.background_process")
SESS_A = "20260518_161627_60b5f4"
SESS_B = "47ec28f66dff"
# session_key (env-contaminated) → resolves to B; spawn owner is A.
ps = _fake_ps(session_key=SESS_B, spawn_session_id=SESS_A)
resolved = bp._resolve_wakeup_target(
process_id="proc_f377e2f552cd",
session_key_resolved_sid=SESS_B,
proc_session=ps,
)
assert resolved == SESS_A, (
f"Option 3 failed to re-route: woke {resolved!r}, the env-immune spawn "
f"owner is {SESS_A!r} — this is the exact agent.log:6632 misroute"
)
def test_resolve_wakeup_target_passthrough_when_consistent():
"""No mismatch (the normal case, and the post-Option 1 case): the resolver is
a pure pass-through and never suppresses a legitimate Option Z wakeup."""
bp = importlib.import_module("api.background_process")
sid = "session-normal"
ps = _fake_ps(session_key=sid, spawn_session_id=sid)
assert bp._resolve_wakeup_target(
process_id="proc_ok",
session_key_resolved_sid=sid,
proc_session=ps,
) == sid
def test_resolve_wakeup_target_passthrough_when_owner_unknown():
"""If the spawn owner is indeterminate (no env-immune field, e.g. a
cron/CLI process sharing the registry, or a pre-Option 1 spawn), the resolver
must fall through to the session_key-resolved sid UNCHANGED — never
suppress a wakeup on uncertainty (Option Z must keep working)."""
bp = importlib.import_module("api.background_process")
sid = "session-unknown-owner"
assert bp._resolve_wakeup_target(
process_id="proc_cron",
session_key_resolved_sid=sid,
proc_session=_fake_ps(session_key=sid, spawn_session_id=""),
) == sid
assert bp._resolve_wakeup_target(
process_id="proc_cron",
session_key_resolved_sid=sid,
proc_session=None,
) == sid