Live-cluster gap found 2026-10-20 (t1/t2/t3): t3 went down, recovered,
went down again — and the second outage never reported. The dedup key
survived the healthy rounds, so the recurring situation looked
"unchanged" forever. Same hazard for pend_stall: a stall that empties
and later re-forms identically stays silent.
Fix: fire/clear pairing. When a detector's condition is fully absent
(all nodes healthy, no stalled jobs, no skew), it clears its scenario's
dedup key; the next occurrence is a fresh situation and fires. Unit
tests pin the outage→recovery→identical-outage cycle for worker_health
and the stall→resolve→re-form cycle for pend_stall.
exit_burst deliberately does NOT clear: RecentExits is a per-scan
window (consumed once), so an empty window is the steady state, not a
resolution — clearing there would re-fire bursts whose situation never
changed.
Cluster-verified cycle: t3 close → fired (glm, 11.1s); open → silent;
close again → fired again (22.1s, 2 findings). All detect tests green;
advisor restarted on the fix.
Co-Authored-By: Claude Code <noreply@anthropic.com>
snapshot: list jobs with AllUsers + submit_time:desc + limit 200 — the
advisor must see every user's PEND jobs newest-first, or pend_stall sees
nothing on a shared cluster (the default list is the caller's own jobs,
oldest first).
detect: drop the per-scenario cooldown from the fire path. The original
cooldown-first order suppressed a CHANGED situation for up to 10 minutes —
on the live cluster a new job joining the stalled set went unreported
until the cooldown expired. Semantics now: an unchanged situation (same
dedup key) never re-evaluates; any changed situation fires immediately.
A thrashing situation (A→B→A) firing per change is honest reporting — a
flapping node or oscillating stalled set is real signal.
Verified on the t1/t2/t3 cluster: two forced-PEND jobs (unsatisfiable
select; 64-slot limit rejection) both diagnosed in the first scan, a
kill changing the stalled set re-triggered within one 30s round, and
identical situations stayed deduped.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/snapshot: compact cluster state via schedulerclient (nodes,
PEND top-N most-starved with parsed pending_reason, cluster sums, queue
garnish, recent exits). Field whitelists + stall-threshold filter keep
the state a few KiB. Module-local PEND-reason table mirrors
mcp-server/internal/pendreason.go (lift into schedulerclient is a P1
cleanup).
- internal/detect: pend_stall / worker_health / util_skew / exit_burst
triggers — threshold, then per-scenario cooldown, then dedup on a hash
of the situation (job set + reasons / node set) so an unchanged cluster
is never re-evaluated.
Co-Authored-By: Claude Code <noreply@anthropic.com>
