---
id: "bug-1515"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "scheduler"
severity: "unknown"
timestamp: "2026-08-10T20:20:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: [".kiro/specs/component-failover-adoption/requirements.md", "design.md", "tasks.md", "research.md"]
tags: ["design-review", "goal-loop", "requirements-math-error", "rejoin-retry", "shutdown-latency", "state-conflation", "scheduler", "apiserver", "worker"]
related: ["bug-1514"]
legacy: true
---

# bug-1515: /goal-driven deep re-review of component-failover-adoption requirements.md+design.md found 5…

## Description
/goal-driven deep re-review of component-failover-adoption requirements.md+design.md found 5 additional distinct defects after bug-1514's fix, several introduced by bug-1514's own Req 7 fix

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
(1) Req 3 AC2's '~6-job grace window' was stated as 'at least 6' (a floor) when 6 is actually the best-case ceiling; the real guaranteed floor is 1-2 depending on decisions-cbuf occupancy at the moment of stall -- direction of the bound was backwards. (2) Citation scheduler.c:1253 for sched_publish_pend_reason was off by 2 (actual line 1251). (3) Cross-cutting: the shared 'check and clear needs_rejoin, then call xcecli_join' pattern (worker/scheduler/apiserver) cleared the flag unconditionally rather than only on xcecli_join's own success -- since xcecli_join is itself XCECLI_FAILOVER_CALL-wrapped and can hit its own retry-budget exhaustion, this could leave a component transport-connected but permanently un-joined with nothing left to retry (no requirement/design text previously covered this). (4) The Req 7 scheduler decision-publish retry loop only checked core.stop in the while-condition (after each attempt), so multiple decisions queued behind a stuck one would each burn a full ~100s budget cycle discovering shutdown was requested, compounding shutdown latency to minutes instead of the claimed ~1-2 min bound. (5) While fixing (3) for apiserver's heartbeat-idle-path, the first draft gated the retry on ApiserverBackendHealth.IsDegraded() -- but that state has its own independent TTL-based auto-clear for circuit-breaker purposes; if the TTL expired before an actual rejoin succeeded, the retry would silently stop, reintroducing the exact starvation bug (3) was fixing. Two unrelated state machines (rejoin-tracking vs request-path circuit-breaker) had been conflated onto one flag.

## Fix
(1) Reworded Req 3 AC2 and all design.md/research.md/tasks.md echoes to state the guarantee qualitatively (bounded, occupancy-dependent, resumes with zero loss per Req 7) instead of a wrong fixed number. (2) Fixed the line citation. (3) Added Requirement 2 AC5 (rejoin retry-on-failure, applies to all three components) and updated the shared 'Rejoin callback constraint' section plus all three component sections in design.md and their tasks.md task bullets to clear the flag only on xcecli_join success. (4) Changed the Req 7 retry loop from do/while to while (check core.stop before each attempt, not only after) so only the actively in-flight decision can cost a full budget cycle; queued decisions behind it skip immediately once stop is set. (5) Introduced a dedicated needs_rejoin flag for apiserver's heartbeat path (set on transition, cleared only on join success) fully decoupled from ApiserverBackendHealth's TTL-governed degraded state; updated the sequence diagram, prose, and tasks.md task 4.4 accordingly, plus an explicit note in ApiserverBackendHealth's section warning against this conflation. Also added Req 7 AC6 documenting that the fix does not survive a scheduler process crash/restart (in-memory only, no persistence).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1514