merge: sync Rongsong's scheduler-framework (88 commits, f01ae353..c5b172c7)

Two-sided divergence from f01ae353 (2026-08-30): theirs +88 (worker
rejoin-resync reconciliation, terminal-state bug-1737/1738/1739/1764,
scheduler-sim verbosity, hermetic offline build, go mod vendor of Go deps
in apiserver -- 3375 pure-adds), ours +89 (job-finish robustness bug-1509
B1/B3, PEND-reason, worker reconnect resend, apiserver job control).

Conflict resolutions (kept both sides' fixes):
- controller_connector.c checking_connection(): combine theirs
  xcecd_resync_is_pending() rejoin deferral gate with ours
  resend_job_status() actual-state re-report on rejoin success.
- job.c: keep both unknown-finish registry (ours, bug-1509) and
  record_job_finish_fact_at (theirs, bug-1747); keep both
  mark_jobs_unknown_core family (ours) and record_job_reconcile_*
  (theirs). No double-notify: on_job_finish unchanged, theirs is
  resync-path-only.
- scheduler handler.c/test_handlers.c/meson: theirs schedExitJob
  exit_code branch (bug-1737) + ours jobDispatchFail success-ack no-op
  (bug-1512) + configpush enq_pend_update (bug-1513) both present; both
  test suites and REPL/sim targets wired.
- apiserver jobs.go: keep ours normalizeJobInt64/jobArrayIdx superset +
  JobIDHuman; drop theirs strict IntOrInt64 early-reject (superseded --
  ours tolerates float64/json.Number, no panic path).
- jobs_test.go: keep both regression suites (ours bug-1490, theirs
  bug-1742).
- .kiro/specs scheduler-sim-*/worker-rejoin-resync + .wolf/*: keep our
  deletions (established convention -- drop Rongsong bookkeeping on merge).

Co-Authored-By: Claude Code <noreply@anthropic.com>

.kiro/specs/commit-batch-window/requirements.md

@@ -0,0 +1,21 @@
+# Requirements Document
+
+## Project Description (Input)
+A general, time-windowed batch-commit facility: gather multiple WAL-worthy facts arriving within a short window and commit them together in a single majority-ack round (group commit), rather than each individual fact paying its own full commit-barrier round-trip. Any operation that needs an externally-visible side effect gated on durability (e.g. "don't tell the worker to start a job until that job's dispatch fact is majority-committed") should be able to enqueue its fact into the shared batch and be notified once the BATCH — not just its own individual entry — achieves majority commit.
+
+This builds on infrastructure that already partially exists: the "group-commit flusher" already advances `durable_lsn` asynchronously (bug-1013, bug-760), and job-array-single-wal-record already batches multiple array elements' submission into a single WAL record. This spec generalizes that pattern into a reusable facility any fact type can use, rather than a one-off per call site.
+
+**Motivating first use case (not the only one), and the strongest candidate specifically: batch/array job dispatch.** Batching only pays off when multiple facts genuinely cluster together in time — a single, isolated job dispatch gains essentially nothing from this facility (there is nothing else to amortize its round-trip cost against). A job ARRAY's dispatch decisions, by contrast, naturally arrive in a tight burst (the scheduler decides many elements together), which is exactly where group commit earns its cost — and exactly the workload pattern both bug-1698 and bug-1741 were reproduced against live this session (a 50-element array dispatching multiple elements to one worker within well under a second). Any design here should prioritize the array/burst case as the primary target, not attempt to make every standalone dispatch faster. dispatch_a_job_() (src/controller/job.c) currently sends xce_startjob_req to the worker BEFORE persist_submit_job_dispatch_fact() is even called — confirmed as a real, non-hypothetical gap via bug-1698's own live WAL evidence (job [12]'s dispatch was never even locally WAL-written before the primary froze). Making dispatch commit-gated would close that gap at the source (Raft's own leader-completeness guarantee then protects it), but only if batching keeps the added latency (measured ~167ms in this codebase's own bug-1487 live soak, dominated by standby fdatasync per bug-561) acceptable for high-throughput dispatch patterns like large job arrays.
+
+**Additional scope dimension: hook/notification timing.** dispatch_a_job_() currently mutates job state and fires HOOK_DETAIL_JOB_START/notifies the scheduler+apiserver synchronously and inline at dispatch time (persist_notify_job_updated_already_persisted, guarded by the t_apply_seam_notifying exactly-once flag) specifically BECAUSE the ordinary apply-time hook path (apply_committed_entry) would otherwise see a no-op transition and silently skip notifying (bug-1679's root cause). A general batch-commit facility moves "this fact is true" to a deferred, asynchronous, batch-boundary-aligned moment for EVERY fact type that uses it, not just dispatch — so hook/notification firing timing needs to be redesigned around that general batch-commit-confirmed moment, without reintroducing bug-1679's original gap (notify never fires) or a double-notify race (the exact class scheduler-dispatch-commit's own design already had to fix once for dispatch specifically). This is the same hook/event-notification machinery this session's bug-1731/1737/1738/1739 fixes were untangling elsewhere in persist.c — any batch-commit design here should be reviewed against that same family of fixes, not designed in isolation.
+
+Sibling specs: scheduler-dispatch-commit (implementation-complete, moved the RUN-state transition timing and added dispatch-failure retry classification in dispatch_a_job_() — a different, already-solved concern) and job-array-single-wal-record (the existing batching precedent, scoped to job submission only). worker-rejoin-resync (in progress, requirements phase) remains valuable as a reconciliation backstop regardless of whether this spec is ever pursued, since it also covers other classes of state divergence, not just the one dispatch-ordering gap this spec's motivating use case addresses.
+
+**Scope exclusion: user-synchronous-wait operations are not batch-eligible.** Not every WAL-worthy fact should participate in this facility — only facts where nothing is synchronously, human-observably waiting on the result (e.g. job dispatch, decided internally by the scheduler with no client blocked on it; a worker's job-finish report, arriving async off the critical path of any interactive command). Operations where a user is actively blocked at a terminal or client waiting for the reply — job submission (jobsub) foremost, and by the same reasoning any other admin/control command that already commits synchronously today (config push, job kill/control) — must be explicitly excluded from batching and continue to commit as soon as possible, not deliberately held to wait for a batch window to fill. Any design for this facility needs an explicit per-fact-type eligibility decision, not a blanket "everything batches" default.
+
+**Candidate mechanism: an explicit classifier, mirroring existing precedent.** This codebase already has exactly this shape of decision elsewhere: xce_classify_dispatch_fail() (dispatch_a_job_()'s own failure classifier, scheduler-dispatch-commit) maps an exhaustive site-identity enum to a decision (TERMINAL vs RETRYABLE) rather than scattering ad hoc logic per call site. A parallel classifier — mapping each fact type/call site to BATCHABLE vs IMMEDIATE commit eligibility — would be the idiomatically consistent way to implement the user-synchronous-wait exclusion above, and should be the starting design candidate rather than an ad hoc per-call-site flag.
+
+**Status note (2026-08-31): parked, not actively pursued.** Assessed as a substantial cross-cutting change touching the core commit pipeline and hook/notification timing broadly, not a small project — revisit in a dedicated future session.
+
+## Requirements
+<!-- Will be generated in /kiro-spec-requirements phase -->

.kiro/specs/commit-batch-window/spec.json

@@ -0,0 +1,24 @@
+{
+  "feature_name": "commit-batch-window",
+  "created_at": "2026-08-31T14:07:43Z",
+  "updated_at": "2026-08-31T14:20:00Z",
+  "language": "en",
+  "phase": "initialized",
+  "approvals": {
+    "requirements": {
+      "generated": false,
+      "approved": false
+    },
+    "design": {
+      "generated": false,
+      "approved": false
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false,
+  "sibling_specs": ["scheduler-dispatch-commit", "job-array-single-wal-record", "worker-rejoin-resync"],
+  "note": "Parked/tracked, not actively being pursued. Originally scoped narrowly as job-dispatch-commit-batching; broadened (2026-08-31, same session) to a general time-windowed batch-commit facility for ANY WAL-worthy fact needing a durability-gated external side effect, not just job dispatch -- job dispatch remains the motivating first use case. User assessed this as a substantial cross-cutting change (core commit pipeline, latency/throughput tradeoffs, hook/notification timing, soak-level correctness verification), not a small project. Revisit in a dedicated future session."
+}

.kiro/specs/scheduler-primary-lifecycle/design.md

@@ -0,0 +1,421 @@
+## Overview
+
+**Purpose**: This feature binds the scheduler process's lifecycle to the controller cluster's primary role, replacing the scheduler's independent-process-with-reactive-rejoin model with a controller-owned child process, so that a controller failover always results in a working scheduler within one promotion cycle instead of a potentially unbounded stuck-PEND deadlock (bug-1760).
+
+**Users**: XCE cluster operators, and transitively every job submitted to the cluster, benefit from this feature never having to notice or intervene after a controller failover.
+
+**Impact**: Changes the current independent, long-lived `xce_scheduler` process (joining any of a configured candidate list, reconnecting reactively) into a process spawned, owned, and torn down by whichever controller is currently primary, pinned to that one controller for its entire lifetime.
+
+### Goals
+- A controller starts exactly one scheduler process the moment it becomes primary, and that scheduler is usable (joined, receiving state) without any operator action.
+- A controller's own scheduler exits promptly on a graceful demotion, and independently detects and exits on any other loss of that controller's primacy (crash, network partition, wedge), without ever attempting to reconnect elsewhere.
+- Scheduler-lifecycle management can never compromise the controller's own liveness, HA correctness, or availability.
+- A wedged-but-connected scheduler is distinguished from a healthy one and recovered automatically.
+- Existing tooling and specs that assume an independently-started, reconnecting scheduler are identified and updated.
+
+### Non-Goals
+- No standby/redundant scheduler processes or lease-election mechanism (superseded by piggybacking on the controller's existing Raft election).
+- No general per-host role-manager component governing worker/controller/scheduler/apiserver placement.
+- No change to worker or apiserver deployment/supervision.
+- No change to the scheduler's job/queue/config-handling logic, wire behavior toward jobs, or picker/allocator internals — only its process lifecycle and its relationship to the controller cluster.
+
+## Boundary Commitments
+
+### This Spec Owns
+- The controller-side decision of *when* to start, stop, or restart its own scheduler child process (tied to promotion, demotion, and a detected wedge).
+- The mechanism used to start/stop/monitor that child process without blocking the controller's Raft drive threads.
+- The scheduler's own startup contract change (pinned to one controller, no candidate list) and its active liveness/primacy check of that one controller.
+- The new controller-to-scheduler wire message used for a graceful-exit notice.
+- Updating existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and flagging any other spec that assumes an independently-started, reconnecting scheduler.
+
+### Out of Boundary
+- The scheduler's job/queue/config decision logic (picker, allocator, plugins) — untouched.
+- The Raft election, replication, and commit-safety mechanisms the controller already implements — this feature is a *consumer* of "who is primary," never a modifier of how that is decided.
+- Worker and apiserver process lifecycle.
+- Any lease/renewal protocol or standby-scheduler redundancy (deliberately rejected — see research.md).
+- `ha_dispatch_gate`'s existing role-based write rejection — this feature depends on it, does not change it.
+
+### Allowed Dependencies
+- `src/controller/slowlane.c` (`slowlane_submit`) — the sole mechanism for running scheduler-management work off the Raft drive threads. No new async-execution primitive may be introduced.
+- `src/controller/ha.c`'s `raft_cb_become_leader` / `raft_cb_become_follower` — the sole trigger points for scheduler start/stop. No parallel promotion/demotion detection path may be built.
+- `src/controller/sched.c`'s existing `on_scheduler_join()` / `handle_scheduler_leave()` / `cluster_t.sched.ep` bookkeeping — extended, not replaced, for tracking whether the owned child has actually joined.
+- The existing controller-scheduler wire connection and Cap'n Proto schema (`src/schema/controller.capnp`) — the graceful-exit notice is a small addition to this existing channel, not a new transport.
+
+### Revalidation Triggers
+- Any change to `raft_cb_become_leader`'s or `raft_cb_become_follower`'s promotion/demotion gating order (a new gate inserted before or after the scheduler-start/stop call site changes when a scheduler is (not) started).
+- Any change to `slowlane_submit`'s dedup-key semantics or admission-cap behavior.
+- Any change to `ha_dispatch_gate`'s role-based opcode allow-lists (the safety argument that a stale scheduler connection cannot mutate state depends on this staying true).
+- Any other spec or tool that starts `xce_scheduler` directly (bypassing controller-owned startup) must be re-checked against this design's single-controller startup contract.
+
+### Project Invariants Checked
+This design touches the controller's HA/process-lifecycle surface, so it is checked against `.kiro/steering/ha-async-io.md`'s documented invariants directly, not deferred to a later audit:
+- **Thread-role contract** (no blocking work on iomgr/delaymgr): satisfied — every scheduler-management action (posix_spawn, waitpid, signal, status check) is submitted through the existing `slowlane_submit()` executor, never invoked inline from a Raft callback. See Architecture Pattern & Boundary Map and the SchedulerProcessManager component.
+- **Controller-never-self-terminate**: satisfied — a scheduler-management failure (start failure, stop timeout, wedge) is logged and retried/handled locally; it never crashes, hangs, or exits the controller process, and is never wired as a new promotion-refusal gate (Requirement 4.2, 4.4). See Error Handling.
+- **`ha_dispatch_gate` role-based write rejection**: depended upon, not modified — this design's safety argument (a stale/orphaned scheduler connection cannot mutate cluster state) rests on this existing, unchanged behavior (see research.md).
+
+## Architecture
+
+### Existing Architecture Analysis
+- The controller today has no concept of owning a child process of any kind — `cluster_t` has no process-management state. The only precedent for one XCE daemon supervising a child process is the worker daemon's job-execution supervision (`process_worker_manager.c`, `uv_ipc.c`), which is libuv-based and job-execution-specific (see research.md) and is not reused directly.
+- The scheduler today is a fully general Raft-cluster client: it holds a candidate list (`--peers`), connects via `xcecli_start`/`xcecli_join`, and recovers from any disconnect via the client library's own `XCECLI_FAILOVER_CALL` rotation plus its own reactive `sched_publish_check_rejoin()`. This general-client shape is replaced for the scheduler specifically; workers and the `xce_test`/CLI clients are unaffected and keep the existing candidate-list/failover-rotation model.
+- `ha_dispatch_gate` already enforces that only a primary controller accepts scheduler write opcodes (job dispatch, pend-reason) — this pre-existing behavior is why this feature is purely about availability, not correctness (see research.md).
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    subgraph HostA
+        CtrlA[Controller Instance A Primary]
+        SchedA[Scheduler Process]
+        CtrlA -->|spawns and owns| SchedA
+    end
+    subgraph HostB
+        CtrlB[Controller Instance B Standby]
+    end
+    subgraph HostC
+        CtrlC[Controller Instance C Standby]
+    end
+    CtrlA <-->|raft election and replication| CtrlB
+    CtrlA <-->|raft election and replication| CtrlC
+    CtrlB <-->|raft election and replication| CtrlC
+    SchedA -->|pinned single connection join and status query| CtrlA
+```
+
+**Architecture Integration**:
+- Selected pattern: controller-owned child process, triggered directly from the existing Raft promotion/demotion callbacks, executed through the existing SlowLane off-drive-thread executor.
+- Domain/feature boundaries: controller-side process ownership (`scheduler_proc.c`, new) is separate from controller-side scheduler-connection bookkeeping (`sched.c`, existing, extended); scheduler-side controller-liveness watching (`sched_owner.c`, new) is separate from scheduler-side scheduling logic (`scheduler.c`, unchanged except for removing the superseded rejoin logic).
+- Existing patterns preserved: SlowLane for off-drive-thread work; `ha_dispatch_gate` as the sole write-safety mechanism; the existing Cap'n Proto wire connection and request-handler dispatch shape on both sides.
+- New components rationale: `scheduler_proc.c` is a new domain (process ownership) with no existing owner; `sched_owner.c` is a new domain (single-controller liveness pinning) that cannot be folded into `scheduler.c`'s existing scheduling-engine responsibility without violating single-responsibility.
+- Steering compliance: respects `ha-async-io.md`'s thread-role contract (all blocking work — posix_spawn/waitpid/kill/status-query — runs via SlowLane or the scheduler's own non-drive-thread context, never inline on iomgr/delaymgr) and the controller-never-self-terminate invariant (a scheduler-management failure logs loudly and retries; it never causes the controller to crash, hang, or exit, and never becomes a new promotion-refusal gate).
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|-----------------|-------|
+| Controller process management | POSIX `posix_spawn`/`waitpid`/signal (C, `src/controller/scheduler_proc.c`) | Start, monitor, and stop the scheduler child process | `product.md`'s locked decision "no `fork()` for snapshots (mutex-inheritance deadlock in multithreaded controller)" applies just as much here: a raw `fork()` only duplicates the calling thread, so a lock held by any *other* thread (glibc malloc, XCE's own `mylock_t` locks) is duplicated in whatever state it was in, and the child can deadlock if it touches malloc or any XCE library call before `execve()` — being on a dedicated SlowLane worker does not change this, since the hazard comes from *other* threads in the same process, not the calling one. `posix_spawn()` is used instead specifically to avoid re-triggering this exact, already-documented risk class, not merely as a style preference |
+| Async execution | `slowlane_submit()` (existing, `src/controller/slowlane.c`) | Runs all scheduler-management actions off the Raft drive threads | Reused as-is, no changes |
+| Wire protocol | Cap'n Proto (existing, `src/schema/controller.capnp`) | Carries the new graceful-exit notice from controller to scheduler over the existing connection | One small schema addition; no new transport |
+| Scheduler startup config | CLI argument (existing mechanism, new single-address mode) | Controller passes its own address to the scheduler it spawns, replacing the multi-candidate `--peers` list for this specific relationship | Deployment tooling generating scheduler startup commands must change accordingly |
+
+## File Structure Plan
+
+### Directory Structure
+```
+src/
+├── controller/
+│   ├── scheduler_proc.c        # NEW: owns posix_spawn/monitor/signal/reap of this controller's scheduler child
+│   ├── include/
+│   │   └── scheduler_proc.h    # NEW: SchedulerProcessManager service interface
+│   ├── ha.c                    # MODIFIED: call scheduler_proc_start() at the tail of raft_cb_become_leader,
+│   │                           #   scheduler_proc_stop() at the top of raft_cb_become_follower
+│   └── sched.c                 # MODIFIED: extend join/leave bookkeeping to feed liveness/wedge detection
+├── scheduler/
+│   └── src/
+│       ├── sched_owner.c       # NEW: pins to one controller address, periodic liveness/role check, exits on loss
+│       ├── scheduler.c         # MODIFIED: remove sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin
+│       │                       #   and the multi-candidate startup path for this role
+│       ├── handler.c           # MODIFIED: scheduler_req_handler gains a case for the graceful-exit notice
+│       └── main.c              # MODIFIED: new single-controller-address startup argument
+├── schema/
+│   └── controller.capnp        # MODIFIED: add the controller-to-scheduler graceful-exit message
+└── controller/testsuite/, scheduler/testsuite/  # new/modified test files, see Testing Strategy
+```
+
+### Modified Files (tooling and specs, outside `src/`)
+- `.claude/skills/xce-deploy-ha/SKILL.md` — remove/replace Step 4b's manual `start-scheduler.sh` guidance; document that a running controller now starts its own scheduler automatically once it becomes primary.
+- `src/tools/deploy-ha.sh` — stop generating a standalone `start-scheduler.sh` intended for manual/independent invocation (or repurpose it to reflect the new single-controller-address contract if a manual escape hatch is still useful for local debugging).
+- `.claude/skills/xce-test-ha/run-ha-soak.sh` — remove the worker-rejoin-resync soak coverage's `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` manual scheduler management added this session (bug-1760's workaround becomes unnecessary — the controller now manages this itself); the soak harness should instead just verify a scheduler is present after each promotion.
+
+## System Flows
+
+### Promotion: controller starts its scheduler
+```mermaid
+sequenceDiagram
+    participant Raft as Raft Election
+    participant HA as ha.c become leader
+    participant Lane as SlowLane Executor
+    participant Proc as scheduler_proc.c
+    participant Sched as Scheduler Process
+    Raft->>HA: on_become_leader
+    HA->>HA: run existing promotion eligibility gates
+    HA->>Lane: slowlane_submit start action dedup key this controller
+    Lane->>Proc: run start action on a lane worker
+    Proc->>Sched: posix_spawn passing this controller address
+    Sched->>HA: join using pinned single controller address
+    HA->>Sched: push jobs queues config existing behavior
+```
+Gating conditions: the SlowLane submission only happens after every existing promotion-refusal gate has already passed (Requirement 1.1). A failed or deferred `slowlane_submit()` (already-in-flight dedup, or admission-cap deferral) is not an error — the work is still queued and will run (Requirement 1.5's retry is satisfied by SlowLane's own queuing semantics plus an explicit retry-on-failure inside the submitted action).
+
+### Graceful demotion: controller signals its scheduler to exit
+```mermaid
+sequenceDiagram
+    participant Raft as Raft Election
+    participant HA as ha.c become follower
+    participant Lane as SlowLane Executor
+    participant Proc as scheduler_proc.c
+    participant Sched as Scheduler Process
+    HA->>Lane: slowlane_submit stop action dedup key this controller
+    HA->>HA: continue completing its own role transition immediately
+    Lane->>Proc: run stop action on a lane worker
+    Proc->>Sched: send graceful exit notice over existing connection
+    Sched->>Sched: finish in flight work and exit
+    Proc->>Proc: wait bounded time then reap or force terminate
+```
+The role transition in `raft_cb_become_follower` never waits on the SlowLane task's completion (Requirement 2.3) — submission is fire-and-forget from the Raft callback's perspective.
+
+### Scheduler self-detects loss of its owning controller
+```mermaid
+sequenceDiagram
+    participant Sched as Scheduler Process
+    participant Owner as sched_owner.c
+    participant Ctrl as Its Owning Controller
+    loop periodic check
+        Owner->>Ctrl: status query over the pinned connection
+        Ctrl-->>Owner: role and reachability
+    end
+    alt controller unreachable or reports it is no longer primary
+        Owner->>Sched: request exit
+        Sched->>Sched: exit
+    end
+```
+This path is independent of the graceful-exit notice above — it is the general-case detector covering crash, network partition, and any demotion trigger the explicit notice did not reach in time (Requirement 3.1-3.3). No candidate list exists to fall back to (Requirement 3.4) — the only outcome of a failed check is exit.
+
+### Wedged-scheduler detection and recovery
+```mermaid
+sequenceDiagram
+    participant HA as ha.c or sched.c
+    participant Lane as SlowLane Executor
+    participant Proc as scheduler_proc.c
+    participant Sched as Scheduler Process
+    HA->>Proc: observe scheduler connected but no scheduling progress signal within threshold
+    Proc->>Lane: slowlane_submit restart action dedup key this controller
+    Lane->>Proc: run restart action
+    Proc->>Sched: terminate
+    Proc->>Proc: posix_spawn a replacement per the promotion flow
+```
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1, 1.2, 1.3 | Scheduler starts on promotion, at most one in flight, no duplicate start | scheduler_proc.c, ha.c | `scheduler_proc_start` | Promotion flow |
+| 1.4 | Scheduler joins and receives job/queue/config state after being started | scheduler_proc.c, sched.c | `scheduler_proc_start` | Promotion flow |
+| 1.5 | Controller retries on start failure, logs loudly | scheduler_proc.c | `scheduler_proc_start` | Promotion flow |
+| 2.1, 2.2, 2.3 | Graceful exit signaled on demotion; scheduler exits within bounded time; controller's own role transition never waits on it | scheduler_proc.c, ha.c, controller.capnp addition, scheduler handler.c | `scheduler_proc_stop`, graceful-exit wire message | Graceful demotion flow |
+| 2.4 | No job left inconsistent or lost regardless of how the scheduler was terminated | scheduler_proc.c (bounded stop/force-kill), controller's existing WAL-commit ownership (unchanged) | `scheduler_proc_stop` | Graceful demotion flow, Data Models |
+| 3.1, 3.2, 3.3 | Scheduler actively verifies its owning controller is reachable and primary; exits on failure of either check | sched_owner.c | `sched_owner_watch_start` | Self-detection flow |
+| 3.4 | Scheduler never joins or operates against any controller other than the one that started it | sched_owner.c, scheduler.c (rejoin logic removed) | `sched_owner_watch_start` | Self-detection flow |
+| 3.5 | Abrupt host/process death needs no scheduler-side action | (no component — a structural consequence of co-location, not new logic) | n/a | Self-detection flow (contrast case) |
+| 4.1, 4.2, 4.3 | Controller never blocks heartbeats/elections, never crashes/hangs, always logs scheduler-management failures | scheduler_proc.c (via SlowLane) | `scheduler_proc_start`/`stop`/`check_liveness` (all SlowLane-submitted) | All flows |
+| 4.4 | Absence of a running scheduler never blocks a controller's own HA role | scheduler_proc.c, ha.c (unchanged promotion gates) | n/a | Promotion flow |
+| 5.1, 5.2 | Controller monitors scheduling progress, not just connection state; flags no-progress past a threshold as unresponsive | SchedulerConnectionBookkeeping (sched.c), scheduler_proc.c | `scheduler_proc_check_liveness` | Wedge detection flow |
+| 5.3, 5.4 | Unresponsive scheduler is terminated and replaced, with an operator-visible log | scheduler_proc.c | `scheduler_proc_check_liveness` | Wedge detection flow |
+| 6.1 | Existing deployment tooling documentation updated to reflect controller-owned scheduler startup | xce-deploy-ha SKILL.md, deploy-ha.sh | n/a (tooling) | Migration Strategy |
+| 6.2 | Existing specs/tooling depending on scheduler survive-and-reconnect behavior identified and reconciled | xce-test-ha run-ha-soak.sh (this session's own bug-1760 workaround is the concrete example identified and reconciled here) | n/a (tooling) | Migration Strategy, Testing Strategy E2E |
+| 6.3 | Scheduler's observable job/queue/config behavior unchanged while running under a live primary | scheduler.c (unchanged scheduling logic) | n/a | Data Models |
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
+|-----------|--------------|--------|---------------|--------------------------|-----------|
+| SchedulerProcessManager | Controller | Owns posix_spawn/monitor/signal/reap of this controller's scheduler child | 1, 2, 4, 5 | SlowLane executor (P0), ha.c callbacks (P0) | Service |
+| SchedulerConnectionBookkeeping | Controller | Existing join/leave tracking, extended to feed liveness/progress signal | 5 | SchedulerProcessManager (P1) | Service |
+| SchedOwnerWatch | Scheduler | Pins to one controller address; periodic liveness/role check; exits on loss or graceful notice | 3 | Existing wire connection/status query (P0) | Service |
+| Graceful-Exit Wire Message | Wire protocol | Controller-to-scheduler notice carried on the existing connection | 2 | Cap'n Proto schema (P0) | Event |
+
+### Controller
+
+#### SchedulerProcessManager
+
+| Field | Detail |
+|-------|--------|
+| Intent | Start, monitor, signal, and reap this controller's own scheduler child process, entirely off the Raft drive threads |
+| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3, 5.4 |
+
+**Responsibilities & Constraints**
+- Owns the decision of whether a scheduler child is currently running for this controller instance; `ha.c` never spawns or signals a process directly.
+- Every action (start, stop, restart) is submitted to the SlowLane executor with a dedup key scoped to this controller instance, never invoked inline from a Raft callback.
+- Never allowed to block, crash, or cause the controller process to exit — every failure path logs an operator-visible error and either retries (start) or proceeds with the role transition regardless (stop).
+
+**Dependencies**
+- Outbound: SlowLane executor (`slowlane_submit`) — P0, the only allowed execution path for this component's blocking work.
+- Inbound: `ha.c`'s `raft_cb_become_leader` (calls `scheduler_proc_start`), `raft_cb_become_follower` (calls `scheduler_proc_stop`) — P0.
+- Inbound: a periodic liveness check (driven from the existing controller heartbeat cadence, not a new timer thread) — P1, feeds `scheduler_proc_check_liveness`.
+
+**Contracts**: Service [x]
+
+##### Service Interface
+```c
+/* src/controller/include/scheduler_proc.h */
+
+/* Requests that this controller's scheduler child be started, if one is not
+ * already running or already starting. Non-blocking: submits to the SlowLane
+ * executor and returns immediately. Safe to call multiple times in quick
+ * succession — the SlowLane dedup key ensures at most one start action is
+ * ever in flight per controller instance.
+ * Returns 0 if the request was accepted (started or already in flight/running),
+ * a negative errno if the request itself could not even be submitted (e.g.
+ * SlowLane shutting down) — never blocks, never crashes the caller.
+ */
+int scheduler_proc_start(cluster_t *c);
+
+/* Requests that this controller's scheduler child be stopped: sends the
+ * graceful-exit notice, waits up to timeout_ms for the child to exit, then
+ * force-terminates if it has not. Non-blocking to the caller: submitted to
+ * the SlowLane executor. Does not block the caller's own role transition.
+ */
+int scheduler_proc_stop(cluster_t *c, uint32_t timeout_ms);
+
+/* Evaluates whether the currently-running scheduler child (if any) is making
+ * scheduling progress, using the existing push/acknowledgment traffic
+ * (SchedulerConnectionBookkeeping) as the progress signal. If the child is
+ * connected but has made no progress for longer than the configured
+ * threshold, submits a restart action. A no-op if no scheduler child is
+ * currently owned. Non-blocking.
+ */
+int scheduler_proc_check_liveness(cluster_t *c);
+```
+- Preconditions: `c` is a valid, initialized `cluster_t`; called only from the controller's own process.
+- Postconditions: the requested action is either already complete, in flight on a SlowLane worker, or queued — never silently dropped.
+- Invariants: at most one scheduler child process is owned by a given controller instance at any time; a stop-then-start (demotion followed by a fresh promotion of the same instance) always completes the stop's reap before a new start is submitted, since a demoted instance must win a fresh election before it can be promoted again.
+
+**Implementation Notes**
+- Integration: hook call sites are the tail of `raft_cb_become_leader` (after all existing promotion-refusal gates) and the top of `raft_cb_become_follower`.
+- Validation: an operator-visible log line on every start failure, stop-timeout-forced-kill, and liveness-triggered restart (Requirement 4.3, 5.4).
+- Risks: sizing the shared SlowLane instance's `max_inflight`/`n_workers` for the additional scheduler-management workload alongside its existing consumers is a follow-up check, not a design change (see research.md).
+
+#### SchedulerConnectionBookkeeping (extends existing `sched.c`)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Track whether the owned scheduler child has joined and is making scheduling progress, feeding `scheduler_proc_check_liveness` |
+| Requirements | 5.1, 5.2 |
+
+**Responsibilities & Constraints**
+- Extends the existing `on_scheduler_join()`/`handle_scheduler_leave()`/`cluster_t.sched.ep` bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it.
+- Does not itself decide to restart anything — it only exposes the progress signal `scheduler_proc_check_liveness` reads.
+
+**Dependencies**
+- Inbound: existing scheduler request handling in `cluster.c`/`sched.c` — P1.
+- Outbound: read by SchedulerProcessManager — P1.
+
+**Contracts**: Service [x]
+
+**Implementation Notes**
+- Integration: reuses existing traffic; no new synthetic heartbeat payload is introduced (see research.md Risks).
+
+### Scheduler
+
+#### SchedOwnerWatch
+
+| Field | Detail |
+|-------|--------|
+| Intent | Pin the scheduler to the single controller instance that started it; periodically confirm that controller is reachable and still primary; exit otherwise |
+| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |
+
+**Responsibilities & Constraints**
+- At startup, the scheduler receives exactly one controller address (not a candidate list) and establishes its connection only to that address.
+- Periodically issues the existing role-agnostic status query against that one connection; on an unreachable connection or a reported non-primary role, requests the scheduler process to exit.
+- Never attempts any other controller address, and never re-derives a candidate list — this component structurally has no such list to fall back to.
+- Also receives and acts on the graceful-exit notice (Requirement 2.2) via the existing request-handler dispatch, exiting within a bounded time.
+
+**Dependencies**
+- Outbound: the existing controller-scheduler wire connection/status query — P0.
+- Inbound: `scheduler_req_handler`'s new graceful-exit case — P0.
+
+**Contracts**: Service [x]
+
+##### Service Interface
+```c
+/* src/scheduler/include/sched_owner.h */
+
+/* Starts the periodic ownership-watch loop against the single controller
+ * address this scheduler process was given at startup. Runs on its own
+ * thread (mirroring the worker daemon's existing checking_connection()
+ * cadence/shape), independent of the picker/allocator threads.
+ */
+int sched_owner_watch_start(const char *owner_host, uint16_t owner_port);
+
+/* Called by the request handler when the graceful-exit notice arrives on the
+ * existing connection. Triggers the same exit path the periodic watch loop
+ * uses on a failed check.
+ */
+void sched_owner_request_exit(void);
+```
+- Preconditions: `owner_host`/`owner_port` identify the specific controller instance that spawned this process.
+- Postconditions: the process exits once either the periodic check fails or a graceful-exit notice is received — no other outcome resumes normal operation with a different controller.
+- Invariants: never issues `xcecli_join`/any wire call against any address other than `owner_host:owner_port`.
+
+**Implementation Notes**
+- Integration: replaces `sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` in `scheduler.c` entirely — these are removed, not left dormant, since Requirement 3.4 makes their cross-controller-rejoin purpose permanently unreachable.
+- Risks: the periodic check interval trades detection latency against status-query load; finalize the exact interval during implementation, following the worker's existing `update_interval`-style convention rather than inventing a new tunable pattern.
+
+### Wire Protocol
+
+#### Graceful-Exit Wire Message
+
+**Event Contract**
+- Published by: the controller, from `scheduler_proc_stop`'s SlowLane action, over the existing controller-scheduler connection.
+- Subscribed by: the scheduler's existing `scheduler_req_handler` (`src/scheduler/src/handler.c`), which invokes `sched_owner_request_exit()`.
+- Delivery: best-effort over the existing persistent connection — if delivery fails (connection already gone), `scheduler_proc_stop`'s bounded wait-then-force-terminate path (Requirement 2.3) still guarantees the child does not outlive the timeout, and `SchedOwnerWatch`'s independent periodic check (Requirement 3) is the backstop if the notice is lost.
+- Ordering/idempotency: at most one graceful-exit notice is ever sent per stop action (SlowLane dedup); receiving it more than once (e.g. a retry) is a no-op if the process is already exiting.
+
+## Data Models
+
+No new persisted entity is introduced. The scheduler remains fully stateless with respect to durable storage — its in-memory job/queue/config state is always rebuilt from the controller's existing push traffic on join, unchanged by this feature (Requirement 6.3). The only new in-memory state is:
+- Controller side: which scheduler child (if any) this controller instance currently owns, its process handle, and its last-progress timestamp — process-lifetime only, never persisted, rebuilt from scratch on controller restart (a restarted controller that is primary starts a fresh scheduler exactly as any other promotion would).
+- Scheduler side: the single owning-controller address it was started with — process-lifetime only.
+
+Since nothing here is written to or read from a durable log, snapshot, or checkpoint, the project's save/restore field-completeness review does not apply to this design.
+
+## Error Handling
+
+### Error Strategy
+Every failure in scheduler-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (`ha-async-io.md`, `controller/CLAUDE.md`): log an operator-visible error, never crash or hang the controller, and retry where retrying is meaningful.
+
+### Error Categories and Responses
+- **Scheduler fails to start** (posix_spawn failure, immediate crash): `scheduler_proc_start` logs an error and retries on the controller's existing heartbeat-driven cadence while it remains primary (Requirement 1.5); the controller itself continues serving every other role unaffected (Requirement 4.4).
+- **Scheduler does not exit within the graceful-exit timeout**: `scheduler_proc_stop` force-terminates the child after the bounded wait; the controller's own demotion completes regardless (Requirement 2.3).
+- **Scheduler connected but not progressing (wedged)**: treated as unresponsive once past the threshold, terminated and replaced (Requirement 5.3), with an operator-visible log (Requirement 5.4).
+- **Scheduler cannot reach or confirm its owning controller**: the scheduler itself exits (Requirement 3.2, 3.3) — no error is reported back to the controller for this case, since the controller side detects the resulting disconnect via existing `handle_scheduler_leave()` bookkeeping.
+
+### Monitoring
+All new log lines (start failure, forced-kill-on-timeout, liveness-triggered restart) use the existing controller logging conventions (`WARN`/`ERROR` macros already used throughout `ha.c`) — no new logging mechanism is introduced.
+
+## Testing Strategy
+
+### Unit Tests
+- `scheduler_proc_start` submits exactly one SlowLane action per promotion and a second concurrent call while one is in flight is deduped (not double-submitted) — directly exercises Requirement 1.2/1.3 via the existing `test_slowlane.c` dedup behavior.
+- `raft_cb_become_leader` calls `scheduler_proc_start` only after every existing promotion-refusal gate returns eligible, and does not call it on any refusal path — regression-guards Requirement 1.1 against a future gate reordering.
+- `raft_cb_become_follower` calls `scheduler_proc_stop` and completes its own role transition without waiting on that call's result — regression-guards Requirement 2.3.
+- `SchedOwnerWatch`'s periodic check exits the process on a simulated unreachable controller and on a simulated non-primary role response, and never attempts a second address — directly exercises Requirement 3.1-3.4.
+- `scheduler_proc_check_liveness` triggers a restart only when the last-progress timestamp exceeds the threshold while the connection remains open (not on a disconnect, which is a separate, already-detected case) — exercises Requirement 5.1/5.2.
+
+### Integration Tests
+- A real 3-node controller cluster promotion results in a scheduler process appearing, joining, and receiving job/queue/config state, with no manual step — exercises Requirement 1.1-1.4 end to end.
+- A graceful step-down (forced via the existing test-only demotion trigger) results in the prior primary's scheduler exiting within the bounded timeout, verified by process absence, not just a log line — exercises Requirement 2.1/2.2.
+- A hard-killed primary (matching this session's original bug-1760 repro exactly) results in the orphaned scheduler self-exiting within its own periodic-check interval, and the newly-elected primary's fresh scheduler successfully dispatching a job submitted after the failover — exercises Requirement 3.1-3.5 and directly closes bug-1760.
+- A scheduler artificially wedged (e.g. its progress-signal thread blocked) while its connection stays open is detected and replaced — exercises Requirement 5.1-5.3.
+- A job dispatched by a scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement scheduler to consider — exercises Requirement 2.4.
+
+### E2E / Soak Tests
+- Extend `.claude/skills/xce-test-ha/run-ha-soak.sh`'s sequential and chaotic modes (already exercising controller failover) to assert a working scheduler is present after every round without any harness-side scheduler start/stop action — this replaces the manual `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` logic added this session as bug-1760's workaround, which becomes dead code once this feature ships.
+- A rapid sequence of promotions/demotions (chaotic mode's overlapping-kill pattern) produces no scheduler spawn/kill thrash beyond one scheduler per genuinely-promoted instance — exercises the SlowLane-dedup mitigation documented in research.md's Risks.
+
+## Migration Strategy
+
+```mermaid
+flowchart TB
+    Phase1[Phase 1 controller and scheduler code changes land behind no behavior change until wired]
+    Phase2[Phase 2 wire promotion and demotion hooks and pinned single controller startup]
+    Phase3[Phase 3 update xce deploy ha and xce test ha tooling to stop manually starting the scheduler]
+    Phase4[Phase 4 remove superseded rejoin logic and multi candidate scheduler startup path]
+    Phase1 --> Phase2 --> Phase3 --> Phase4
+```
+- Rollback trigger: if a real deployment depends on independently placing the scheduler on a dedicated host for resource-isolation reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (superseded-logic removal) rather than force the migration through, since Phase 4 is the only phase that is hard to reverse cheaply.
+- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
+
+## Open Questions / Risks
+- The exact wire opcode/schema shape for the graceful-exit notice is left to implementation (a small, additive `controller.capnp` change, following the same pattern recent specs have used for new controller-to-daemon messages) — this is appropriately an implementation detail, not a design decision, since no existing opcode fits (confirmed by inspection: `OPCode_shutdown` is an inbound admin-to-controller command, not a controller-to-scheduler one, and is not reusable here).
+- The precise "no scheduling progress" threshold and periodic-check interval (Requirement 5, Requirement 3) are implementation-time tuning decisions, not design decisions — see research.md Risks for the reasoning against inventing a bespoke synthetic heartbeat.
+- Confirm during implementation whether any other in-tree consumer starts `xce_scheduler` with a multi-candidate `--peers` list for a reason unrelated to this feature (e.g. a test fixture) that would need explicit review rather than silent breakage (carried over from research.md).

.kiro/specs/scheduler-primary-lifecycle/requirements.md

@@ -0,0 +1,85 @@
+# Requirements Document
+
+## Project Description (Input)
+Who has the problem: XCE cluster operators, and by extension every job that depends on the scheduler to be reachable after a controller failover.
+
+Current situation: the scheduler (`xce_scheduler`) runs today as a fully independent, long-lived process that joins the controller cluster via `xcecli_join(HostRole_scheduler)` and publishes dispatch/pend-reason decisions via `sched_publish_dispatch()`/`sched_publish_pend_reason()`. Its only rejoin mechanism after a controller failover is reactive: `sched_publish_check_rejoin()` re-issues `xcecli_join()` only as a side effect of its own *next successful* publish call. After a real controller failover this creates a chicken-and-egg deadlock — the scheduler has nothing queued to publish (no new job ever reaches its picker, since the controller-side registration of "where the scheduler is" is live-connection-scoped state on whichever controller instance the scheduler last explicitly joined, and does not appear to carry over to a newly-elected primary) — so it can never publish anything to trigger its own rejoin. This was live-confirmed against a real 3-node cluster: after killing the primary, the scheduler showed zero log activity of any kind (no WARN/INFO/ERROR, no reconnect attempt) for 60+ seconds, while a job submitted to the new primary sat stuck PEND. Logged as bug-1760 in `.wolf/buglog.json` with full repro evidence. Separately, there is today no story at all for what happens if the scheduler's own host/process dies, is network-partitioned, or wedges internally (e.g. a disk stall blocking its recording I/O, or a deadlocked allocator thread) — no periodic health check, no standby, no automatic recovery.
+
+What should change (scope decided across discussion, see rationale below): bind the scheduler process's lifecycle to the controller cluster's primary role, replacing its current independent-process-with-reactive-rejoin model entirely — including replacing its client-side candidate-rotation/rejoin logic, not just supplementing it.
+- When a controller transitions to `CTRL_PRIMARY` (wins election / is promoted), it spawns and owns a scheduler process **co-located on its own host** (a genuine child process, not a signal to some independently-running scheduler).
+- When that controller loses primary status **gracefully** (demoted, discovers a higher term, steps down), it explicitly signals its child scheduler to exit before completing the transition.
+- The scheduler process itself **actively monitors** whether the specific controller instance that started it is still alive and still primary, and terminates itself the moment it can no longer confirm that — this is the primary detection mechanism (not a passive fallback), because it also covers cases OS-level parent-death propagation cannot guarantee (e.g. the controller process dying while its host survives). Co-located death (host crash, etc.) provides a redundant backstop for the same outcome. Critically, the scheduler does **not** attempt to rediscover or reconnect to whichever controller becomes the *next* primary — that job belongs entirely to the next primary's own act of spawning a fresh scheduler. This is a deliberate simplification: a scheduler's operating logic no longer needs any cross-primary candidate-rotation or rejoin behavior at all, because its lifetime is scoped to exactly one controller instance's primacy.
+- The newly-elected primary spawns its own fresh scheduler exactly as it would on any other promotion, via the controller's existing, already-hardened Raft election — no new election/lease mechanism for schedulers.
+- This eliminates: `sched_needs_rejoin`/`sched_publish_check_rejoin`'s reactive rejoin logic and the scheduler's client-side failover candidate rotation for its own role (no longer needed — a scheduler's lifetime is now scoped to a single controller's primacy, so it never needs to survive and reconnect across a primary change); the need for a scheduler-side lease/renewal mechanism; and the need for standby scheduler processes — host/process death is subsumed into "controller host dies," a failure mode the controller's HA layer already handles robustly, so no second, parallel HA/authority mechanism is being built for schedulers.
+- In scope, smaller: a liveness/health check the parent controller performs on its own live child scheduler, to catch the case where the process is alive (so co-location death didn't trigger) but internally wedged (e.g. blocked disk I/O, deadlocked picker/allocator thread) and therefore not doing useful work despite an open connection. This does not require a lease or standby scheduler — it is a local parent-watches-child health signal, and on failure the controller may restart its own child scheduler.
+- Explicitly OUT of scope for this spec (raised and deliberately deferred during discussion):
+  - Standby/redundant scheduler processes competing for a lease — rejected in favor of piggybacking on the controller's own already-tested election mechanism.
+  - A general per-host "role manager" component that starts/stops arbitrary XCE components (worker, controller, scheduler, apiserver) based on declared host roles and cluster state. This was discussed as a possible bigger direction but deliberately deferred: the controller is uniquely positioned to trigger scheduler startup at zero extra latency the instant it becomes primary, whereas a separate agent would need to watch controller election state itself and introduces a new process whose own supervision becomes a new problem. If a genuine, independent need for general host/role management across all XCE components emerges later, it should be its own spec, not inherited scope from this one.
+  - Any change to how `worker` or `apiserver` processes are deployed/supervised.
+
+Constraint carried into design: scheduler and controller currently value independent deployability (separate hosts, separate scaling/tuning, separate release cadence per `.kiro/steering/structure.md`'s component layout). This spec deliberately trades that independence away for the scheduler specifically, in exchange for eliminating an entire class of availability bugs (reactive rejoin deadlock, host-death handling, wedge detection) by reusing already-hardened controller HA machinery instead of building a parallel one. The design phase must make this tradeoff and its consequences (scheduler can no longer run on a dedicated host, failover now includes scheduler cold-start time, controller gains new child-process-supervision responsibility) explicit rather than incidental.
+
+Related code to ground requirements/design in: `src/controller/ha.c` (promotion/demotion — e.g. `ha_start_primary_heartbeat`, `raft_cb_become_leader`; `ha_dispatch_gate` already safely rejects scheduler writes against a non-primary, confirmed by reading the code, so no split-brain risk exists today from a stale scheduler connection — this spec is solely about availability, not correctness), `src/controller/sched.c` (`on_scheduler_join()`, `handle_scheduler_leave()` — existing controller-side scheduler-connection lifecycle hooks), `src/include/xce/cluster_def.h` (`XCE_CONTROLLER_HOOK_SCHEDULER`, `HOOK_DETAIL_SCHEDULER_JOIN`/`HOOK_DETAIL_SCHEDULER_DEAD` hooks already exist and may be relevant plumbing), `src/scheduler/src/scheduler.c` (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` — the mechanism being replaced), `src/worker/src/process_worker_manager.c` (existing precedent in this codebase for a daemon spawning/supervising a child process, though for job execution rather than another XCE daemon), `.kiro/steering/ha-async-io.md` (controller-never-self-terminate invariant, and the thread-role contract that no liveness-carrying thread — iomgr/delaymgr — may block; any new spawn/monitor/signal mechanism the controller gains must respect this), `.claude/skills/xce-deploy-ha/SKILL.md` and `.claude/skills/xce-test-ha/run-ha-soak.sh` (existing tooling that models the scheduler as an independently-started process via `start-scheduler.sh` — needs to be reconciled with the new model, since multiple existing specs/soak-harness modes depend on today's independent-process assumption). Bug-1760 in `.wolf/buglog.json` has the full live-repro evidence motivating this spec.
+
+## Boundary Context
+
+- **In scope**: controller-triggered scheduler startup on promotion; graceful scheduler shutdown signaled on demotion; scheduler self-termination via active monitoring of its own originating controller's liveness/primary status; controller detection of, and response to, an unresponsive-but-alive ("wedged") child scheduler; controller resilience so that scheduler-lifecycle management never compromises the controller's own availability or HA correctness; reconciling existing deployment tooling and specs that assume an independently-started scheduler.
+- **Out of scope**: standby/redundant scheduler processes and any lease-election mechanism for schedulers; a general host/role manager governing worker, controller, or apiserver process lifecycle; any change to worker or apiserver deployment/supervision.
+- **Adjacent expectations**: this feature depends on the controller's existing Raft-based primary election (unchanged by this spec) as the sole source of truth for "who may run a scheduler right now." It also depends on the existing `ha_dispatch_gate` behavior (a non-primary controller already rejects scheduler write operations) continuing to hold, since this spec's safety argument rests on that already-verified behavior rather than re-implementing it. Existing operator-facing deployment tooling that starts a scheduler as an independent process (documented in `xce-deploy-ha`/`xce-test-ha`) is expected to change to reflect controller-owned scheduler startup; any spec or tooling that still assumes an independently-started, long-lived scheduler process is an adjacent consumer that must be revisited, not an unstated exception to this feature's behavior.
+
+## Requirements
+
+### Requirement 1: Scheduler startup on controller promotion
+**Objective:** As a cluster operator, I want a scheduler to start automatically whenever a controller becomes primary, so that the cluster can resume scheduling jobs without any manual intervention after a failover.
+
+#### Acceptance Criteria
+1. When a controller transitions to the primary role, the Controller shall start a Scheduler Process associated with itself.
+2. While a controller remains primary, the Controller shall ensure at most one Scheduler Process it started is running.
+3. If a controller is already primary and already has a running Scheduler Process associated with itself, the Controller shall not start a second Scheduler Process.
+4. When the Controller starts a Scheduler Process, the Scheduler Process shall join the cluster and become able to receive job, queue, and configuration state from that specific controller instance.
+5. If the Controller fails to start a Scheduler Process, the Controller shall record an operator-visible error and shall continue to retry starting one while it remains primary.
+
+### Requirement 2: Scheduler shutdown on graceful controller demotion
+**Objective:** As a cluster operator, I want the scheduler tied to a controller to stop cleanly when that controller steps down, so that no scheduler keeps running on behalf of a controller that is no longer authoritative.
+
+#### Acceptance Criteria
+1. When a controller that started a Scheduler Process transitions away from the primary role through a graceful demotion (voluntary step-down, or discovering a higher election term while still running), the Controller shall signal that Scheduler Process to terminate.
+2. When a Scheduler Process receives a termination signal from its originating controller, the Scheduler Process shall exit within a bounded time.
+3. The Controller shall not remain in the primary-to-standby transition indefinitely waiting on its Scheduler Process to exit; a Scheduler Process that does not exit within the bounded time shall not prevent the Controller from completing its own role transition.
+4. When a Scheduler Process is terminated for any reason (graceful signal, self-detected loss of its originating primary, or abrupt process/host death), no job shall be left in an inconsistent or lost state as a result — any job the terminated Scheduler Process had not yet caused to be durably recorded shall remain available for a subsequent Scheduler Process to consider.
+
+### Requirement 3: Scheduler self-termination on loss of its originating primary controller
+**Objective:** As a cluster operator, I want the scheduler itself to notice when its own controller is no longer a live primary and exit on its own, so that a scheduler can never keep running detached from any authoritative controller, and so that no cross-controller reconnect logic is needed in the scheduler at all.
+
+#### Acceptance Criteria
+1. While a Scheduler Process is running, the Scheduler Process shall continuously verify that the specific controller instance that started it is reachable and still holds the primary role.
+2. When a Scheduler Process can no longer confirm that its originating controller instance is reachable, the Scheduler Process shall terminate.
+3. When a Scheduler Process learns that its originating controller instance is no longer primary, the Scheduler Process shall terminate, even if that controller instance is still reachable.
+4. The Scheduler Process shall not attempt to join, rejoin, or continue operating against any controller instance other than the one that started it.
+5. If the host or process running a Scheduler Process terminates abruptly (crash, host failure) before the Scheduler Process can detect the loss of its originating controller on its own, no further scheduler-side action is required — the Scheduler Process is expected to be gone as a direct consequence of that failure.
+
+### Requirement 4: Controller resilience to scheduler-lifecycle management
+**Objective:** As a cluster operator, I want a controller's own availability and correctness to never depend on its child scheduler's health, so that scheduler-management responsibilities cannot introduce a new way for the controller to become unavailable.
+
+#### Acceptance Criteria
+1. The Controller shall never block its own heartbeat processing or election-timeout handling while starting, monitoring, or signaling a Scheduler Process.
+2. If starting, monitoring, or signaling a Scheduler Process fails for any reason, the Controller shall not crash, hang, or self-terminate as a result.
+3. The Controller shall log an operator-visible error whenever it fails to start, loses, or forcibly restarts a Scheduler Process it owns.
+4. The absence of a running Scheduler Process shall not prevent a controller from being elected primary, remaining primary, or serving read-only and replication traffic.
+
+### Requirement 5: Detecting and recovering a wedged Scheduler Process
+**Objective:** As a cluster operator, I want the controller to notice when its own scheduler is alive but not making progress, so that jobs do not silently stop being scheduled while everything still looks connected.
+
+#### Acceptance Criteria
+1. While a Controller has a running Scheduler Process, the Controller shall monitor whether that Scheduler Process is making scheduling progress, not merely whether its connection remains open.
+2. If a Scheduler Process is connected but fails to make scheduling progress for longer than an operator-observable threshold, the Controller shall treat that Scheduler Process as unresponsive.
+3. When the Controller treats its Scheduler Process as unresponsive, the Controller shall terminate that Scheduler Process and start a replacement.
+4. The Controller shall log an operator-visible event whenever it terminates a Scheduler Process because it was found unresponsive.
+
+### Requirement 6: Compatibility with existing scheduler operation and tooling
+**Objective:** As a cluster operator or spec author relying on existing scheduler deployment tooling, I want the change in scheduler ownership to be reflected everywhere the scheduler's lifecycle is currently assumed to be independent, so that existing deployments, tests, and specs do not silently break or become misleading.
+
+#### Acceptance Criteria
+1. Where existing deployment tooling documents starting a scheduler as an independent, manually-managed process, that tooling shall be updated to reflect that a running controller now starts its own scheduler automatically.
+2. Where an existing spec or test tooling depends on a scheduler surviving and reconnecting across a controller failover, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
+3. The Scheduler Process's observable behavior toward jobs, queues, and configuration (join, receive state, dispatch/pend decisions) shall remain unchanged by this feature for the duration that it is running under a live primary controller.

.kiro/specs/scheduler-primary-lifecycle/research.md

@@ -0,0 +1,82 @@
+## Summary
+- **Feature**: `scheduler-primary-lifecycle`
+- **Discovery Scope**: Extension (existing controller + scheduler components; no new daemon type, no new external dependency)
+- **Key Findings**:
+  - `slowlane_submit()` (`src/controller/slowlane.c`) already exists, is tested, and is the exact primitive this feature needs: a bounded, admission-capped executor with per-key in-flight dedup that runs submitted work off the Raft drive threads (iomgr/delaymgr). No new async-execution mechanism needs to be built.
+  - `XCE_CONTROLLER_HOOK_SCHEDULER` / `HOOK_DETAIL_SCHEDULER_JOIN` / `HOOK_DETAIL_SCHEDULER_DEAD` (`src/include/xce/cluster_def.h`) are declared but never fired or consumed anywhere in `src/controller/*.c` — confirmed by exhaustive grep. They are dead enum values, not existing plumbing to build on.
+  - The controller already tracks scheduler connection lifecycle via `on_scheduler_join()` / `handle_scheduler_leave()` (`src/controller/sched.c`) and a single `cluster_t.sched.ep` slot — this is reusable bookkeeping for "is my child scheduler currently joined," but it has no concept today of *owning* the scheduler process itself.
+  - The scheduler's current rejoin logic (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin`, `src/scheduler/src/scheduler.c`) and its use of the general-purpose, candidate-rotating client library (`xcecli_join`, `XCECLI_FAILOVER_CALL`) both become dead weight under the new model and should be removed, not left dormant — the requirements (3.4) explicitly forbid the scheduler from ever trying another controller.
+  - `ha_dispatch_gate()` (`src/controller/ha.c:4573`) already rejects scheduler write opcodes against a non-primary controller (`CTRL_STANDBY`'s allow-list excludes job dispatch / pend-reason). Confirmed by reading the gate's full switch statement. This means the safety argument for this feature is already true today — the feature is solely about *availability*, not correctness.
+
+## Research Log
+
+### Where does controller promotion actually complete?
+- **Context**: Need the concrete integration point for "start a scheduler on promotion."
+- **Sources Consulted**: `src/controller/ha.c`, `raft_cb_become_leader` (line 1615), read via codegraph.
+- **Findings**: `raft_cb_become_leader` is long and gated: election-timeout cancellation, safe-committed-LSN computation, first-boot `cluster_init_phase2`, WAL truncation-floor handling, then a sequence of promotion-refusal gates (recovery-truncated, apply-divergent, self-restart-pending, apply-resource-fenced, config-serve-fenced) that each `return` without completing promotion, staying alive per the project's never-self-terminate invariant. Only after every gate passes does the function proceed to authoritative promotion (role flip to `CTRL_PRIMARY`, `ha_start_primary_heartbeat`, `repl_add_standby` for reachable voters).
+- **Implications**: The scheduler-start action must be triggered only at the tail of this function, after all refusal gates have passed — never speculatively earlier. A promotion attempt that gets refused by any gate must not start a scheduler.
+
+### Where does demotion happen?
+- **Context**: Need the integration point for "signal my scheduler to exit."
+- **Sources Consulted**: `src/controller/ha.c`, `raft_cb_become_follower` (line 2120), read via codegraph.
+- **Findings**: `raft_cb_become_follower` is the Raft library's `on_become_follower` callback, invoked from multiple paths (losing an election, observing a higher term, an append-entries request from a legitimate newer leader). This is the single funnel for every graceful role-loss case named in Requirement 2.1 (voluntary step-down and higher-term discovery both route through it).
+- **Implications**: One hook site covers all of Requirement 2's graceful-demotion triggers; no separate handling needed per trigger type.
+
+### Is there an existing async-safe way to do blocking work from the HA path?
+- **Context**: `.kiro/steering/ha-async-io.md`'s thread-role contract forbids blocking work on iomgr/delaymgr, and spawning/signaling/monitoring a child process is unavoidably blocking-ish (fork/exec, waitpid, kill).
+- **Sources Consulted**: `src/controller/slowlane.c`, `src/controller/include/slowlane.h`, existing callers in `ha_convergence.c`, `config_repl.c`, `persist.c`.
+- **Findings**: `slowlane_submit(sl, key, fn, arg)` already exists for exactly this purpose ("long blocking I/O off the Raft drive threads"), is non-blocking to the caller, runs `fn` on a dedicated worker thread with no lock held, and provides per-key in-flight dedup (`-EEXIST` on a duplicate submit of the same key while one is still running/queued). A `test_slowlane.c` suite already covers non-blocking submission, dedup, and admission-cap behavior.
+- **Implications**: Build vs. Adopt — **adopt**. Scheduler start/stop/monitor actions are ordinary `slowlane_submit()` work items. The dedup key (scoped to "this controller's scheduler-management action") mechanically satisfies Requirement 1.2/1.3 ("at most one Scheduler Process," "shall not start a second") as a side effect of reusing an already-tested primitive, with no new synchronization to design or verify.
+
+### Is there an existing pattern for one XCE daemon supervising a child process?
+- **Context**: Wanted to know if `process_worker_manager.c`/`uv_ipc.c` (the worker daemon's job-process supervision) is directly reusable.
+- **Sources Consulted**: `src/worker/src/process_worker_manager.c`, `src/worker/src/uv_ipc.c` (libuv-based named-pipe IPC with a heartbeat callback between the worker daemon and its forked job-executor children).
+- **Findings**: The pattern (fork a child, communicate over a local IPC channel, monitor a heartbeat) is directly analogous to what this feature needs between a controller and its scheduler child. The concrete code, however, is tightly coupled to job-execution concerns (cgroups, dbus, job exit-status plumbing) and to libuv, a dependency the controller does not currently carry.
+- **Implications**: Build vs. Adopt — **adopt the pattern, not the code**. The controller gets its own small, purpose-built scheduler-process-manager component (fork/exec + reap + a liveness signal), following the same shape (parent owns child lifecycle, detects child exit, watches liveness) without pulling in libuv or the worker's job-execution-specific machinery. Simplification lens: a shared abstraction with the worker's job-process manager was considered and rejected — the two "child processes" (a job executable vs. a companion XCE daemon) differ enough (job exit codes/signals/cgroups vs. a single well-known daemon binary with its own wire protocol) that forcing a shared interface would add an abstraction layer with only one real implementation on each side.
+
+### How does the scheduler detect its own controller is dead or no longer primary?
+- **Context**: Requirement 3 needs an active detection mechanism, not just reliance on OS-level parent-death propagation (discussed and explicitly not relied upon as the primary mechanism, since it doesn't cover "controller process dies while the host lives").
+- **Sources Consulted**: `src/lib/client.c` (`xcecli_join`, `XCECLI_FAILOVER_CALL`), existing `ctrlstatus`/`OPCode_ctrlstatus` read-only status query (confirmed answerable in every controller role per `ha_dispatch_gate`), `src/controller/sched.c`.
+- **Findings**: The scheduler already opens a persistent, Cap'n-Proto-framed TCP connection to a controller and already has a role-agnostic status query (`ctrlstatus`) available in every role. Connection loss (RDHUP) is already detected by the existing transport layer everywhere else in this codebase (workers, clients) — nothing new needs to be built to notice "my controller is unreachable." What is missing is (a) pinning the scheduler to exactly one controller address with no candidate list/rotation, and (b) a periodic check of that one controller's reported role, so a controller that demotes without the connection ever dropping (the common case — TCP stays up across a graceful step-down) is still detected.
+- **Implications**: Build vs. Adopt — adopt the existing wire protocol and status query; build only the pin-to-one-controller startup mode and the periodic self-check loop (a much smaller version of the worker's `checking_connection()` shape, minus any rejoin/rotation behavior). The explicit graceful-exit signal (Requirement 2) rides the same existing connection as a new, small message case in the scheduler's existing request handler (`scheduler_req_handler`, `src/scheduler/src/handler.c`) rather than a separate channel.
+
+## Architecture Pattern Evaluation
+
+| Option | Description | Strengths | Risks / Limitations | Notes |
+|--------|-------------|-----------|---------------------|-------|
+| Controller-owned child process (selected) | Primary controller forks/execs a scheduler bound to itself; scheduler self-exits on losing that specific controller | Reuses controller's already-hardened Raft election for "who's authoritative"; host death handled for free; no new election/lease protocol | Controller gains a new, bounded process-supervision responsibility; scheduler loses independent host placement | Matches decision reached in requirements discussion |
+| Independent scheduler + periodic reconnect (rejected) | Keep scheduler as today, add an unconditional periodic `xcecli_join()` retry (worker-style `checking_connection()`) | Smaller code change; preserves independent deployment | Does not solve scheduler-host-death or wedged-scheduler detection; still needs a second mechanism for those | Considered and superseded once host-death handling became an explicit requirement |
+| Standby schedulers + controller-granted lease (rejected) | Multiple scheduler processes compete for a renewable lease from the primary controller | Handles wedge/host-death without host co-location; scheduler keeps independent placement | A second, parallel authority/liveness protocol to design, build, and prove correct, duplicating the controller's own Raft election in spirit | Rejected in favor of piggybacking on existing Raft election (simplification lens) |
+| General per-host role-manager agent (deferred) | A new component per host starts/stops worker/controller/scheduler/apiserver by declared role and cluster state | Generalizes beyond just scheduler | New always-on process whose own supervision is unsolved; would need to watch controller election state itself, adding latency and an indirection layer | Explicitly deferred in requirements as a separate, independently-motivated initiative, not scope for this spec |
+
+## Design Decisions
+
+### Decision: Use `slowlane_submit()` for all scheduler process management
+- **Context**: Starting, monitoring, and signaling a child process from `raft_cb_become_leader`/`raft_cb_become_follower` must not block the Raft drive threads.
+- **Alternatives Considered**:
+  1. Spawn a brand-new dedicated pthread per promotion — works, but reinvents admission control and dedup that already exist.
+  2. Do the fork/exec inline in the Raft callback — violates the thread-role contract outright (fork/exec/waitpid are blocking).
+- **Selected Approach**: Submit scheduler-management actions (start on promotion, signal-and-reap on demotion, restart on detected wedge) to the existing SlowLane executor with a per-controller-instance dedup key.
+- **Rationale**: Already built, already tested, already the established pattern for exactly this class of problem in this codebase (`raft-drive-async` spec).
+- **Trade-offs**: None material — this is a pure reuse with no new abstraction.
+- **Follow-up**: Confirm the SlowLane instance's `max_inflight`/`n_workers` sizing (set at `slowlane_init()` call sites in `cluster.c`) has headroom for scheduler-management tasks alongside its existing consumers (peer connect, snapshot transfer, config-push); this is a sizing check, not a new design.
+
+### Decision: Scheduler pins to exactly one controller address at startup, no candidate list
+- **Context**: Requirement 3.4 forbids the scheduler from ever joining or continuing to operate against any controller other than the one that started it.
+- **Alternatives Considered**:
+  1. Keep the existing `--peers` full-candidate-list startup mode and just suppress rotation in the failover-call path — leaves a footgun where the shape of the config still implies "any of these are fine."
+  2. Give the scheduler a single-controller startup mode entirely, with no candidate list construct at all.
+- **Selected Approach**: Option 2. The controller passes its own address (not the full peer list) when it starts its scheduler.
+- **Rationale**: Makes the "one controller owns one scheduler" invariant structurally true from the scheduler's own configuration, not just an unenforced behavioral convention.
+- **Trade-offs**: The scheduler's startup contract changes (breaking change to its CLI/config surface); existing deployment tooling that passes a full peer list to `xce_scheduler` must be updated (see Requirement 6 / Integration & Migration Notes in design.md).
+- **Follow-up**: Confirm during implementation whether any other in-tree consumer starts `xce_scheduler` with a multi-candidate list for a reason unrelated to this feature (e.g. a test fixture) — such usages need explicit review, not silent breakage.
+
+## Risks & Mitigations
+- Risk: A rapid succession of promotions/demotions (chaotic failover) could thrash scheduler process spawn/kill cycles. — Mitigation: the SlowLane dedup key naturally coalesces overlapping start requests for the same controller instance into at most one in-flight action; a demotion always signals-and-reaps before that controller instance could ever legitimately be promoted again (a demoted node must win a fresh election first).
+- Risk: The wedged-scheduler health check (Requirement 5) needs a "making progress" signal, not just an open connection — inventing a bespoke internal probe risks becoming its own source of false positives. — Mitigation: reuse the existing job/queue/config push traffic the controller already sends its scheduler as the progress signal (e.g., confirmation that a recent push was acted on), rather than adding a new synthetic heartbeat payload; finalize the exact signal during implementation against real traffic patterns.
+- Risk: Existing tooling and specs (xce-deploy-ha, xce-test-ha's soak harness, any other spec assuming an independent, reconnecting scheduler) silently drift out of sync with the new model. — Mitigation: explicit File Structure Plan entries in design.md for every piece of tooling identified as depending on today's model (see design.md Integration & Migration Notes).
+
+## References
+- `.wolf/buglog.json` bug-1760 — original live repro motivating this spec.
+- `.kiro/specs/raft-drive-async/design.md` — original SlowLane design (cited verbatim in `slowlane.h`'s own header comment).
+- `.kiro/steering/ha-async-io.md` — thread-role contract and never-self-terminate invariant this design must respect.

.kiro/specs/scheduler-primary-lifecycle/spec.json

@@ -0,0 +1,26 @@
+{
+  "feature_name": "scheduler-primary-lifecycle",
+  "created_at": "2026-09-02T04:39:22Z",
+  "updated_at": "2026-09-02T07:25:03Z",
+  "language": "en",
+  "phase": "superseded",
+  "superseded_by": "xce-node-manager",
+  "superseded_note": "2026-09-02: after further discussion, the user decided the controller should not directly own spawning/killing its own scheduler process. Instead, a new general per-node xce-node-manager component (one per host, role-assigned: manager/worker/apiserver/reverse-proxy) will own starting XCE components for its node's role, including starting/stopping the scheduler on a manager node in reaction to that node's controller reporting itself primary vs. standby over a local heartbeat (Unix domain socket, client logic shared via libxce). This supersedes this spec's core mechanism (controller-internal posix_spawn of its own scheduler via scheduler_proc.c/ha.c) though the underlying requirements this spec captured -- bug-1760's reactive-rejoin deadlock, wedge detection, graceful shutdown, controller-resilience invariants -- carry forward into xce-node-manager's requirements. This spec's approved design.md and its review findings (notably the fork()-vs-posix_spawn mutex-inheritance-deadlock analysis) remain a useful reference and are kept in place, not deleted. Do not implement this spec's tasks.md.",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true,
+      "validation_note": "kiro-validate-design 2026-09-02: pass 1 found 1 critical issue -- Technology Stack's raw fork()+execve() justification for scheduler_proc.c incorrectly claimed no mutex-inheritance-deadlock risk because the call happens on a dedicated SlowLane worker thread; this doesn't hold, since the hazard comes from OTHER threads in the same multithreaded controller process holding a lock at fork time (glibc malloc, XCE mylock_t), directly re-triggering the risk class product.md's own locked decision (\"no fork() for snapshots\") already rejected. Fixed by switching to posix_spawn() throughout design.md (Technology Stack, both sequence diagrams, File Structure Plan, Components summary, architecture diagram, Error Handling) with the corrected reasoning stated explicitly. Pass 2 (fresh re-read + verified posix_spawn's argv/envp compatibility with execve, and confirmed via grep that the controller has zero existing SIGCHLD/signal handling to conflict with the new reaping logic) found no additional issues. GO."
+    },
+    "tasks": {
+      "generated": true,
+      "approved": false,
+      "validation_note": "kiro-spec-tasks 2026-09-02: 6 major tasks / 15 sub-tasks (plus 2 collapsed single-item major tasks) drafted covering all 25 requirement IDs (1.1-6.3) and every design.md component (SchedulerProcessManager, SchedulerConnectionBookkeeping, SchedOwnerWatch, the graceful-exit wire message). Independent task-graph sanity review (fresh subagent, pass 1): NEEDS_FIXES -- (1) task 1.2's (P) marker missing a _Boundary:_ tag, (2) task 2.4's _Depends:_ omitted 2.2 despite needing it, (3) Requirement 1.5's heartbeat-driven start-retry had no task wiring an actual periodic call site, (4) the old-rejoin-logic removal task was sequenced in Core, before ha.c wiring and tooling reconciliation, contradicting design.md's own Migration Strategy phase order (removal must be last), (5) design.md's own Testing Strategy lists a dedicated \"job survives scheduler killed mid-decision\" integration test with no owning task, (6) the tooling-reconciliation task covered only the known run-ha-soak.sh case, not design.md's broader \"flag any other dependent spec\" commitment. All 6 fixed: boundary tag added; dependency completed; new task 4.3 combines the heartbeat-driven start-retry and wedge-check wiring; the removal task became standalone Task 6 (last in the file, depending on 3, 4.1, 4.2, 5.7, 5.8, matching the Migration Strategy phases); new task 5.3 added for the job-durability scenario; task 5.8 broadened to sweep other specs/tooling. Sanity review pass 2 (fresh subagent): PASS -- all 6 fixes verified correct, plus fresh independent checks (full requirement-ID coverage, all _Depends:_ references resolve to real tasks including the two collapsed single-item major tasks, no (P)-marker boundary conflicts, Task 6's dependency list matches the Migration Strategy phases). Awaiting user approval before implementation."
+    }
+  },
+  "ready_for_implementation": false
+}

.kiro/specs/scheduler-primary-lifecycle/tasks.md

@@ -0,0 +1,129 @@
+# Implementation Plan
+
+- [ ] 1. Foundation: wire protocol schema and scheduler startup contract
+- [ ] 1.1 Define the controller-to-scheduler graceful-exit wire message
+  - Add a new message the controller can send an already-joined scheduler over their existing connection, carrying no payload beyond identifying it as a graceful-exit request
+  - Observable completion: the schema builds cleanly and the new message is visible in the generated bindings, unused by any caller yet
+  - _Requirements: 2.1, 2.2_
+
+- [ ] 1.2 (P) Give the scheduler a single-controller startup mode
+  - Replace the scheduler's current multi-candidate `--peers`-style startup configuration with a mode that accepts exactly one controller address
+  - Observable completion: the scheduler binary accepts the new single-address startup argument; the existing multi-candidate argument is no longer accepted for this role
+  - _Requirements: 3.4_
+  - _Boundary: scheduler main.c_
+
+- [ ] 2. Core: controller-side scheduler process manager
+- [ ] 2.1 Start a scheduler process safely off the Raft drive threads
+  - Implement the start action (spawn via `posix_spawn`, passing the controller's own address using task 1.2's new startup mode) submitted through the existing SlowLane executor with a dedup key, never invoked inline from a Raft callback
+  - Observable completion: requesting a start when one is already in flight or running is deduped (no second process spawned); a start whose spawn fails is logged and does not crash or block the caller
+  - _Requirements: 1.1, 1.2, 1.3, 1.5, 4.1, 4.3_
+  - _Boundary: scheduler_proc.c_
+  - _Depends: 1.2_
+
+- [ ] 2.2 Stop a scheduler process gracefully, with a bounded force-terminate fallback
+  - Implement the stop action: send the graceful-exit wire message from task 1.1 over the existing connection, wait up to a bounded timeout for the process to exit, then force-terminate it
+  - Observable completion: a scheduler that exits on its own within the timeout is not force-terminated; one that does not is force-terminated exactly once after the timeout elapses; the caller never blocks waiting for this to finish
+  - _Requirements: 2.1, 2.3, 2.4, 4.2, 4.3_
+  - _Boundary: scheduler_proc.c_
+  - _Depends: 1.1_
+
+- [ ] 2.3 (P) Track whether the owned scheduler is making real scheduling progress
+  - Extend the controller's existing scheduler join/leave bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it
+  - Observable completion: the timestamp advances on real scheduler activity, using only existing traffic — no new synthetic ping is introduced
+  - _Requirements: 5.1, 5.2_
+  - _Boundary: sched.c_
+
+- [ ] 2.4 Detect a wedged scheduler and replace it
+  - Implement a liveness check that reads task 2.3's progress signal; if the scheduler is connected but has made no progress past a configured threshold, submit a restart action (stop via task 2.2, then start via task 2.1) through the same SlowLane path
+  - Observable completion: a connected scheduler with no progress past the threshold is terminated and replaced exactly once per stale period, with an operator-visible log; a scheduler that is merely disconnected (already handled by existing join/leave bookkeeping) is not double-counted as a wedge
+  - _Requirements: 5.3, 5.4, 4.1, 4.3_
+  - _Boundary: scheduler_proc.c_
+  - _Depends: 2.1, 2.2, 2.3_
+
+- [ ] 3. (P) Core: pin the scheduler to its one controller and watch it
+  - Using task 1.2's single-address startup mode, connect only to that one controller; periodically query its reachability and role over that same connection
+  - On an unreachable controller, a reported non-primary role, or receipt of task 1.1's graceful-exit message via the existing request handler, exit the process
+  - Observable completion: the scheduler process exits within one check interval of its controller becoming unreachable or non-primary, or immediately on receiving the graceful-exit message; it never attempts any address other than the one it started with
+  - _Requirements: 2.2, 3.1, 3.2, 3.3, 3.4, 3.5_
+  - _Boundary: sched_owner.c, scheduler request handler_
+  - _Depends: 1.1, 1.2_
+
+- [ ] 4. Integration: wire promotion, demotion, and liveness/retry checking into controller HA role transitions
+- [ ] 4.1 Start the scheduler on promotion
+  - Call task 2.1's start action at the point a controller completes promotion, after every existing promotion-refusal gate has already passed
+  - Observable completion: a real promotion results in exactly one scheduler process being started, joining, and receiving job/queue/config state; a refused promotion attempt (any existing gate) never starts one
+  - _Requirements: 1.1, 1.4, 4.4_
+  - _Boundary: ha.c_
+  - _Depends: 2.1_
+
+- [ ] 4.2 Stop the scheduler on graceful demotion
+  - Call task 2.2's stop action at the point a controller begins a graceful role transition away from primary, without waiting for it to complete
+  - Observable completion: a real graceful demotion's own role transition completes without waiting on the scheduler's exit; the scheduler process is gone within the bounded timeout
+  - _Requirements: 2.1, 2.3_
+  - _Boundary: ha.c_
+  - _Depends: 2.2_
+
+- [ ] 4.3 Wire periodic scheduler presence and liveness checking into the controller's existing cadence
+  - While primary with no scheduler currently owned (e.g. a prior start attempt failed), retry task 2.1's start action on the controller's existing heartbeat-driven cadence — the mechanism Requirement 1.5's retry depends on, with no new timer introduced
+  - While primary with a scheduler owned, run task 2.4's liveness check on that same cadence
+  - Observable completion: a scheduler that failed to start is retried on the next cadence tick without a new timer thread; a wedged scheduler is detected and replaced within one cadence tick of crossing the threshold
+  - _Requirements: 1.5, 5.1, 5.2, 5.3, 5.4_
+  - _Boundary: ha.c_
+  - _Depends: 2.1, 2.4_
+
+- [ ] 5. Validation: end-to-end coverage and deployment tooling reconciliation
+- [ ] 5.1 Full promotion round trip against a real cluster
+  - Exercise a real controller cluster promotion end to end: a scheduler process appears, joins, and receives job/queue/config state with no manual step
+  - Observable completion: the round trip produces a working, job-dispatching scheduler in one live test run
+  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
+  - _Depends: 4.1_
+
+- [ ] 5.2 Graceful demotion round trip
+  - Force a graceful step-down against a real cluster and confirm the prior primary's scheduler exits within the bounded timeout, verified by process absence
+  - Observable completion: the demoted controller's own role transition completes promptly and the scheduler process is confirmed gone afterward
+  - _Requirements: 2.1, 2.2, 2.3_
+  - _Depends: 4.2, 3_
+
+- [ ] 5.3 A job survives scheduler termination mid-decision
+  - Dispatch a job to a running scheduler, kill the scheduler process before its dispatch decision is durably recorded, and confirm the job remains available (not lost, not left inconsistent) for a subsequently-started scheduler to consider
+  - Observable completion: the job is present and correctly schedulable after the replacement scheduler starts, with no trace of the killed scheduler's incomplete decision
+  - _Requirements: 2.4_
+  - _Depends: 2.2_
+
+- [ ] 5.4 Bug-1760 original repro: hard-killed primary recovers automatically
+  - Reproduce this feature's own motivating scenario: hard-kill the primary controller and confirm its orphaned scheduler self-exits, the newly-elected primary starts its own fresh scheduler, and a job submitted after the failover dispatches successfully
+  - Observable completion: the exact bug-1760 symptom (a job stuck PEND after a real controller failover) does not reproduce
+  - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_
+  - _Depends: 3, 4.1_
+
+- [ ] 5.5 Wedged scheduler is detected and recovered live
+  - Artificially wedge a running scheduler's progress signal while its connection stays open, against a real cluster, and confirm it is detected and replaced
+  - Observable completion: the wedged scheduler is terminated and a working replacement is dispatching jobs again within the configured threshold
+  - _Requirements: 5.1, 5.2, 5.3, 5.4_
+  - _Depends: 4.3_
+
+- [ ] 5.6 Rapid promotion and demotion does not thrash scheduler processes
+  - Drive a rapid sequence of promotions and demotions against a real cluster and confirm no more than one scheduler spawn/kill cycle occurs per genuine promotion
+  - Observable completion: no orphaned or duplicate scheduler processes remain after the sequence settles
+  - _Requirements: 4.1, 4.2, 4.3, 4.4_
+  - _Depends: 5.1, 5.2_
+
+- [ ] 5.7 Update xce-deploy-ha for controller-owned scheduler startup
+  - Update the deployment skill's documentation and generated scripts so they no longer describe or provide a manual, independent scheduler-start step
+  - Observable completion: following the updated deployment steps produces a working scheduler with no manual scheduler-start action, matching task 5.1's proof
+  - _Requirements: 6.1_
+  - _Depends: 5.1, 5.2_
+
+- [ ] 5.8 Update the HA soak harness and sweep for other dependent tooling
+  - Remove the soak harness's own manual scheduler start/restart logic (added as this feature's own motivating bug's workaround) and replace it with a check that a working scheduler is present after each round, with no harness-side management action
+  - Also search other specs and test tooling in the repository for the same independently-started, reconnecting-scheduler assumption and flag or reconcile any found, not just the soak harness
+  - Observable completion: the soak harness's sequential and chaotic modes pass unmodified in every other respect, with the manual scheduler-management code paths gone; any other dependent spec/tool found is either reconciled or explicitly flagged with a stated reason it is not
+  - _Requirements: 6.2_
+  - _Depends: 5.4_
+
+- [ ] 6. Cleanup: remove the scheduler's superseded cross-controller rejoin logic
+  - Remove the reactive rejoin mechanism and the general multi-candidate failover-rotation path for the scheduler's own role, now that task 3 fully replaces its purpose and tasks 4-5 have proven and reconciled the new model end to end — matching design.md's own Migration Strategy ordering, where this removal is deliberately the last, hardest-to-reverse phase
+  - Observable completion: the removed functions no longer exist; the existing scheduler test suite passes unmodified with no regression; the scheduler's own job/queue/config handling behavior is unchanged
+  - _Requirements: 3.4, 6.3_
+  - _Boundary: scheduler.c_
+  - _Depends: 3, 4.1, 4.2, 5.7, 5.8_

.kiro/specs/scheduler-sim-diagnostics/requirements.md

@@ -0,0 +1,53 @@
+# Requirements Document
+
+## Introduction
+Operators investigating scheduler behavior in the `--interactive` simulation REPL (shipped by `scheduler-core`) can already inspect job, queue, worker, and resource state through the REPL's `info` command family, but cannot see what resource limits are configured or how much of each is currently accounted for. This document covers the first capability of the open-ended `scheduler-sim-diagnostics` spec: giving the REPL visibility into limits — both a limit's configured setting and the live, accounted usage tracked against it — so an operator can answer "what limit is in effect, and how close is it to being hit" without reading cluster configuration files or adding ad hoc instrumentation.
+
+## Boundary Context
+- **In scope**: REPL-visible display of a limit's configured setting (the resource it constrains, its configured value or percentage, and which scope dimensions it varies by) and its live accounted usage (per distinct scope instance), in simulation mode only.
+- **Out of scope**: any command that mutates scheduler state; live, controller-connected (non-simulation) inspection; explaining *why* a specific job did or did not trigger a limit (decision-reasoning trace — expected to be added later as a separate requirement of this same open-ended spec, not covered here); any change to scheduling, allocation, or limit-matching behavior.
+- **Adjacent expectations**: this capability extends the REPL's existing `info` command family (shipped by `scheduler-core`, implementation-complete) without altering any existing subcommand's behavior or output; it must not perturb the guarantee that a simulation replay reproduces identical scheduling decisions and identical accounting state regardless of what inspection commands are issued during it.
+
+## Requirements
+
+### Requirement 1: Limit Configuration Display
+**Objective:** As an operator investigating scheduler behavior, I want to see each limit's configured setting, so that I can confirm what constraint is actually in effect without reading cluster configuration files.
+
+#### Acceptance Criteria
+1. When an operator requests information for a specific limit by name, the REPL shall display that limit's name, the resource it constrains, its configured value, whether that value is an absolute amount or a percentage, and which scope dimensions (user, queue, project, host) it varies by.
+2. When an operator requests information for all limits, the REPL shall display the configuration described in Acceptance Criterion 1 for every limit currently configured in the simulation.
+3. If an operator requests information for a limit name that is not configured, then the REPL shall report that no such limit exists and shall not display partial or fabricated information.
+4. Where a limit's configured value is a percentage, the REPL shall also display the current absolute amount that percentage resolves to, since that amount can change as the underlying resource's value changes.
+5. Where a limit's scope is restricted to a specific set of users, queues, projects, or hosts rather than applying to all of them, the REPL shall display that restricted membership.
+
+### Requirement 2: Live Accounted Usage Display
+**Objective:** As an operator watching a simulation replay, I want to see how much of each limit is currently accounted for, so that I can tell whether a limit is close to being exhausted or was the reason a job was denied.
+
+#### Acceptance Criteria
+1. When an operator requests information for a limit, the REPL shall display, for each of that limit's currently active instances, the amount currently accounted against it alongside its configured capacity.
+2. While a limit has not yet been evaluated against any job, the REPL shall report that it has no active usage instance yet, rather than displaying a fabricated zero-usage instance.
+3. The REPL shall display accounted usage reflecting the simulation's state at the moment the command is issued.
+
+### Requirement 3: Instance Scoping and Enumeration
+**Objective:** As an operator, I want limits that vary by user, queue, project, or host to show every distinct instance currently in effect, so that I don't mistake one instance's usage for the limit's usage as a whole.
+
+#### Acceptance Criteria
+1. When a limit varies by more than one scope dimension at once (for example per-user and per-queue together, or per-host and per-user together), the REPL shall display each distinct combination of scoped values as its own instance with its own accounted usage.
+2. When a limit varies by host, the REPL shall break out accounted usage separately for each host on which that limit has been evaluated, even when the limit also varies by another scope dimension at the same time.
+3. When a limit does not vary by any scope dimension, the REPL shall display a single instance representing the entire simulation.
+4. The REPL shall label each displayed instance with the scope values it represents, so an operator can tell instances apart.
+
+### Requirement 4: Command Invocation
+**Objective:** As an operator, I want a predictable way to ask for limit information, consistent with how existing REPL info commands work, so that this new capability fits how I already work.
+
+#### Acceptance Criteria
+1. The REPL shall provide a way to request limit information for a single named limit or for all configured limits, consistent with how existing `info` subcommands accept a name or "all".
+2. The REPL shall accept a request for limit information at any point it accepts other `info` subcommands.
+
+### Requirement 5: No Interference with Scheduling or Determinism
+**Objective:** As an operator relying on simulation replay to faithfully reproduce recorded production behavior, I want limit inspection to be strictly read-only, so that asking for limit information never changes what the simulation does.
+
+#### Acceptance Criteria
+1. The REPL shall not alter any limit's configuration, usage accounting, or any scheduling decision as a result of displaying limit information.
+2. The REPL shall produce identical scheduling decisions and identical final accounting state for a replay whether or not limit information was requested during that replay.
+3. The REPL shall preserve all existing `info` subcommands' behavior and output unchanged.

.kiro/specs/scheduler-sim-diagnostics/spec.json

@@ -0,0 +1,24 @@
+{
+  "feature_name": "scheduler-sim-diagnostics",
+  "created_at": "2026-08-16T11:02:06Z",
+  "updated_at": "2026-09-01T21:15:00Z",
+  "language": "en",
+  "phase": "requirements-generated",
+  "description": "Open-ended simulation-mode REPL diagnostics spec. First capability: limit inspection (info limit <name|all>) showing configured setting (scope, value/percent) and live accounted usage per instance. Future capability (seed capability 2, not yet in requirements.md): decision-reasoning trace for pend/dispatch outcomes.",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": false
+    },
+    "design": {
+      "generated": false,
+      "approved": false
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false,
+  "blocked_on": []
+}

.kiro/specs/scheduler-sim-reconcile-terminal-guard/design.md

@@ -0,0 +1,139 @@
+# Design Document
+
+## Overview
+`sim_reconcile_unaccounted_jobs()` (`sim_loop.c:664`) triggers reconciliation for any job with `job->accounts == NULL` (line 690), deliberately state-independent since `scheduler-sim-eager-reconciliation` widened it to catch a PEND job's early-recorded DISPATCH before its RUN confirmation replays. `xce_core_push_events()`'s existing end-of-life cleanup (`core_events.c:1127-1135` for `JOB_STATE_CHANGE`, `:1179-1186` for `JOB_UPDATE`) already frees `job->allocs`/`job->accounts` on a terminal transition — which leaves `job->accounts == NULL`, the exact same trigger condition. If a later round's `decoded_session->decisions` still carries an unconsumed DISPATCH entry for that job id, reconciliation matches it and resurrects the allocation on an already-dead job, with nothing to stop it (bug-1764).
+
+This feature adds one additional exclusion to the trigger check: a job whose current state is already terminal (DONE or EXIT) is never considered for reconciliation, regardless of whether `job->accounts == NULL` or a matching DISPATCH entry exists. Every other aspect of the function — the eager-capture behavior for non-terminal jobs, the decision-reporting sequence, the array-compaction/ownership-transfer logic in the FOUND branch — is unchanged.
+
+**Users**: Operators reading `info worker`'s assigned-jobs listing or `info summary`'s running/pending counts through the interactive REPL, and any future consumer of `job->allocs`/`job->accounts`'s correctness.
+**Impact**: `sim_reconcile_unaccounted_jobs()` gains one additional condition in its existing trigger check; no other function changes.
+
+### Goals
+- A job that has reached DONE or EXIT is never re-allocated by reconciliation, no matter what the replay stream still contains for it.
+- The existing eager-capture behavior for PEND/RUN jobs is completely unaffected.
+- The fix reuses this codebase's own already-established "terminal is permanent" pattern rather than inventing a new one.
+
+### Non-Goals
+- Changing how or when a job's state actually transitions to DONE/EXIT.
+- Changing `repl_info.c`'s assigned-jobs rendering — it is a correct consumer of `job->allocs`, not the defect.
+- Changing the JOBS_PUSH resync-merge branch (`core_events.c`, bug-1734/bug-1738 lines ~822-901) — investigated during design (see Existing Architecture Analysis) and confirmed to already guard against this exact class of problem via `job_state_regresses_terminal()`; left unmodified.
+- Investigating bug-1611 (the separate, still-open question of why fresh DISPATCH decisions are rare in replay mode) — unrelated to this fix, which only prevents an EXISTING duplicate/leftover DISPATCH entry from being misapplied.
+
+## Boundary Commitments
+
+### This Spec Owns
+- The terminal-state exclusion added to `sim_reconcile_unaccounted_jobs()`'s trigger check (`sim_loop.c`).
+- A regression test in `test_sim_reconcile_unaccounted.c` proving a terminal job's allocation is never resurrected by a stray recorded DISPATCH entry.
+
+### Out of Boundary
+- `repl_info.c`'s assigned-jobs listing — unmodified; its correctness is a direct, automatic consequence of this fix, not something it implements itself.
+- `xce_core_push_events()`'s JOB_STATE_CHANGE/JOB_UPDATE terminal-transition cleanup, and its JOBS_PUSH resync-merge branch's own terminal-state guard (`core_events.c`) — both confirmed correct, neither modified.
+- `sched_commit_decision()` (`scheduler.c`) and the live-mode dispatch path — untouched; this fix is confined to the simulation driver's replay-reconciliation path.
+
+### Allowed Dependencies
+- `JOB_IS_DONE()`/`JOB_IS_EXIT()` (`xce/types.h`, existing) — the same state-testing macros `job_state_regresses_terminal()` (`core_events.c:696-701`) already uses for the analogous "terminal is permanent" rule this fix mirrors.
+- `xce_sched_job_t.state` (existing field) — read-only from this fix's perspective; never written by `sim_reconcile_unaccounted_jobs()`.
+
+### Revalidation Triggers
+- If a new job-lifecycle terminal state is ever introduced beyond DONE/EXIT, this exclusion and `job_state_regresses_terminal()` must both be re-checked together — they now encode the same invariant in two places.
+- If `sim_reconcile_unaccounted_jobs()`'s trigger condition is ever restructured (e.g. keyed on something other than `job->accounts == NULL`), re-verify the terminal-state exclusion still composes correctly with the new trigger.
+
+## Architecture
+
+### Existing Architecture Analysis
+`sim_reconcile_unaccounted_jobs()`'s current trigger (line 690) is a single `if` guarding the per-job loop body: `if ((job == NULL) || (job->accounts != NULL)) { continue; }`. This is the ONLY place in the function that decides eligibility; the FOUND branch below it (array search, ownership transfer, decision reporting) runs unconditionally once a job passes this gate. Adding the terminal-state exclusion here — rather than inside the FOUND branch, after a match is already located — is both minimal and correct: a terminal job should never even be searched for a match, since any match found for it would necessarily be the exact stale/duplicate entry this fix exists to ignore.
+
+`job_state_regresses_terminal()` (`core_events.c:696-701`) already encodes "once DONE or EXIT, a state change to anything else is rejected" for the JOB_STATE_CHANGE event-application path and the JOBS_PUSH resync-merge path. This fix does not call that function (it is a state-CHANGE guard, not an allocation guard, and lives in a different file/domain); instead it applies the equivalent `JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state)` test directly, matching the same semantic rule at the point this fix actually needs it.
+
+Investigated during design, per this feature's Non-Goals: the JOBS_PUSH resync-merge branch (`core_events.c`, the `RUN`-merge and non-`RUN`-merge sub-branches around lines 822-901) explicitly calls `job_state_regresses_terminal(existing->state, job->state)` before ever touching `existing->allocs`/`existing->accounts` — confirmed already safe for this exact class of problem. No change needed there.
+
+**Architecture Integration**:
+- Selected pattern: extend one existing function's own trigger condition; no new component, no new file.
+- Domain/feature boundaries: strictly inside the simulation driver's Recorded-Decision Reconciliation component (`scheduler-sim-decision-parity`/`scheduler-sim-eager-reconciliation`'s existing domain, `sim_loop.c`); no other domain touched.
+- Existing patterns preserved: `JOB_IS_DONE()`/`JOB_IS_EXIT()` as the standard terminal-state test; "terminal is permanent" as an invariant already established and now applied consistently at a second enforcement point.
+- New components rationale: none — a one-condition change to an existing trigger check.
+- Steering compliance: `src/scheduler/CLAUDE.md`'s 800-line-per-file cap — `sim_loop.c`'s current size should be checked before landing; this change adds at most a few lines to one existing `if` condition plus doc-comment updates, not expected to be a meaningful contributor either way.
+
+## File Structure Plan
+
+### Modified Files
+- `src/scheduler/src/sim_loop.c` — `sim_reconcile_unaccounted_jobs()`'s trigger check gains a terminal-state exclusion: `if ((job == NULL) || (job->accounts != NULL) || JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state)) { continue; }`. The function's own doc comment (lines 589-608) is updated to state the terminal exclusion alongside the existing state-independence rationale, so a future reader sees both rules together rather than the doc comment appearing to contradict the code.
+
+### New/Modified Test Files
+- `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` (extend, existing home for this function's tests) — a new regression test mirroring the reverted probe that originally found this bug: drive a job to `JOB_STAT_DONE` via `xce_core_push_events()` (confirming its `allocs`/`accounts` are correctly nulled by the existing cleanup), then call `sim_reconcile_unaccounted_jobs()` with a `decoded_session` carrying a DISPATCH entry for that same job id, and assert `allocs`/`accounts` remain NULL afterward. A second test confirms the existing eager-capture case (a PEND job's early DISPATCH entry) still succeeds exactly as before, guarding against a regression in the opposite direction.
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1 | Terminal job never re-allocated by reconciliation | Terminal-State Guard | `sim_reconcile_unaccounted_jobs()` | n/a |
+| 1.2 | Non-terminal eager-capture unaffected | Terminal-State Guard | `sim_reconcile_unaccounted_jobs()` | n/a |
+| 1.3 | Terminal state treated as permanent, consistent with existing invariant | Terminal-State Guard | `sim_reconcile_unaccounted_jobs()` | n/a |
+| 2.1 | Assigned-jobs listing never shows a terminal job | Out of Boundary (automatic consequence, `repl_info.c` unmodified) | n/a | n/a |
+| 2.2 | No REPL code change required | Out of Boundary | n/a | n/a |
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
+|-----------|--------------|--------|---------------|---------------------------|-----------|
+| Terminal-State Guard | Scheduler Core / sim_loop.c | Exclude a terminal job from reconciliation's trigger check | 1.1, 1.2, 1.3 | `JOB_IS_DONE()`/`JOB_IS_EXIT()` (P0) | Service |
+
+### Scheduler Core (sim_loop.c)
+
+#### Terminal-State Guard
+| Field | Detail |
+|-------|--------|
+| Intent | Prevent `sim_reconcile_unaccounted_jobs()` from populating or modifying a terminal job's allocation/accounting state |
+| Requirements | 1.1, 1.2, 1.3 |
+
+**Responsibilities & Constraints**
+- Adds exactly one condition to the existing per-job trigger check (line 690): a job whose state is `JOB_IS_DONE()` or `JOB_IS_EXIT()` is skipped before the DISPATCH-entry search runs, regardless of `job->accounts`'s value.
+- Does not alter `job->state`, `job->allocs`, or `job->accounts` for a terminal job in any way — the guard's only effect is to leave the loop body entirely, exactly like the existing `job->accounts != NULL` skip already does for a non-terminal, already-captured job.
+- Does not alter the FOUND branch's array-compaction, ownership-transfer, or decision-reporting logic for a non-terminal job — those paths are reached identically to today whenever a non-terminal job passes the (now three-part) trigger check.
+
+**Dependencies**
+- Inbound: `sched_sim_driver_run()`'s round-boundary call site (`sim_loop.c`, existing, unmodified) (P0)
+- Outbound: `JOB_IS_DONE()`, `JOB_IS_EXIT()` (`xce/types.h`, existing) (P0)
+
+**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
+
+##### Service Interface
+```c
+/* sim_reconcile_unaccounted_jobs() -- signature unchanged. Behavioral
+ * contract addition: a job whose state is JOB_IS_DONE() or JOB_IS_EXIT()
+ * is now guaranteed to leave this call with its allocs/accounts fields
+ * bit-for-bit unchanged from whatever they were on entry, regardless of
+ * decoded_session's contents. */
+static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
+                                            replay_session_t* decoded_session,
+                                            sched_record_session_ctx_t* record_ctx);
+```
+- Preconditions: unchanged from today (`sim`/`decoded_session` NULL-safe no-op; `record_ctx` may legitimately be NULL).
+- Postconditions: a terminal job's `allocs`/`accounts` are identical before and after the call; a non-terminal job's behavior is identical to today's.
+- Invariants: a job's terminal state, once reached, is never overridden by this function — mirrors `job_state_regresses_terminal()`'s existing invariant in `core_events.c`.
+
+**Implementation Notes**
+- Integration: no change to this function's callers, signature, or the decision-reporting sequence in its FOUND branch.
+- Validation: covered by the new regression test plus the existing `test_sim_reconcile_unaccounted.c` suite's pre-existing eager-capture tests, which must continue to pass unmodified.
+- Risks: none beyond the small `sim_loop.c` growth already addressed in Architecture Integration above.
+
+## Error Handling
+
+### Error Strategy
+No new failure modes: this feature adds a pure state comparison to an existing conditional — no allocation, no I/O, no new error path.
+
+### Monitoring
+No new logging — this is a silent skip, matching the function's existing silent skip for an already-captured (`job->accounts != NULL`) job; no operator-facing signal is needed since Requirement 2's observable outcome (a correct assigned-jobs listing) is itself the confirmation.
+
+## Testing Strategy
+
+### Unit Tests (extend `test_sim_reconcile_unaccounted.c`)
+- A job driven to `JOB_STAT_DONE` via `xce_core_push_events()` (confirming its `allocs`/`accounts` are nulled by the existing cleanup), then presented with a `decoded_session` carrying a DISPATCH entry for that job id: `sim_reconcile_unaccounted_jobs()` leaves `allocs`/`accounts` NULL (1.1).
+- The same scenario with `JOB_STAT_EXIT` instead of `JOB_STAT_DONE` (1.1, 1.3 — both terminal states must be covered, not just one).
+- A PEND job's early-recorded DISPATCH entry is still captured exactly as before (regression guard for 1.2 — must continue passing unmodified from the existing suite, and a new test if the existing suite doesn't already cover this exact "not yet RUN, gets captured" shape explicitly).
+
+### Integration Tests
+**Correction (post-review, task 2.2)**: this section originally read "existing suites, re-run, no new tests required," reasoning that `test_sim_e2e_original_motivation.c`/`test_repl_resource_table.c` already exercise `info worker`'s real display path closely enough. `tasks.md`'s own task 2.2 text (approved at the same time as this document) correctly went further and required a new, purpose-built test for the exact bug-1764 scenario — those existing suites exercise the general display path, not this specific stray-dispatch-after-terminal-state case. That new test now lives in `test_sim_reconcile_unaccounted.c` (`test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch`), alongside the unit tests below rather than in a separate REPL-focused file, since this file already carries every fixture primitive the scenario needs. Review's first round rejected an initial version of this test for using a mismatched host string on the stray entry (`"workerarr-stray"` instead of `"workerarr"`), which made its `info worker`/`info summary` assertions pass vacuously regardless of whether the fix existed; round 2 confirmed via an independently-reproduced mutation test (fix reverted, test's own accounting-level pre-check bypassed) that the corrected test's REPL-level assertions now genuinely, independently detect the bug.
+- `test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch` (`test_sim_reconcile_unaccounted.c`, new): a worker with one job driven to a terminal state and two jobs genuinely still RUN, all real allocations to the SAME worker (mirroring bug-1764's actual live shape — a stray dispatch entry for the SAME worker the job was originally sent to); after reconciling a stray DISPATCH entry for the terminal job's id, `info worker <host>` excludes it while still listing the two running jobs, and `info summary`'s running count is consistent with what `info worker` shows (2.1, 2.2).
+- `test_sim_e2e_original_motivation.c` / `test_repl_resource_table.c`'s `info worker` assigned-jobs tests: re-run to confirm no regression to the general display path (unrelated to bug-1764's specific scenario, but a useful sanity check).
+- Full scheduler test suite: confirm no regression anywhere else, matching the discipline already established in this session's other specs.

.kiro/specs/scheduler-sim-reconcile-terminal-guard/requirements.md

@@ -0,0 +1,26 @@
+# Requirements Document
+
+## Introduction
+The simulation driver's decision-reconciliation logic (`sim_reconcile_unaccounted_jobs()`) exists to catch a job whose DISPATCH decision was recorded in a replay round before that job's RUN confirmation has been decoded — a deliberate, state-independent widening introduced by `scheduler-sim-eager-reconciliation` so an early-recorded dispatch is never missed. That widening did not account for jobs that have since moved all the way to a terminal state (DONE or EXIT): once a job completes, its resource allocation is correctly released, but a later round's replay stream can still carry a leftover or duplicate DISPATCH entry for that same job id (for example, from an array job with retried dispatch attempts). Because reconciliation currently triggers on "no allocation recorded" alone, it cannot distinguish "not yet allocated" from "already completed and released," and resurrects the allocation on the completed job. This is directly observable through `info worker`'s assigned-jobs listing, which then shows a job as still consuming a worker's resources long after it finished — bug-1764, confirmed via a live interactive session and independently reproduced with a targeted probe. This feature closes that gap: reconciliation must never act on a job that has already reached a terminal state, while continuing, unchanged, to capture the early-dispatch case for jobs that have not yet terminated.
+
+## Boundary Context
+- **In scope**: adding a terminal-state check to the simulation driver's reconciliation logic (`sim_reconcile_unaccounted_jobs()`) so it never populates or modifies a job's allocation/accounting state once that job has reached DONE or EXIT.
+- **Out of scope**: any change to how `info worker`/`info summary` render a job's assignment (the display code is a correct, unmodified consumer of the job's actual allocation state — the defect is upstream of it); any change to how a job's state actually transitions to DONE/EXIT; any change to the replay/decision-stream format; the JOBS_PUSH resync-merge branch's own terminal-state handling in `core_events.c`, which is believed to already be correct for an analogous problem and is confirmed, not modified, by this feature unless a genuine gap is found (in which case it is reported, not silently folded into this fix).
+- **Adjacent expectations**: this feature depends on the job-state-change event handling that already frees a job's allocation on a terminal transition continuing to behave exactly as it does today — this feature does not change when or how a job's allocation is released, only what reconciliation is allowed to do to a job whose allocation is already gone because it terminated.
+
+## Requirements
+
+### Requirement 1: Terminal-State Guard on Job Reconciliation
+**Objective:** As an operator relying on the simulation driver's internal job/resource bookkeeping to reflect reality, I want reconciliation logic to never re-establish a resource allocation for a job that has already finished, so that a completed job can never be treated as still holding resources.
+
+#### Acceptance Criteria
+1. When the simulation driver's reconciliation logic considers a job whose current state is DONE or EXIT, the simulation driver shall not populate or modify that job's allocation or accounting state, regardless of whether a matching recorded dispatch entry exists in the replay stream.
+2. While a job's state is not terminal, the simulation driver's reconciliation logic shall continue to capture a matching recorded dispatch entry for that job exactly as it does today, unaffected by this feature.
+3. The simulation driver shall treat a job's terminal state as permanent for reconciliation purposes, consistent with how terminal states are already treated as permanent elsewhere in this driver's job-state handling.
+
+### Requirement 2: Assigned-Resource Display Consistency
+**Objective:** As an operator reading `info worker`'s assigned-jobs listing, I want it to never show a job that has already completed, so the listing stays consistent with the running/pending counts `info summary` reports.
+
+#### Acceptance Criteria
+1. When a job has reached a terminal state, the REPL's assigned-jobs listing for the worker that job was dispatched to shall not include that job, even if a later replay round's decoded session carries a recorded dispatch entry for that job id.
+2. The REPL shall require no code change of its own to satisfy this requirement — the listing already derives its content solely from the job's actual allocation state, and shows only what Requirement 1 keeps correct.

.kiro/specs/scheduler-sim-reconcile-terminal-guard/research.md

@@ -0,0 +1,35 @@
+# Research & Design Decisions
+
+## Summary
+- **Feature**: `scheduler-sim-reconcile-terminal-guard`
+- **Discovery Scope**: Extension (bug fix inside an existing component, `scheduler-sim-eager-reconciliation`'s Recorded-Decision Reconciliation)
+- **Key Findings**:
+  - The bug's root cause and exact fix point were already pinned down before this spec was created, via a live reproduction and a temporary, fully-reverted probe test (see `.wolf/buglog.json` bug-1764) — this spec formalizes and completes that investigation through the project's TDD process rather than re-deriving it.
+  - The analogous JOBS_PUSH resync-merge branch (`core_events.c`, bug-1734/bug-1738) was checked during design and confirmed to already guard against this exact class of problem via `job_state_regresses_terminal()` — no gap found there, so it is explicitly out of this spec's scope rather than silently expanded into.
+
+## Research Log
+
+### Does the JOBS_PUSH resync-merge branch have the same gap?
+- **Context**: bug-1764's own fix-direction note flagged this as worth checking before finalizing scope.
+- **Sources Consulted**: `core_events.c`'s `XCE_CORE_EVENT_JOBS_PUSH` case, specifically the resync-merge branch handling a duplicate jobid (~lines 822-901).
+- **Findings**: This branch explicitly calls `job_state_regresses_terminal(existing->state, job->state)` before merging a resync redelivery's state into an existing tracked job, and refuses the merge (including any `allocs`/`accounts` transfer) when the existing job is already terminal and the redelivery would regress it — the exact protection bug-1764 is missing from `sim_reconcile_unaccounted_jobs()`.
+- **Implications**: Confirmed no analogous gap exists there; design.md's Non-Goals explicitly excludes this branch from this spec's scope, with the confirmation recorded here rather than left as an open question.
+
+## Design Decisions
+
+### Decision: Guard at the trigger check, not inside the FOUND branch
+- **Context**: the terminal-state exclusion could be added either at line 690's trigger check (before searching for a match) or inside the FOUND branch (after a match is located, before acting on it).
+- **Alternatives Considered**:
+  1. Guard inside the FOUND branch, after the search — would still perform an unnecessary array scan for every terminal job, every round, for the rest of the replay.
+  2. Guard at the trigger check — **selected**. A terminal job is excluded before any search happens, matching the existing `job->accounts != NULL` skip's own placement and semantics exactly (both are "this job is not eligible for reconciliation at all" conditions, not "this job might have a match but we should reject it").
+- **Selected Approach**: `JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state)` added as an additional disjunct in the existing trigger `if`.
+- **Rationale**: Minimal, consistent with the existing code's own structure, and avoids wasted per-round work on jobs that can never legitimately match again.
+- **Trade-offs**: None material.
+- **Follow-up**: None.
+
+## Risks & Mitigations
+- Risk: a future terminal state beyond DONE/EXIT is added without updating this guard — mitigated by design.md's Revalidation Triggers explicitly calling out that this exclusion and `job_state_regresses_terminal()` now encode the same invariant in two places and must be updated together.
+
+## References
+- `.wolf/buglog.json` bug-1764 — the original root-cause investigation and reproduction this spec formalizes.
+- `.kiro/specs/scheduler-sim-eager-reconciliation/` — the spec that introduced the state-independent trigger widening this fix narrows for the terminal case only.

.kiro/specs/scheduler-sim-reconcile-terminal-guard/spec.json

@@ -0,0 +1,22 @@
+{
+  "feature_name": "scheduler-sim-reconcile-terminal-guard",
+  "created_at": "2026-09-03T06:32:45Z",
+  "updated_at": "2026-09-03T07:05:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}

.kiro/specs/scheduler-sim-reconcile-terminal-guard/tasks.md

@@ -0,0 +1,25 @@
+# Implementation Plan
+
+- [ ] 1. Core: terminal-state guard on job reconciliation
+- [x] 1.1 Write failing tests for the terminal-state guard
+  - Cover: a job driven to JOB_STAT_DONE via xce_core_push_events() (confirming allocs/accounts are nulled by the existing cleanup), then presented with a decoded_session carrying a DISPATCH entry for that job id, must leave allocs/accounts NULL after sim_reconcile_unaccounted_jobs() runs; the same scenario with JOB_STAT_EXIT instead of JOB_STAT_DONE; a PEND job's early-recorded DISPATCH entry is still captured exactly as today (explicit regression guard, not just relying on the pre-existing suite)
+  - Observable: the new test cases for DONE/EXIT fail against today's sim_reconcile_unaccounted_jobs(), which resurrects allocs/accounts on any job with accounts == NULL regardless of state
+  - _Requirements: 1.1, 1.2, 1.3_
+  - _Boundary: Terminal-State Guard_
+
+- [x] 1.2 Implement the terminal-state exclusion in sim_reconcile_unaccounted_jobs()
+  - Add JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state) as an additional exclusion in the function's existing per-job trigger check, alongside the existing job->accounts != NULL skip; update the function's own doc comment to state the terminal exclusion alongside its existing state-independence rationale so the two don't read as contradictory
+  - Observable: all tests from 1.1 pass; a terminal job's allocs/accounts are provably unchanged before and after a call that presents a matching stray DISPATCH entry for it
+  - _Requirements: 1.1, 1.2, 1.3_
+  - _Boundary: Terminal-State Guard_
+
+- [ ] 2. Validation: regression and display-consistency verification
+- [x] 2.1 Verify no regression to existing reconciliation behavior across the scheduler test suite
+  - Confirm the full scheduler test suite passes with no new failures beyond any already-documented pre-existing ones, and that the pre-existing eager-capture tests in test_sim_reconcile_unaccounted.c and the JOBS_PUSH resync-merge tests in test_core_events.c continue to pass unmodified
+  - Observable: full test suite result showing no regression, with the pre-existing eager-capture and resync-merge test names explicitly confirmed still passing
+  - _Requirements: 1.2_
+
+- [x] 2.2 Re-verify against a realistic array-job scenario that info worker's assigned-jobs listing no longer shows a completed job
+  - Using the same capture_dispatch()-driven REPL test harness already established in test_sim_e2e_original_motivation.c/test_repl_resource_table.c, construct a multi-job scenario mirroring bug-1764's live-observed shape: an array job where some elements are driven to JOB_STAT_DONE/JOB_STAT_EXIT via xce_core_push_events() (as in 1.1's tests) while others remain running, then advance the sim driver through a further round whose decoded_session carries a stray/duplicate DISPATCH entry (built the same way as 1.1's regression tests) for one of the now-terminal job ids. Issue info worker <host> and info summary through capture_dispatch() and assert the completed job is absent from the assigned-jobs listing, the still-running jobs are present, and the two commands' counts agree
+  - Observable: a captured info worker transcript showing the completed job absent from the assigned-jobs listing, with the still-running jobs present and info summary's counts consistent with it
+  - _Requirements: 2.1, 2.2_

.kiro/specs/scheduler-sim-step-verbosity/design.md

@@ -0,0 +1,146 @@
+# Design Document
+
+## Overview
+`repl_print_event_summary()` (`repl_break.c`) prints a `"[sim] t=<ns> (<ISO8601>) session=<N>"` preamble line unconditionally before every echoed event's own content line — this is its only call site, `xce_repl_on_event()`, which already gates WHICH events reach it per `scheduler-sim-repl-verbosity`'s Event Echo Gate (structural events always; routine events only in `step` mode). Because the preamble is a fixed prefix printed on every call regardless of what the previous call printed, an operator watching several consecutive events that land in the same displayed second and session sees that context restated once per event, diluting the actual event content it's meant to introduce.
+
+This feature makes `repl_print_event_summary()` preamble-aware of its own immediately-preceding output: it prints the preamble only when the rendered (second-granularity) timestamp or the displayed session number has changed since the last time a preamble was actually printed; otherwise it prints only the event's content line. No other REPL behavior changes.
+
+**Users**: Operators running `step` or `continue` (structural-event) transcripts through `xce_scheduler --sim <file> --interactive`.
+**Impact**: `repl_print_event_summary()` gains a short-circuit at its top; `xce_sched_repl_state_s` gains two small cache fields recording what was last actually printed.
+
+### Goals
+- Print the timestamp/session preamble once per distinct (rendered timestamp, session) run of consecutive echoed events, not once per event.
+- Base "distinct" on the rendered, second-granularity timestamp and displayed session number — not the underlying raw nanosecond clock value, which changes on every event and would otherwise defeat deduplication entirely.
+- Leave every echoed event's own content line, its ordering, and whether it is echoed at all completely unchanged.
+
+### Non-Goals
+- Changing the Event Echo Gate's routine/structural classification (`repl_event_is_structural()`) — which events reach `repl_print_event_summary()` at all is unaffected.
+- Changing the Round Rollup / collapsing behavior `continue` mode already uses for routine events and per-round decision summaries.
+- Changing decision output (`sim_print_decision`/`sim_print_decision_gate`), which has no preamble of its own and is untouched.
+- Reformatting the preamble's own content (timestamp precision, session-number convention) — only whether it is printed, not what it looks like when it is.
+
+## Boundary Commitments
+
+### This Spec Owns
+- The preamble-print decision inside `repl_print_event_summary()`'s single call site.
+- Two new cache fields on `xce_sched_repl_state_s` (`repl_internal.h`) recording the last preamble actually printed.
+
+### Out of Boundary
+- The Event Echo Gate's classification logic (`repl_event_is_structural()`, `repl_break.c`) — unmodified, this spec only changes what happens after an event has already been selected for echoing.
+- The Round Rollup / collapsing logic (`repl_rollup.c`) and `sim_print_decision_gate()` (`sim_loop.c`) — both unmodified.
+- Every REPL command's parsing, dispatch, or output other than this one preamble line.
+
+### Allowed Dependencies
+- `xce_sched_repl_state_s` (`repl_internal.h`, existing) — the two new cache fields live here, alongside this file's other REPL-session-scoped state.
+- `repl_format_ts_human()` (`repl_fmt.c`, existing, from `scheduler-sim-repl-verbosity`) — reused as-is to render the human-readable timestamp string used for the dedup comparison; not modified.
+
+### Revalidation Triggers
+- If `repl_format_ts_human()`'s rendered granularity or format ever changes, re-verify the dedup key still reflects what an operator actually sees as "the same displayed context."
+- If `repl_print_event_summary()` ever gains a second call site, revisit whether a single "last printed" cache still correctly models "immediately preceding echoed event" across both sites.
+
+## Architecture
+
+### Existing Architecture Analysis
+`repl_print_event_summary()` is `static` and has exactly one caller, `xce_repl_on_event()` (`repl_break.c`), itself reached only when `sim->interactive` and `sim->repl_state != NULL` — both already guaranteed by the time this function runs, so no new NULL-handling is introduced. The function currently: (1) unconditionally builds and prints the preamble, then (2) switches on `ev->type` to print the event's own content. This design touches only step (1); step (2) and the surrounding gate in `xce_repl_on_event()` are unchanged.
+
+`xce_sched_repl_state_s` already carries several REPL-session-scoped caches added by prior tasks (the Round Rollup's collapse-state fields, `units_mode`) — the two new fields here follow that same established pattern: session-lifetime state, zero-initialized by `repl_state_ensure()`'s existing `calloc`, with no additional lifecycle code required (an empty string and a `0` session number can never equal a real echoed event's rendered values, so the "no preceding echoed event" case, AC 1.2, is satisfied by zero-initialization alone — the same convention every field in this struct except `units_mode` already relies on).
+
+**Architecture Integration**:
+- Selected pattern: extend one existing function's own early logic; no new component, no new file.
+- Domain/feature boundaries: strictly inside the REPL Output domain (`repl_break.c`/`repl_internal.h`) `scheduler-sim-repl-verbosity` already established; no other domain (Round Rollup, Resource Formatter, Help/Breakpoint) is touched.
+- Existing patterns preserved: `xce_sched_repl_state_s` as the single home for REPL-session-scoped display state; `repl_format_ts_human()` as the sole human-timestamp renderer.
+- New components rationale: none — this is a localized behavior change to one existing function plus two state fields, not a new component.
+- Steering compliance: `src/scheduler/CLAUDE.md`'s 800-line-per-file cap applies to `repl_break.c`, which is already at 949 lines (an accepted residual from `scheduler-sim-repl-verbosity` task 9.2). This spec's addition is small (a handful of lines inside one existing function), but implementation must re-check the file's line count after landing it and, per this file family's established practice, empirically test extracting `repl_print_event_summary()` (and its Event Echo Gate neighbors) into a new file if the total meaningfully exceeds the residual already accepted — not a decision to make speculatively before the actual diff size is known.
+
+## File Structure Plan
+
+### Modified Files
+- `src/scheduler/src/repl_break.c` — `repl_print_event_summary()` gains a preamble-print decision at its top: render the candidate human timestamp and displayed session number, compare against `rs->last_echo_ts_human`/`rs->last_echo_session`; print the preamble and update both cache fields only when either differs (or the cache is still at its zero-initialized default), otherwise skip straight to the event's own content switch.
+- `src/scheduler/src/repl_internal.h` — `xce_sched_repl_state_s` gains `last_echo_ts_human` (fixed-size char buffer, same size as the existing `repl_format_ts_human()` output buffers elsewhere in this file family) and `last_echo_session` (`uint64_t`), both doc-commented as this feature's cache, zero-initialized by the existing `repl_state_ensure()` allocation path with no additional lifecycle code.
+
+### New/Modified Test Files
+- `src/scheduler/testsuite/test_repl_event_echo.c` (extend, existing home for Event Echo Gate behavior this feature builds directly on top of) — the new preamble-dedup tests (see Testing Strategy). This file is at 705 lines going into this spec; re-check its line count after landing these tests and split into a new `test_repl_step_verbosity.c` (following this file family's established "duplicate fixture helpers, don't share" convention — see `scheduler-sim-repl-verbosity` design.md's own Corrections for the precedent) if it approaches this project's ~1000-line split trigger, rather than growing past it.
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1 | Omit preamble when rendered timestamp+session unchanged | Event Echo Preamble Deduplicator | `repl_print_event_summary()` | n/a |
+| 1.2 | Print preamble when rendered timestamp+session changed, or no prior echo | Event Echo Preamble Deduplicator | `repl_print_event_summary()` | n/a |
+| 1.3 | Dedup key is rendered (second-granularity) timestamp + session, not raw ns | Event Echo Preamble Deduplicator | `repl_print_event_summary()` | n/a |
+| 1.4 | Event content/ordering/echo-selection unaffected | Event Echo Preamble Deduplicator | `repl_print_event_summary()` | n/a |
+| 2.1 | No event newly suppressed or added | Out of Boundary (Event Echo Gate unmodified) | n/a | n/a |
+| 2.2 | Ordering preserved | Event Echo Preamble Deduplicator | `repl_print_event_summary()` | n/a |
+| 2.3 | Every other command's behavior unchanged | Out of Boundary | n/a | n/a |
+| 2.4 | No change to scheduling decisions/state | Out of Boundary (display-only change) | n/a | n/a |
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
+|-----------|--------------|--------|---------------|---------------------------|-----------|
+| Event Echo Preamble Deduplicator | REPL / repl_break.c | Print the timestamp/session preamble once per distinct rendered context instead of once per event | 1.1-1.4, 2.2 | `repl_format_ts_human()` (P0), `xce_sched_repl_state_s` (P0) | State |
+
+### REPL Output (repl_break.c)
+
+#### Event Echo Preamble Deduplicator
+| Field | Detail |
+|-------|--------|
+| Intent | Suppress the timestamp/session preamble on an echoed event when it would exactly restate the immediately preceding echoed event's rendered context |
+| Requirements | 1.1, 1.2, 1.3, 1.4, 2.2 |
+
+**Responsibilities & Constraints**
+- Computes the candidate human-readable timestamp via the existing `repl_format_ts_human(xce_core_now(&sim->core), ...)` call already present in `repl_print_event_summary()`, and the candidate displayed session number (`rs->session_count + 1`, the existing expression already used) — no new value sources.
+- Compares the candidate pair against `rs->last_echo_ts_human`/`rs->last_echo_session`. Prints the preamble and updates both cache fields when they differ, or when the cache is still at its zero-initialized "never printed" default (empty string / session `0`, values no real echoed event can legitimately produce). Skips the preamble entirely, with no cache update, when they match (Requirement 1.1).
+- The comparison is on the RENDERED values only — the underlying raw nanosecond `xce_core_now()` return value is never part of the dedup key, since it changes on every event and two events sharing the same displayed second still have different raw values (Requirement 1.3).
+- The event's own content line (the existing `switch (ev->type)` block) is printed unconditionally regardless of the preamble decision — this component only ever suppresses the preamble, never the content (Requirement 1.4, 2.2).
+
+**Dependencies**
+- Inbound: `repl_print_event_summary()` (repl_break.c, existing) (P0)
+- Outbound: `repl_format_ts_human()` (repl_fmt.c, existing) (P0)
+
+**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x]
+
+##### State Management
+- State model: two new fields on `xce_sched_repl_state_s` — `last_echo_ts_human` (the last rendered human timestamp actually printed in a preamble) and `last_echo_session` (the last displayed session number actually printed).
+- Persistence & consistency: in-memory, scoped to one REPL session, written only by this component, read only by this component. Zero-initialized by `repl_state_ensure()`'s existing `calloc` — no dedicated default-assignment step is needed (unlike `units_mode`'s documented exception in `scheduler-sim-repl-verbosity`), because the zero defaults (empty string, session `0`) can never collide with a real echoed event's rendered values.
+- Concurrency strategy: none required — single-threaded sim driver, same as every other REPL state field.
+
+**Implementation Notes**
+- Integration: no change to `repl_print_event_summary()`'s call site or signature — purely internal to the function body.
+- Validation: covered entirely by the child-process stdout-capture technique `test_repl_event_echo.c` already establishes for Requirement 1 (Event Echo Gate) — no new test harness needed.
+- Risks: `repl_break.c`'s existing 949-line accepted residual (see Architecture Integration above) — watch, don't pre-split.
+
+## Data Models
+
+### Domain Model
+This feature adds no new domain concept — it extends the existing "REPL session-scoped display state" already modeled on `xce_sched_repl_state_s` (`repl_internal.h`), the same struct the Round Rollup and Resource Value Formatter components added fields to in `scheduler-sim-repl-verbosity`.
+
+### Logical Data Model
+
+| Field | Type | Purpose |
+|-------|------|---------|
+| `last_echo_ts_human` | `char[32]` | The human-readable timestamp last actually printed in an event's preamble (same buffer size `repl_format_ts_human()`'s other call sites already use) |
+| `last_echo_session` | `uint64_t` | The displayed session number last actually printed in an event's preamble |
+
+**Consistency & Integrity**: both fields reset together only at REPL session start, via `repl_state_ensure()`'s existing `calloc` zero-initialization — no explicit reset code path exists or is needed, since no other event in this struct's lifecycle should cause "what was last displayed" to become stale mid-session.
+
+## Error Handling
+
+### Error Strategy
+No new failure modes: this feature performs only in-memory string/integer comparisons and a `fprintf` that already exists at this call site — no allocation, no I/O beyond the pre-existing `stdout` write.
+
+### Monitoring
+No new logging — the existing `[sim] ...` stdout convention is the only output surface, and this feature only changes how often one of its existing lines appears.
+
+## Testing Strategy
+
+### Integration Tests (extend `test_repl_event_echo.c`, child-process output capture — see File Structure Plan for the split-if-needed note)
+- Two consecutive echoed events with identical rendered timestamp and session: the second event's preamble line is absent, its content line still appears (1.1, 1.4).
+- Two consecutive echoed events whose session differs (a session boundary occurs between them): both events print their own full preamble (1.2).
+- Two consecutive echoed events whose raw high-resolution timestamps differ but whose rendered (rounded) timestamp and session are identical: the second event's preamble is still omitted — the critical case distinguishing this feature's actual dedup key from a naive raw-timestamp comparison (1.3).
+- The very first echoed event in a fresh REPL session prints its preamble (no prior echo exists to compare against) (1.2).
+- A structural event echoed under `continue` mode after a stretch of suppressed routine events: its preamble prints or is suppressed based on the same rendered-context rule, exercising the dedup logic outside `step` mode too (1.1, 1.2).
+- `step` mode's full event/content ordering and count are unchanged versus this feature's baseline — regression check alongside this file's existing Event Echo Gate assertions (2.1, 2.2).
+
+### Regression (existing suites, re-run, no new tests)
+- `test_repl_execution_control.c` and `test_repl_equivalence.c`: confirm no other REPL command's output changed and the REPL/batch decision-equivalence guarantee still holds (2.3, 2.4) — the same spot-check discipline `scheduler-sim-repl-verbosity`'s task 10.1/10.2 already established for this exact concern.

.kiro/specs/scheduler-sim-step-verbosity/requirements.md

@@ -0,0 +1,29 @@
+# Requirements Document
+
+## Introduction
+xce_scheduler's `--interactive` simulation REPL echoes a timestamp/session preamble line immediately before every event it prints — in `step` mode for every echoed event, and in `continue` mode for every structural event the already-shipped `scheduler-sim-repl-verbosity` spec still echoes. In practice, many consecutive echoed events share the identical rendered (second-granularity) timestamp and session number, so the operator sees the same context line repeated over and over, burying the actual event content between near-duplicate preambles. This feature collapses that repetition: the preamble prints once per distinct (timestamp, session) context, not once per event, while leaving which events are echoed, their content, their ordering, and every other REPL behavior exactly as `scheduler-sim-repl-verbosity`'s Requirement 5 already established.
+
+## Boundary Context
+- **In scope**: the timestamp/session preamble line printed immediately before an echoed event's own content line, in every mode/context where that echoing currently occurs (step mode's per-event echo, and continue mode's structural-event echo already shipped by `scheduler-sim-repl-verbosity`).
+- **Out of scope**: which events are echoed, an event's own content line, event ordering, decision output, the Round Rollup/collapsing behavior already shipped for continue mode's routine-event suppression, and any REPL command's behavior other than this one preamble's rendering.
+- **Adjacent expectations**: this spec builds directly on `scheduler-sim-repl-verbosity`'s Requirement 5 ("preserve step mode's output outside this feature's explicit exceptions") and Requirement 1 (Event Echo Gate). It narrows that preservation guarantee to allow exactly this one, previously out-of-scope redundancy to change — it does not reopen or relax any other part of that guarantee.
+
+## Requirements
+
+### Requirement 1: Preamble Deduplication Across Consecutive Same-Context Events
+**Objective:** As an operator reading a step-by-step or continue-mode structural-event transcript, I want the timestamp/session preamble shown once per distinct context instead of before every single event, so repeated identical context doesn't bury the actual event content.
+
+#### Acceptance Criteria
+1. When the REPL echoes an event whose rendered timestamp and session number are identical to the immediately preceding echoed event's rendered timestamp and session number, the REPL shall omit the timestamp/session preamble line for that event.
+2. When the REPL echoes an event whose rendered timestamp or session number differs from the immediately preceding echoed event's, or when there is no preceding echoed event, the REPL shall print the full timestamp/session preamble line before that event.
+3. The REPL shall determine whether two consecutive echoed events share the same context using each event's rendered, second-granularity human-readable timestamp and its displayed session number — not the underlying raw high-resolution timestamp value, which may differ between two events that share the same displayed second and session.
+4. The REPL shall not alter an event's own content line, its position in the output, or whether it is echoed at all — deduplication under this requirement applies only to the preamble line.
+
+### Requirement 2: Preservation of Existing Detail, Ordering, and Behavior
+**Objective:** As an operator relying on step mode's full per-event/per-decision detail and continue mode's structural-event echo, I want this feature to change only the repeated preamble, so my existing workflows and expectations about output completeness remain reliable.
+
+#### Acceptance Criteria
+1. The REPL shall continue to echo every event that step mode currently echoes, and every structural event that continue mode currently echoes, with no event newly suppressed or newly added as a result of this feature.
+2. The REPL shall preserve the exact ordering of echoed events and their content lines.
+3. The REPL shall preserve every other REPL command's existing behavior and output unchanged.
+4. The REPL shall not alter any scheduling decision or the simulation's underlying state — this feature affects console display only.

.kiro/specs/scheduler-sim-step-verbosity/research.md

@@ -0,0 +1,37 @@
+# Research & Design Decisions
+
+## Summary
+- **Feature**: `scheduler-sim-step-verbosity`
+- **Discovery Scope**: Extension (existing REPL Output domain, `scheduler-sim-repl-verbosity`)
+- **Key Findings**:
+  - The redundant preamble is printed unconditionally by `repl_print_event_summary()` (`repl_break.c`), which has exactly one call site (`xce_repl_on_event()`) — no cross-cutting fan-out to design around.
+  - The raw nanosecond timestamp (`xce_core_now()`) is effectively unique per event (the virtual clock ticks per decoded event via `sim_advance_virtual_clock()`), so any dedup key based on the raw value would almost never actually trigger — this was confirmed directly against a real user-supplied transcript before writing requirements, and materially changed Requirement 1's acceptance criteria (see Decision below).
+  - `xce_sched_repl_state_s` already has an established, low-risk pattern for adding session-scoped display-only cache fields (the Round Rollup's collapse state, `units_mode`) that this feature reuses without new lifecycle code.
+
+## Research Log
+
+### What actually repeats between consecutive echoed events?
+- **Context**: The initial feature description assumed collapsing on "(t, session)" where `t` was the raw timestamp value shown in the preamble.
+- **Sources Consulted**: `repl_print_event_summary()` (`repl_break.c:672-682`); a real interactive-session transcript supplied by the user (`--sim`/`--interactive`, session 47) showing several consecutive events.
+- **Findings**: In the real transcript, the raw `t=<ns>` value differed on every single line (e.g. `t=1788169162308792367` then `t=1788169162308787761`), while the parenthesized human-readable ISO8601 timestamp and the `session=` number were identical across many consecutive lines.
+- **Implications**: A literal "(t, session)" grouping key as originally worded would essentially never collapse anything, since raw `t` never repeats. Requirement 1's acceptance criteria were corrected to key on the rendered, second-granularity human timestamp and the displayed session number — confirmed with the user before writing requirements.md (see Design Decision below).
+
+## Design Decisions
+
+### Decision: Dedup key is the rendered (human timestamp, session) pair, not the raw nanosecond value
+- **Context**: Requirement 1.3 needed to state precisely what "same context" means for two consecutive echoed events.
+- **Alternatives Considered**:
+  1. Raw nanosecond timestamp + session — rejected: virtually never repeats between events, so it would satisfy the letter of "dedup" while delivering none of the actual verbosity reduction the feature exists for.
+  2. Session number alone (ignore timestamp for grouping) — considered but not selected: presented to the user as an option; they chose option 3 below instead.
+  3. Rendered second-granularity human timestamp + session number — **selected**. Matches what an operator actually perceives as "the same displayed context," and matches what genuinely repeats in real output.
+- **Selected Approach**: Cache the last-printed rendered timestamp string and displayed session number on `xce_sched_repl_state_s`; compare before printing a new preamble; print and update the cache only on a change (or on the very first echoed event, naturally covered by the cache's zero-initialized default).
+- **Rationale**: Directly addresses the literal, evidence-based redundancy pattern observed in real output, without introducing a new formatting concept.
+- **Trade-offs**: None material — this is a strict readability improvement with no behavioral cost; the cache adds 40 bytes to a per-session struct.
+- **Follow-up**: None — the design's Testing Strategy already covers the "raw ns differs, rendered value doesn't" case explicitly (the case that motivated this decision).
+
+## Risks & Mitigations
+- `repl_break.c` is already at an accepted 949-line residual from `scheduler-sim-repl-verbosity` task 9.2; this feature's addition is small but non-zero — mitigated by design.md's explicit "watch, don't pre-split" note and the established empirical-test-before-splitting practice this file family already follows.
+- `test_repl_event_echo.c` (705 lines) may approach the ~1000-line split trigger once this feature's tests land — mitigated by design.md's File Structure Plan flagging a proactive split into `test_repl_step_verbosity.c` if needed, following the exact precedent already set for sibling test files in `scheduler-sim-repl-verbosity`.
+
+## References
+- `.kiro/specs/scheduler-sim-repl-verbosity/design.md` — the sibling spec this one extends; source of the file-size-cap precedent and the `xce_sched_repl_state_s` state-field convention reused here.

.kiro/specs/scheduler-sim-step-verbosity/spec.json

@@ -0,0 +1,22 @@
+{
+  "feature_name": "scheduler-sim-step-verbosity",
+  "created_at": "2026-09-03T03:00:41Z",
+  "updated_at": "2026-09-03T03:35:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}

.kiro/specs/scheduler-sim-step-verbosity/tasks.md

@@ -0,0 +1,26 @@
+# Implementation Plan
+
+- [ ] 1. Core: event echo preamble deduplication
+- [x] 1.1 Write failing tests for preamble deduplication across consecutive echoed events
+  - Cover: two consecutive echoed events with identical rendered timestamp and session omit the second's preamble; two consecutive events with a different session both print their own preamble; two consecutive events whose raw high-resolution timestamps differ but whose rendered timestamp and session are identical still dedupe; the first echoed event in a fresh REPL session always prints its preamble; a structural event echoed under continue mode after a stretch of suppressed routine events follows the same rule; every event's own content line and ordering remain present and unaffected regardless of the preamble decision
+  - Observable: the new test cases fail against today's REPL, which prints the timestamp/session preamble unconditionally before every echoed event
+  - After adding the new test cases, check `test_repl_event_echo.c`'s resulting line count against this project's ~1000-line split trigger (design.md's File Structure Plan); if approached or exceeded, split the preamble-dedup tests into a new `test_repl_step_verbosity.c` (duplicating shared fixture helpers per this file family's established convention) and register the new file's test target in `testsuite/meson.build`
+  - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2_
+  - _Boundary: Event Echo Preamble Deduplicator_
+
+- [x] 1.2 Implement the preamble deduplication cache and print-skip logic
+  - Add the last-printed rendered timestamp and session cache to the REPL session state, zero-initialized alongside the state's existing fields; update the event echo path to compare the candidate rendered timestamp and session against the cache, printing and updating it only when either has changed (or no preamble has been printed yet this session), while always still printing the event's own content line unconditionally
+  - Observable: all tests from 1.1 pass; a captured transcript shows exactly one preamble line per distinct rendered (timestamp, session) run of consecutive echoed events, with every event's own content line still present
+  - After landing the implementation change, re-check `repl_break.c`'s line count against its already-accepted 949-line residual (design.md's Architecture Integration); if it meaningfully exceeds that residual, evaluate extracting `repl_print_event_summary()` and its Event Echo Gate neighbors into a new file rather than pre-splitting speculatively
+  - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2_
+  - _Boundary: Event Echo Preamble Deduplicator_
+
+- [ ] 2. Validation: regression and real-transcript verification
+- [x] 2.1 Verify every other REPL command and mode's existing behavior remain unchanged outside the preamble's own deduplication
+  - Confirm which events are echoed, event ordering, event content, and every other REPL command's behavior and output match their pre-feature baseline exactly except for the preamble line's now-conditional printing; confirm no scheduling decision or simulation state changed as a result of this feature
+  - Observable: the existing REPL test suite's pre-existing assertions all still pass, updated only where a specific assertion pinned the old always-print-preamble shape for two events that legitimately share rendered context (each such update documented with which acceptance criterion justifies it), and a step-mode transcript comparison against the pre-feature baseline shows no textual difference outside the preamble line's presence or absence
+  - _Requirements: 2.1, 2.3, 2.4_
+
+- [x] 2.2 Run the full scheduler test suite and re-verify against a realistic multi-event transcript
+  - Observable: the full test suite passes with no regressions, and a step/continue pass over a transcript containing several consecutive same-context events shows the preamble collapsing to one line per distinct context while every event's own content line still appears in order
+  - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4_

.kiro/specs/worker-rejoin-resync/design.md

@@ -0,0 +1,501 @@
+# Design Document: worker-rejoin-resync
+
+## Overview
+
+On every worker (re)join — a fresh network reconnect after a disconnect, or a reconnect to a new primary after controller failover — this feature has the controller ask the worker directly what really happened to every job the controller still believes is running there, and reconciles the worker's answer into the job table through the existing idempotent commit-then-apply path. This closes the gap left by fire-and-forget job start/finish/fail notifications (`XCECLI_FF_CALL`): a report lost to a disconnect or failover window today leaves a job stuck in `RUN` forever, with nothing in the cluster ever correcting it (bug-1741, bug-396).
+
+**Users**: Internal controller/worker/scheduler subsystems — there is no end-user-facing surface. Cluster operators benefit indirectly: jobs self-heal out of a stuck-`RUN` state without manual intervention.
+
+**Impact**: Extends `rejoin_worker`/`apply_join_state` (`worker-join-refactor`, implemented) with a job-state reconciliation round; adds one new opcode pair to the wire protocol; adds a short-lived finished-job retention to the worker daemon (`xcecd`), which retains none today; adds two narrow new fact-recording functions to the controller's job table, reusing its existing idempotency guards for everything else; adds two new shared-config fields; adds one line to the scheduler's existing candidate-host filter; and, because Requirement 4.6 is the first path in this codebase that re-admits an already-dispatched job back to the scheduler's PEND pool, closes a previously-dormant gap (bug-1631) in `dispatch_a_job_`'s retryable-failure handling that this feature would otherwise make reachable.
+
+### Goals
+- Trigger reconciliation on every worker join where the controller holds non-terminal jobs for that host, without needing to distinguish first-join from reconnect (Requirement 1).
+- Query only the jobs in question; let the worker answer definitively for each, plus report anything running/recently-finished the controller didn't ask about (Requirements 2, 3).
+- Apply every reconciled outcome through the existing idempotent commit-then-apply path — no second mutation mechanism (Requirement 4).
+- Withhold new dispatch to a worker while its reconciliation is outstanding, and fail fast (not queue) a dispatch to a disconnected worker (Requirement 5).
+- Stay safe under repeated/interrupted rejoin (Requirement 6) and reach the scheduler through the same notification path as any ordinary job-state change (Requirement 7).
+- Work regardless of how many jobs a host is running, via a bounded/paginated exchange (Requirement 8).
+
+### Non-Goals
+- apiserver/scheduler cluster/queue/job-*list* resync on reconnect (`component-rejoin-resync`, a disjoint sibling spec — different data shape, not merged here).
+- Any change to the primary-side pending-facts queue (the commit engine) itself — this feature is an additional caller into it, never a modification of it.
+- Delta-based / session-token re-attach optimization for rejoin in general (separate, deferred, explicit YAGNI).
+- Coupling the Requirement 3.7 reporting window to `commit-batch-window`'s (separately parked) commit-urgency tuning — disclaimed in requirements.md's Boundary Context and not referenced anywhere in this design.
+- Recovering a reconciliation-assigned job's original resource allocation (see Data Models — this is a real, explicitly-documented limitation, not a silent gap).
+
+---
+
+## Boundary Commitments
+
+### This Spec Owns
+- The wire protocol for the reconciliation exchange: two new opcodes, `OPCode_jobStateQuery` (controller → worker) and `OPCode_jobStateReport` (worker → controller), both paginated.
+- Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign) — **Correction (2026-09-01, bug-1747, task 3.2): a third new function, `record_job_finish_fact_at`, was added for the finished-success outcome's own Requirement 4.2 fix; see the Persistence Field-Completeness Review section's own correction note for the full rationale** — gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly *after* the worker-join RPC (`xce_join`/`xcecli_join`) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
+- Worker-side: a new short-lived retention of recently-finished job outcomes (jobid, exit status, real finish time), bounded by a configurable window; answering the controller's query from that retention plus the still-running job list; reporting extra jobs outside the query; a new local gate flag that delays the worker's own resumption of normal post-join operation until its report has finished sending (Requirement 3.8).
+- Scheduler-side: one narrow addition — the existing candidate-host filter (`scheduler.c:1211`, today `IS_WORKER_UNAVAIL`-only) also excludes a host with `XCE_WORKER_RECONCILING` set, so a RECONCILING host is never proposed as a dispatch target in the first place (Requirement 5.1). No other scheduler-side change; the worker-status mirror this reads from already exists (`core_events.c:527`/`:595`) and needs no new wire content for this bit.
+- Two new shared-config fields (the Requirement 3.7 reporting window; the Requirement 5.3 reconciliation timeout).
+- **Addendum (2026-09-01, added during implementation):** xceadmin's `config build` YAML→`ClusterConfig` pipeline, scoped narrowly to making the two new fields operator-settable through it, consistent with how their own cited precedent (`heartbeatInterval`/`inactiveHeartbeatCount`) is already wired. This was missed by the original design review — design.md's Configuration Model section says these fields are distributed "via the existing shared-config push mechanism," which is only true end-to-end if xceadmin can actually produce a `ClusterConfig` binary containing operator-supplied values for them; as originally scoped, neither field had a path from YAML into the binary at all. Concretely, in scope: `src/configs/schema/xce.schema.json` (add both fields, same shape/validation style as `heartbeatInterval`), `src/xceadmin/src/xcecommon/mapping.py` (add both to the YAML-field-name → capnp-field-name tuple list at line ~179, immediately following the `heartbeat_interval`/`inactive_heartbeat_count` pair), and the checked-in golden-fixture binaries under `src/xceadmin/tests/fixtures/{minimal,full}/expected.bin` (regenerate — task 1.1 already changed `ClusterConfig`'s wire layout, which these fixtures predate; regenerating them before this field is wired would need redoing once it lands, so do both together). Out of scope, still: any change to `xceadmin`'s validation/business-rule logic beyond the two-field addition, and any question of whether these two fields *should* be operator-facing at all versus hardcoded-only (assumed yes, matching their cited precedent, not re-litigated here).
+
+### Out of Boundary
+- apiserver/scheduler resync-on-reconnect content (`component-rejoin-resync`).
+- The commit engine / pending-facts queue mechanism itself (`commit.c`) — used, never modified.
+- Delta/session-token rejoin optimization.
+- `commit-batch-window`'s commit-urgency tuning — no dependency, no reference.
+- Recovering `job->allocs` (the scheduler's resource allocation) for a job reconciled via Requirement 4.8 that the controller never held a dispatch record for — the data was never known to this controller and cannot be reconstructed from the worker's report. Converges via the existing reserved/actual accounting policy on the job's next rusage report (see Data Models), same as any legitimately-unreserved job today.
+- A parallel WAL fact/decode path for the reconcile-assign case: bug-1678 (fixed 2026-08-25, predates this design) already gave `WAL_JOB_DISPATCH` a `persist_submit_job_dispatch_fact(persist_t*, jobid, state, ts, alloc)` primitive with both apply seams (`apply_committed_entry`, `apply_wal_entry`) decoding an *optional* trailing `JobAlloc` on the **decode** side, and `record_job_reconcile_assign_fact` calls this primitive directly with `alloc=NULL` (see Data Models) rather than inventing a second payload shape — out of boundary is building anything new here, not reusing what already exists. **Correction (2026-09-01, found live during task 2.2 implementation, bug-1746):** the *encode* side of "NULL alloc" was never actually exercised or tested before this feature — `dispatch_a_job_` (bug-1678's only caller) always passes a real, non-NULL allocation, so a latent NULL-pointer dereference in `capnp_encode_joballoc()` (`persist.c`, calls the unsafe `encode_JobAlloc()` instead of the already-existing NULL-safe `encode_JobAlloc_ptr()` used elsewhere in the same generated file) was never triggered until this feature's `record_job_reconcile_assign_fact` became the first genuinely-NULL caller. Fixed as a minimal guard in `capnp_encode_joballoc` (produces a true zero-length buffer for NULL input, matching exactly what the decode side's `if (alloc_len > 0)` check already expects) — in boundary for task 2.2, since it's the direct, minimal companion fix blocking this call, not new machinery.
+- Extending `xce_join()`/`xcecli_join()`'s own blocking wait (`cbuf_deadline_10s`, `xceapi.c`) to also cover config push or job-state reconciliation. That call is a shared, role-agnostic API (worker, apiserver, scheduler, and `xce_test`/`xceadmin` clients all use it) and already returns as soon as the join reply arrives — well before `on_worker_join` (config push) even runs, unchanged by this feature. Reconciliation has its own, independent, server-side timeout (`resync.c`'s `delaymgr` trigger against `workerResyncTimeoutMs`) and is not gated on, or gating of, the join RPC's own deadline in any way.
+
+### Allowed Dependencies
+- `rejoin_worker` / `apply_join_state` (`worker-join-refactor`, implemented) — the plug-in point for triggering reconciliation; this design must not regress its existing attribute/endpoint-update/admin-bit-preservation behavior.
+- The job table's commit-then-apply seam (`record_job_*_fact`, `apply_job_*_transition`, `job_state_is_terminal`, `job_state_is_backward`, `apply_committed_entry`) — reused, never duplicated.
+- `XCE_CONTROLLER_HOOK_JOB` fan-out, `sched.c`'s hook callbacks, and `notify()` (`dataex.c:399`) — the existing controller→scheduler notify path.
+- `xce_reply_get`/`xce_push_data_req`'s `{pos, count, n_total}` pagination idiom (`request.c`) — the template for the new opcode pair.
+- `commit.c`'s `COMMIT_WAIT_RESERVED` admission-reservation pattern — the template for any bounded wait on the worker's report.
+- `config-design.md`'s shared-config classification rule, applied to the two new config fields.
+
+### Revalidation Triggers
+- Any change to `worker_t.host.status` bitmask semantics or `dispatch_a_job_`'s connected-state check (job.c:1511-1512) — the dispatch-gating reuse (Requirement 5.1) depends on both staying exactly as observed. **This trigger fired**: `scheduler-dispatch-commit` (implemented 2026-08-24/25, after this design's original research) added `xce_classify_dispatch_fail()`'s retry/terminal-fail machinery behind this exact check, and exposed bug-1631 (job.allocs never released on a retryable-failure path, deliberately left unfixed pending a re-admission path this design provides). Re-validated 2026-08-31/09-01 — see Architecture, Components, Error Handling for the resulting design changes (scheduler-side candidate exclusion, split disconnected-worker branch, explicit allocs release in `record_job_reconcile_pending_fact`).
+- Any change to `apply_job_update_transition` / `job_state_is_backward`'s RUN↔PEND handling.
+- Any change to the `XCE_CONTROLLER_HOOK_JOB` fan-out shape or the `NotifyData` capnp union.
+- Any change to `JoinData`'s capnp union arms (this design deliberately does not touch it, but a future change there should re-check this design's choice to use a separate opcode pair instead).
+- `component-rejoin-resync`'s eventual design, if it ever proposes touching the join/rejoin wire exchange mechanics this design also touches.
+
+---
+
+## Architecture
+
+### Existing Architecture Analysis
+
+`rejoin_worker`/`apply_join_state` (`worker.c:403`, `:321`) already run under the per-worker write lock at the exact moment `w->ep` is set to the new live endpoint and `XCE_WORKER_UNAVAIL` is cleared — this is the moment the host becomes "connected" again, and the natural trigger point for Requirement 1's "does the controller hold non-terminal jobs for this host" check. `handle_worker_join` (join.c:48) commits the join durably (`persist_commit_barrier`) before `on_worker_join` runs — a second, later point, after the join itself can no longer be rolled back, where the reconciliation query can be sent.
+
+The job table's existing commit-then-apply split is the load-bearing idempotency mechanism this feature must reuse (per requirements.md's Boundary Context): `record_job_*_fact()` functions append a WAL fact (durable, no mutation), and `apply_job_*_transition()` functions are the guarded mutation core, invoked both as a peek at receipt time and later by `apply_committed_entry` once the fact commits. `job_state_is_terminal`/`job_state_is_backward` reject duplicate or backward (terminal→non-terminal) transitions — this is the exact "no duplicate facts, no clobbering newer state" guarantee Requirement 4 asks for, already built and tested (`test_guard_replay_finish_equals_single_application`).
+
+Dispatch gating has two layers today, and both matter to this design. `dispatch_a_job_` (job.c:1278) refuses to dispatch when `w->ep == NULL || w->host.status != 0` (job.c:1511-1512) — but since `scheduler-dispatch-commit` (implemented 2026-08-24/25, after this design's original research), hitting that check no longer just returns an error: it routes through `xce_classify_dispatch_fail()` into a bounded-retry-then-`mark_job_fail()` outcome (job.c:1538-1569, site `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY`) for `w->ep == NULL` and any nonzero status bit alike, with no distinction between them. Separately, bug-1631 (confirmed live, deliberately left unfixed) documents that none of `dispatch_a_job_`'s retryable-failure branches release an already-committed `job->job.allocs` — scoped as unreachable *only* because nothing in the current system re-admits an already-`RUN` job back into the scheduler's PEND pool. Requirement 4.6 is exactly such a re-admission path, so this design cannot treat that gap as pre-existing/out-of-scope (see Components, Error Handling).
+
+Given both findings, `XCE_WORKER_RECONCILING` dispatch-gating is primarily enforced one layer up, at scheduler candidate selection (`scheduler.c:1211`'s existing `IS_WORKER_UNAVAIL`-style filter, extended — see Components), so the scheduler simply never proposes a RECONCILING host as a dispatch target. `dispatch_a_job_`'s status check remains an unchanged defensive backstop for the rare race where a decision predates the bit being observed, correctly falling into the same retry-then-terminal-fail path every other non-UNAVAIL status bit already takes today — unmodified. Requirement 5.4 (a truly disconnected worker, `w->ep == NULL`) needs a genuinely new branch in `dispatch_a_job_`, split out from `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY`.
+
+The controller→scheduler notification path is a hook→notify→wire→dequeue pipeline: job-state-change code fires `XCE_CONTROLLER_HOOK_JOB` (job.c), `sched.c`'s registered callback (`sched_job_hook_fun`) dispatches by `HOOK_DETAIL_*` to `sched_job_finish_fun`/`sched_job_fail_fun`/`sched_job_update_fun`, each of which calls the shared `notify()` (dataex.c:399) with a distinct `NotifyData_data_sched*` event over `c->sched.ep`. On the scheduler side, `sched_handle_notify()` (handler.c:525) is the **inbound** receipt switch that calls `sched_job_state_change()`; this is architecturally distinct from `scheduler.c`/`driver.c`'s own **outbound** decision-publish direction, which a standing test (`test_no_worker_resync_symbols_in_scheduler_publish_path`) mechanically guards against ever referencing "resync." This fixes the only permitted integration seam: `handler.c`'s inbound switch.
+
+The worker daemon retains no finished-job state today: `cleanup_worker_list()` (xcecd_worker_utils.c:385) deletes both the in-memory entry and its sqlite row on the pass immediately after a job exits, and neither `xcecd_worker_t` nor `xcecd_job_t` carries a timestamp field. `JoinData`'s capnp union (`nodata`/`attrs`) carries no job content and cannot be extended in place without forcing the exchange through `xce_join()`'s single-shot synchronous reply, which cannot paginate.
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    subgraph Controller
+        JC[handle_worker_join / join.c]
+        RW[rejoin_worker / apply_join_state / worker.c]
+        RS[Reconciliation state: worker_t.resync]
+        RQ[Reconciliation query+timeout / new resync.c]
+        JOBC[job.c: record_job_finish_fact / record_job_fail_fact<br/>record_job_reconcile_pending_fact NEW<br/>record_job_reconcile_assign_fact NEW<br/>record_job_finish_fact_at NEW - bug-1747]
+        COMMIT[commit.c / apply_committed_entry — unchanged, reused]
+        HOOK[XCE_CONTROLLER_HOOK_JOB fan-out]
+        SCHEDC[sched.c hook callbacks]
+        NOTIFY[notify / dataex.c]
+    end
+    subgraph Worker[xcecd]
+        WJOIN[Join / rejoin]
+        RETAIN[NEW: finished-job retention]
+        RUNNING[Existing: xcecd_worker_t running list]
+        QHANDLE[NEW: job-state query handler]
+    end
+    subgraph Scheduler
+        HANDLER[handler.c sched_handle_notify calls sched_job_state_change]
+        CANDFILT[scheduler.c:1211 candidate filter<br/>NEW: also skips IS_WORKER_RECONCILING<br/>same shape as existing IS_WORKER_UNAVAIL check]
+        PUBLISH[scheduler.c / driver.c: outbound decision publish — NOT touched otherwise]
+    end
+
+    JC --> RW --> RS
+    RS -->|non-terminal jobs for host?| RQ
+    RQ -->|OPCode_jobStateQuery, paginated| WJOIN
+    WJOIN --> QHANDLE
+    QHANDLE --> RETAIN
+    QHANDLE --> RUNNING
+    QHANDLE -->|OPCode_jobStateReport, paginated| RQ
+    RQ --> JOBC
+    JOBC --> COMMIT --> HOOK --> SCHEDC --> NOTIFY --> HANDLER
+    HANDLER -.->|never referenced from| PUBLISH
+    RS -.->|existing worker-status mirror, core_events.c:527/595<br/>no new wire content| CANDFILT
+    CANDFILT -.->|ordinary status-bit read, not a resync concept —<br/>no resync symbol added to this file| PUBLISH
+```
+
+**Architecture Integration**:
+- Selected pattern: additive extension of an existing daemon-to-daemon RPC + existing fact/apply pipeline — no new subsystem, no new process.
+- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (three new functions — `record_job_reconcile_pending_fact`, `record_job_reconcile_assign_fact`, and `record_job_finish_fact_at` (**Correction, 2026-09-01, bug-1747**: added by task 3.2 for Requirement 4.2's finished-success fix; the original count of two predates that fix) — rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is) / scheduler candidate-filter (one added disjunct to an existing check, see below — not zero, but not new machinery either).
+- Existing patterns preserved: commit-then-apply idempotency, `{pos,count,n_total}` pagination, hook→notify scheduler path, per-worker write-lock discipline, and — newly leaned on — the scheduler's existing worker-status-bit candidate filter (`IS_WORKER_UNAVAIL` at `scheduler.c:1211`).
+- New components rationale: reconciliation bookkeeping and worker retention have no existing analog to extend; everything else is reuse.
+- **Scheduler-side touch, and why it doesn't cross `test_no_worker_resync_symbols_in_scheduler_publish_path`'s fence**: that standing test guards `scheduler.c`/`driver.c`'s *outbound decision-publish* direction against ever learning reconciliation *protocol* concepts (round tokens, query/report messages, retention windows). The one line this design adds there — skip a host if `IS_WORKER_RECONCILING(w->status)` — is not that: it's the same *category* of check as the pre-existing `IS_WORKER_UNAVAIL(w->status)` line right next to it, reading an ordinary worker-status bit the scheduler already mirrors for an unrelated reason (`core_events.c:527`/`:595`). No opcode, no reconciliation-round concept, and no symbol named "resync" is introduced into either file. This is called out explicitly, not left for an implementer to discover the fence and guess whether it applies.
+- Steering compliance: async-I/O thread-role contract (bounded wait via admission reservation, not raw blocking); persistence per-field completeness review (see Data Models); config-design.md classification (see Data Models).
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|------------------|-------|
+| Wire protocol | Cap'n Proto, `src/schema/controller.capnp` | New `OPCode_jobStateQuery`/`OPCode_jobStateReport`, new payload structs | Mirrors `JobData`/`WorkerData`/`ResourceVals`'s existing `{ntotal,pos,count}` shape |
+| Controller | C (gnu17), `src/controller/` | Reconciliation trigger, bookkeeping, fact recording, notify | `worker.c`, `job.c`, new `resync.c`, `cluster.c` (opcode dispatch) |
+| Worker daemon | C (gnu17), `src/worker/` | Finished-job retention, query handling | `xcecd_worker_utils.c/h`, new `xcecd_resync.c` |
+| Async I/O | iomgr/vtaskmgr/delaymgr, `src/lib/` | Non-blocking query send/receive; bounded wait via admission reservation | Reuses `commit.c`'s `COMMIT_WAIT_RESERVED` pattern |
+| Shared config | Cap'n Proto `ClusterConfig` | Reporting window (Req 3.7); reconciliation timeout (Req 5.3) | Alongside existing `heartbeatInterval`/`inactiveHeartbeatCount` |
+
+---
+
+## File Structure Plan
+
+### Modified Files
+- `src/schema/controller.capnp` — add `OPCode_jobStateQuery`, `OPCode_jobStateReport` to the `OPCode` enum; add `JobStateQuery{jobids: List(UInt64), pos, count, ntotal}`, `JobStateReport{entries: List(JobStateEntry), pos, count, ntotal}`, `JobStateEntry{jobid, status: JobStateStatus, exitCode, finishTimeMs, queried: Bool}` (`queried=false` marks a Requirement 3.6 extra job); add both as `Request`/`Reply` union arms; add `workerResyncWindowMs`/`workerResyncTimeoutMs` fields to `ClusterConfig`.
+- `src/controller/worker.c` — `rejoin_worker`/`apply_join_state`: after clearing `XCE_WORKER_UNAVAIL`, check for non-terminal jobs assigned to this host (Requirement 1) and, if any, set `XCE_WORKER_RECONCILING` and initialize the per-worker reconciliation record before releasing the write lock. No change to admin-bit preservation or attribute/endpoint update behavior. `on_worker_join` (worker.c:623) — extended, not `handle_worker_join`: it already calls `worker_push_config` as its one existing step (the config the worker needs — hosttype/hostmodel/resource variables — is foundational baseline state, and its own stub comment, `/* consider to push jobs on hosts to */`, already anticipates this exact follow-on). The reconciliation query send is added right after it, in that fixed order (config push, then query) — both sent fire-and-forget, no new ack/reply invented for this. This ordering is safe, not just conventional: the worker's incoming-request dispatch (`myreq_handler`, `controller_connector.c:1563`) is a plain synchronous `switch` invoked one message at a time off its single long-lived client connection (`xcecli_ctx`) — confirmed by reading it, not assumed — so the two requests are not just sent in order, they are *handled* to completion in that same order on the worker side; no thread-pool fan-out is in the path (the worker's own `thread_pool_t` is for job-forking work, unrelated to request dispatch). The actual wait in this design belongs at the *end* of the sequence, not in the middle: the controller does not consider the worker's rejoin/reconciliation complete until the job-state *report* arrives (or the Requirement 5.3 timeout fires) — see Error Handling.
+- `src/controller/join.c` — **no change**. `handle_worker_join` itself is untouched; the reconciliation query is sent from the (extended) `on_worker_join`, which `handle_worker_join` already calls unconditionally after the join reply.
+- `src/controller/job.c` — **Correction (2026-09-01, bug-1747, task 3.2): three new functions, not two — see below.** `record_job_reconcile_pending_fact()` and `record_job_reconcile_assign_fact()` both follow `dispatch_a_job_`'s eager-mutation-plus-explicit-notify pattern (not `record_job_finish_fact`'s deferred one — see Components for why): `record_job_reconcile_pending_fact()` (Requirement 4.6, eagerly calls `apply_job_update_transition(job, JOB_STAT_PEND)` under the write lock, releases `job->job.allocs` — `free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;` — on an applied (non-no-op) transition, durably appends via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)`, then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking; this is bug-1631's own documented candidate fix, applied here because Requirement 4.6 is exactly the "RUN job re-admitted to the PEND pool" precondition bug-1631's fix was deferred pending), `record_job_reconcile_assign_fact()` (Requirement 4.8, eagerly sets `job->job.state`/host assignment under the write lock, calls the existing `persist_submit_job_dispatch_fact(c->persist, jobid, state, ts, alloc=NULL)` — reusing bug-1678's already-tested NULL-alloc decode path on both apply seams verbatim, not a new fact shape — then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking, same as above and for the same reason). Existing `record_job_finish_fact`/`record_job_fail_fact` were originally scoped to be called as-is for the finished-success/finished-failure cases; **found live during task 3.2 implementation that `record_job_finish_fact`'s specific call site has no way to carry the worker-reported finish time (Requirement 4.2) — its own primitive, `persist_submit_job_finish_fact`, accepts one, but `record_job_finish_fact` always derives `time(NULL)` instead.** Fixed with a THIRD new function, `record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms)` — `record_job_finish_fact`'s own worker-reported-finish-time sibling, following `record_job_finish_fact`'s own DEFERRED pattern (durable append only, not eager like the other two new functions above), differing only in passing the caller-supplied (ms→s converted) time to `persist_submit_job_finish_fact` instead of `time(NULL)`; `record_job_finish_fact` itself stays unmodified, still used as-is for `on_job_finish`'s ordinary (non-reconciliation) receipt path. `record_job_fail_fact` is still called as-is for the finished-failure case — Requirement 4.2 remains genuinely unsatisfiable there via any existing path (see the Persistence Field-Completeness Review section's correction note; tracked as the FAIL half of bug-1747, deliberately deferred). Also: split `dispatch_a_job_`'s `w->ep == NULL || w->host.status != 0` check (job.c:1511-1512) into two branches, checked in order — `w->ep == NULL` is resolved directly via `record_job_reconcile_pending_fact()` with **no retry-counter increment and no `mark_job_fail()`**, *before* the code ever reaches `xce_classify_dispatch_fail()` (a deliberate bypass, not a third value on the shared `xce_dispatch_fail_class_t` enum — see Components); `w->host.status != 0` keeps its existing `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY` behavior unchanged (now rarely hit for `XCE_WORKER_RECONCILING` specifically, since the scheduler-side filter below keeps it from being selected in the first place).
+- `src/controller/cluster.c` — new `case OPCode_jobStateReport:` in `myreq_handler`'s dispatch switch (runs after `ha_dispatch_gate`); route to the new reconciliation-apply logic.
+- `src/scheduler/src/scheduler.c` — extend the existing candidate-host filter (`scheduler.c:1211`, today `if (IS_WORKER_UNAVAIL(w->status)) { ... }`) to also skip a host with the new reconciling bit set — same shape, one added disjunct, no new call site. **Correction (2026-09-01, bug-1751, task 5): `scheduler.c:1211`'s `sched_reset_worker_()` is reached only via `sched_begin_session()`, which has zero live callers in `driver.c`/`sim_loop.c` today — both call `xce_core_begin_session()` (`src/scheduler/src/core_session.c`) directly, which runs an independently-maintained mirror, `core_reset_workers()` (`core_session.c:102`, its own header comment: "mirrors scheduler.c's sched_reset_workers/sched_reset_worker_"), not listed below. The identical disjunct is required in BOTH files for Requirement 5.1/5.2 to hold on the actual live path; `scheduler.c:1211`'s copy is applied for parity/dead-code-hygiene but is not itself exercised in production today.
+- `src/include/xce/types.h` — add `XCE_WORKER_RECONCILING` to the worker status bitmask plus an `IS_WORKER_RECONCILING(status)` macro (mirroring `IS_WORKER_UNAVAIL`'s existing shape) for the scheduler-side filter above; add a `resync_state_t` (queried job-ID set, round token, deadline) referenced from `worker_t`.
+- `src/lib/request.c` — `xce_job_state_query_req()`/`xce_job_state_report_req()` encode/decode helpers, following `xce_reply_get`/`xce_push_data_req`'s pagination loop shape.
+
+### New Files
+- `src/controller/resync.c` (+ `src/controller/include/resync.h`) — per-worker reconciliation bookkeeping: arm/clear `XCE_WORKER_RECONCILING`, track queried job IDs and round token (Requirement 6.3's late-report discard), the delaymgr-triggered timeout (Requirement 5.3), and the glue that maps each `JobStateEntry` outcome to the correct `job.c` call.
+- `src/worker/src/xcecd_resync.c` (+ header) — handles incoming `OPCode_jobStateQuery`: looks up each queried job ID against the running list (`xcecd_worker_t`) and the new finished-job retention; builds and sends paginated `OPCode_jobStateReport` replies, followed by extra-job pages (Requirement 3.6).
+- `src/controller/testsuite/test_worker_resync.c` — controller-side unit tests (trigger, query construction, per-outcome fact application, dispatch gating, timeout, discard-on-disconnect), following `test_worker_join.c`'s fixture conventions.
+- `src/worker/testsuite/` — **new directory**; `src/worker` has no `testsuite/` today (per `structure.md`/`testing.md`). This feature is the first worker-side C test coverage; add `test_xcecd_resync.c` and the minimal harness scaffolding it needs (mirroring `controller/testsuite/test.c`'s runner pattern).
+
+### Modified Worker Files
+- `src/worker/include/xcecd_worker_utils.h` / `src/worker/src/xcecd_worker_utils.c` — add a finished-job retention structure (jobid, exit status, real finish time, retain-until) populated at job exit *before* `cleanup_worker_list()` would otherwise reap the entry; `cleanup_worker_list()` retains an entry in this new structure instead of discarding all trace of it, until the configured window elapses.
+
+---
+
+## System Flows
+
+```mermaid
+sequenceDiagram
+    participant W as Worker
+    participant JC as ControllerJoin
+    participant WC as ControllerWorker
+    participant R as ControllerResync
+    participant JOBC as ControllerJob
+    participant SCHED as Scheduler
+
+    W->>JC: OPCode_join (reconnect or failover reconnect)
+    JC->>WC: rejoin_worker(...)
+    WC->>WC: clear UNAVAIL, set ep (per-worker WRITE lock)
+    WC->>R: non-terminal jobs assigned to this host?
+    alt none
+        R-->>WC: no reconciliation needed
+    else one or more
+        R->>WC: arm XCE_WORKER_RECONCILING, init round token + deadline
+    end
+    WC-->>JC: idx
+    JC->>JC: persist_commit_barrier (join durable)
+    JC->>W: xce_reply (join success reply, unchanged)
+    JC->>JC: on_worker_join
+    JC->>W: OPCode_push (config push, unchanged worker_push_config call)
+    JC->>W: OPCode_jobStateQuery (NEW — sent right after, same on_worker_join)
+    Note over W: single connection, synchronous per-message dispatch<br/>(myreq_handler in controller_connector.c line 1563) — config push<br/>is handled to completion before the query is even read as<br/>a distinct message, no ack or reply needed to enforce this
+
+    loop each query page
+        W->>JC: OPCode_jobStateReport (queried=true entries)
+    end
+    W->>JC: OPCode_jobStateReport (queried=false entries — extras, Req 3.6)
+    Note over W: worker holds off entering its normal post-join operating<br/>mode (resuming periodic background reporting) until its full<br/>report has finished sending (Req 3.8) — a new gate flag,<br/>mirroring the existing needs_rejoin pattern in<br/>controller_connector.c, not a new blocking wait
+
+    loop each reported entry
+        alt finished success
+            JC->>JOBC: record_job_finish_fact (existing, reused)
+        else finished failure
+            JC->>JOBC: record_job_fail_fact (existing, reused)
+        else still running
+            JC->>R: clear this jobid from outstanding set (no-op fact)
+        else unrecognized
+            JC->>JOBC: record_job_reconcile_pending_fact (NEW) -- eager mutation + allocs release, durable append, then notify
+            JOBC->>SCHED: persist_notify_job_updated_already_persisted (Req 7.2), same HOOK_DETAIL_JOB_UPDATE path dispatch_a_job_ already uses
+        else unqueried extra job
+            JC->>JOBC: record_job_reconcile_assign_fact (NEW) -- eager mutation, durable append, then notify
+            JOBC->>SCHED: persist_notify_job_updated_already_persisted, same call dispatch_a_job_'s own success path already makes
+        end
+        Note over JOBC: finished success/failure stay fully deferred (record_job_finish_fact/<br/>record_job_fail_fact, unchanged) -- mutation + notify happen later via<br/>apply_committed_entry once the fact commits, same as any ordinary finish/fail.<br/>The two NEW functions above are eager instead, mirroring dispatch_a_job_'s<br/>established pattern, not this deferred one -- see Components for why.
+    end
+
+    R->>WC: outstanding set empty? clear XCE_WORKER_RECONCILING
+    Note over R: if deadline fires first (delaymgr trigger),<br/>resolve outstanding as answered/timed out (Req 5.3)<br/>rather than withholding dispatch indefinitely
+```
+
+**Gating conditions**: Requirement 5.1's primary enforcement is one layer up from `dispatch_a_job_` — the scheduler's own candidate filter (`scheduler.c:1211`) excludes a `RECONCILING` host from consideration before a dispatch decision is ever made (see Components). `dispatch_a_job_`'s existing check (`w->host.status != 0`, job.c:1511) still refuses dispatch as a backstop for the rare decision-predates-the-bit race, and — unchanged — routes through the existing retry-then-terminal-fail classification the same as any other non-UNAVAIL status bit already does today. A dispatch attempt to a disconnected worker (`w->ep == NULL`) is handled separately and differently: it fails immediately and returns the job to pending (Requirement 5.4) via `record_job_reconcile_pending_fact`, checked and resolved *before* the status-bit check reaches `xce_classify_dispatch_fail()` at all — not the existing retry-then-terminal-fail branch (whose retry semantics — retry the dispatch itself — don't fit "the worker isn't even connected," and whose bounded-retry-then-`mark_job_fail()` outcome is the wrong terminal shape for a case Requirement 5.4 wants recoverable-via-reschedule instead). This split is a required code change to `dispatch_a_job_`, not a reuse — see Components.
+
+**Discard-on-disconnect** (Requirement 6): if the worker disconnects before its report is fully processed, `rejoin_worker`'s next invocation (on the next join) finds the prior round's token stale and starts a fresh one; any report that later arrives carrying the stale token is discarded (Requirement 6.3) rather than applied against the wrong round.
+
+---
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|---|---|---|---|---|
+| 1.1, 1.2, 1.3 | Trigger reconciliation iff non-terminal jobs exist for the host, regardless of join type | `rejoin_worker`, `resync.c` | trigger check | Sequence: rejoin → trigger |
+| 2.1, 2.2 | Query lists exactly the non-terminal jobs by unique ID | `resync.c`, wire `JobStateQuery` | `xce_job_state_query_req` | Sequence: OPCode_jobStateQuery |
+| 3.1–3.5 | Worker reports definitive status + real finish time per queried job | worker `xcecd_resync.c`, retention | `JobStateEntry` | Sequence: report loop |
+| 3.6 | Worker reports extra unqueried jobs | worker `xcecd_resync.c` | `JobStateEntry.queried=false` | Sequence: extras page |
+| 3.7 | Bounded, cluster-configurable reporting window | `ClusterConfig.workerResyncWindowMs`, worker retention | config field | Data Models |
+| 3.8 | Worker delays resuming normal post-join operation until its report has finished sending | worker `xcecd_resync.c`, new gate flag | resync-in-progress flag | Sequence: post-report note |
+| 4.1–4.5 | Apply via existing idempotent commit-then-apply; no-op duplicates; never clobber newer/terminal state | `job.c` (existing fns), `apply_job_*_transition` | existing | Sequence: apply per outcome |
+| 4.6 | Unrecognized → return to pending, eligible for scheduling, allocation released | `record_job_reconcile_pending_fact` (NEW, incl. `job->job.allocs` release — bug-1631) | new fn | Sequence: unrecognized branch |
+| 4.7, 4.8 | Reconcile jobs outside the query; assign if no existing non-terminal record | `record_job_reconcile_assign_fact` (NEW, calls `persist_submit_job_dispatch_fact(alloc=NULL)`) | new fn | Sequence: extras branch; Data Models limitation |
+| 5.1, 5.2 | Gate dispatch while reconciliation outstanding; restore eligibility on completion | `scheduler.c:1211` candidate filter (NEW disjunct, primary), `XCE_WORKER_RECONCILING` bit + `dispatch_a_job_` (backstop, reused) | status bitmask | Architecture: dispatch gating |
+| 5.3 | Bounded timeout resolves outstanding reconciliation | `resync.c` delaymgr trigger, `ClusterConfig.workerResyncTimeoutMs` | timeout | Sequence: deadline note |
+| 5.4 | Dispatch to disconnected worker fails fast, returns to pending | new pre-classifier `w->ep == NULL` branch in `dispatch_a_job_`, `record_job_reconcile_pending_fact` (NEW) | — | Error Handling |
+| 6.1–6.3 | Disconnect mid-round discards and re-triggers; idempotent re-application; stale-round reports discarded | `resync.c` round token | round token | Sequence: discard-on-disconnect note |
+| 7.1–7.3 | Reconciled changes reach the scheduler via the existing notify path | `XCE_CONTROLLER_HOOK_JOB`, `sched.c`, `handler.c` (all reused, unchanged) | `notify()` | Sequence: apply → notify |
+| 8.1–8.3 | Works for any job count; paginated multi-message exchange | `JobStateQuery`/`JobStateReport` pagination | `{pos,count,ntotal}` | Sequence: loop each query/report page |
+
+---
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
+|---|---|---|---|---|---|
+| `resync.c` (new) | Controller | Per-worker reconciliation bookkeeping + timeout | 1, 5.3, 6 | `worker.c` (P0), `commit.c` admission pattern (P1) | State, Batch |
+| `job.c` additions | Controller | Two new idempotent fact-recording functions + split disconnected-worker dispatch-fail site | 4.6, 4.8, 5.4 | existing commit-then-apply seam (P0), `xce_classify_dispatch_fail()` (P0) | Service |
+| `xcecd_resync.c` (new) | Worker | Answer job-state query from running list + retention | 3, 8 | `xcecd_worker_utils.c` (P0) | Batch, Event |
+| `xcecd_worker_utils.c` additions | Worker | Short-lived finished-job retention | 3.5, 3.7 | none new | State |
+| Wire opcode pair | Protocol | Paginated query/report exchange | 2, 3, 8 | `xce_reply_get`/`xce_push_data_req` pattern (P0) | API/Batch |
+| `scheduler.c` candidate-filter addition | Scheduler | Exclude a RECONCILING host from dispatch candidacy | 5.1, 5.2 | existing `IS_WORKER_UNAVAIL` filter (P0) | State |
+
+### Controller: `resync.c`
+
+| Field | Detail |
+|-------|--------|
+| Intent | Own the reconciliation-outstanding lifecycle for one worker: arm on trigger, track queried IDs + round token, resolve on completion or timeout |
+| Requirements | 1.1–1.3, 5.1–5.3, 6.1–6.3 |
+
+**Responsibilities & Constraints**
+- Called from `rejoin_worker`/`apply_join_state` under the per-worker write lock already held there — must not introduce a new lock ordering.
+- Owns the round token that distinguishes the current reconciliation attempt from a stale/superseded one (Requirement 6.3).
+- Schedules the Requirement 5.3 timeout via a delaymgr one-shot trigger (non-blocking fire, per the async-I/O thread-role contract) — never a blocking wait on iomgr or delaymgr.
+
+**Dependencies**
+- Inbound: `worker.c` (P0) — trigger call site
+- Outbound: `job.c` (P0) — per-outcome fact application
+- Outbound: `commit.c`'s admission-reservation pattern (P1) — template for any bounded wait, if a synchronous variant is ever needed instead of the pure delaymgr-trigger design above
+
+**Contracts**: Batch [x] / State [x]
+
+##### Batch / Job Contract
+- Trigger: `rejoin_worker` finds ≥1 non-terminal job for the rejoining host
+- Input / validation: the controller's own non-terminal job set for that host (Requirement 2.1) — no external input to validate
+- Output / destination: `job.c` fact-recording calls, one per reported entry
+- Idempotency & recovery: round token discards stale reports (6.3); a fresh round replaces a discarded one on the next join (6.1); applying a report twice for the same round is a no-op by construction of the underlying `job.c` guards (6.2)
+
+##### State Management
+- State model: one `resync_state_t` per `worker_t` — `{round_token, queried_jobids[], deadline}`; absent/cleared when no reconciliation is outstanding
+- Persistence & consistency: **not itself persisted** — it is in-memory reconciliation-in-progress bookkeeping, not a durable fact; the durable facts it produces are the existing job-table WAL entries. A controller crash mid-reconciliation loses this bookkeeping, which is correct: the next rejoin (post-failover) re-triggers reconciliation from scratch (Requirement 1.3), so nothing needs to survive a crash here.
+- Concurrency strategy: guarded by the same per-worker write lock `rejoin_worker` already holds
+
+### Controller: `job.c` — `record_job_reconcile_pending_fact`, `record_job_reconcile_assign_fact`
+
+| Field | Detail |
+|-------|--------|
+| Intent | The two reconciliation outcomes with no existing idempotent fact analog, plus a required correction to a dormant retryable-dispatch-failure gap this feature reactivates |
+| Requirements | 4.6, 4.7, 4.8, 5.4 |
+
+**Responsibilities & Constraints**
+
+Both new functions follow `dispatch_a_job_`'s own established pattern (job.c:1686-1748: eager, synchronous mutation under the write lock, immediately followed by a durable append, immediately followed by an explicit `persist_notify_job_updated_already_persisted(c, job)` call) — **not** `record_job_finish_fact`'s purely-deferred pattern (state mutation withheld entirely until `apply_committed_entry` applies the committed fact). This choice is deliberate, not copied without thought: `persist_notify_job_updated_already_persisted`'s own doc comment (`persist.h`) frames it generically ("e.g. `dispatch_a_job_()`'s own... call" — an example, not an exclusive binding) as the sanctioned way to notify for *any* caller whose "live, synchronous state... mutation" would otherwise leave `apply_committed_entry`'s later replay finding "state already matches" and silently skipping its own notify (bug-1679's exact failure mode). Both reconciliation outcomes share dispatch_a_job_'s justification for going eager rather than deferred: the fact being recorded is something the *worker* already confirmed really happened (or definitely didn't) — ground truth already changed outside this controller — not a same-node admin command whose commit could still be rolled back with nothing external yet depending on it (that second, riskier shape is `job_ctrl`'s `mark_job_update`, explicitly flagged in `roadmap.md`'s backlog as a legacy pattern to migrate *away* from, not a template to extend). Mixing an eager mutation for one field (e.g. `allocs`) with a deferred mutation for another (`state`) on the *same* transition was considered and rejected — it would create a real window where a job reads back as `RUN` with `allocs == NULL`, an inconsistent combination nothing else in this codebase produces; both fields move together, eagerly, under one lock hold.
+
+- `record_job_reconcile_pending_fact(cluster_t*, job_t*)`: under the job's write lock, calls `apply_job_update_transition(job, JOB_STAT_PEND)` directly (not as a "peek" — as the real, live mutation) — this reuses the existing `job_state_is_backward` guard unchanged, returning early as a no-op if it reports `JOB_TRANSITION_REJECTED_BACKWARD` or `JOB_TRANSITION_NOOP_DUPLICATE` (Requirement 4.3/4.4). Only on `JOB_TRANSITION_APPLIED`: releases `job->job.allocs` (`free_JobAlloc_ptr(&job->job.allocs); job->job.allocs = NULL;`) if non-NULL, still under the same lock hold — this is required, not optional: `apply_job_update_transition` only ever touches `job->job.state` (job.c:2808-2834, confirmed by reading it) and never allocs, and bug-1631 (confirmed live, deliberately left unfixed) documents exactly this gap on `dispatch_a_job_`'s own retryable-failure paths, scoped as unreachable "only because nothing in the current system re-admits an already-RUN job back into the scheduler's PEND pool." Requirement 4.6 is precisely that re-admission path. Then, still under the lock: durably appends the fact via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)` (the plain `{jobid, state}` shape is sufficient here — no extra fields, unlike DISPATCH/START/FINISH's richer siblings). After unlocking: calls `persist_notify_job_updated_already_persisted(c, job)`, reaching the scheduler through Requirement 7's existing notify path exactly as `dispatch_a_job_` already does for its own eager mutation. Also used for Requirement 5.4 (dispatch-to-disconnected-worker fail-fast) — see the `job.c` split below; the allocs release is a no-op there since a disconnected-worker dispatch attempt never committed one.
+- `record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*, xce_job_state_entry_t*)`: under the job's write lock, eagerly sets `job->job.state` and the host assignment to match the worker's report (mirroring `dispatch_a_job_`'s own inline mutation at job.c:1686-1717, minus the allocation fields — there are none to set); `job->job.allocs` stays `NULL` (see Data Models — there genuinely is none to write). Then, still under the lock: calls the existing `persist_submit_job_dispatch_fact(&c->persist, jobid, state, ts, alloc)` (job.c:1731-1734, the exact primitive `dispatch_a_job_`'s own success path already calls) directly, with `alloc = NULL` — bug-1678's fix (2026-08-25) already made this primitive's NULL-alloc case an explicit, backward-compatible branch on both **apply/decode** seams; the **encode** side had a latent NULL-deref (never exercised until this task — bug-1746, fixed as part of task 2.2, a minimal guard in `capnp_encode_joballoc`), so this is a thin caller into existing code that this task also had to make genuinely NULL-safe, not a new WAL fact shape. After unlocking: calls `persist_notify_job_updated_already_persisted(c, job)`, same as above and for the same reason — without it, `apply_committed_entry`'s later replay of this exact `WAL_JOB_DISPATCH` entry finds state already matching and silently skips its own notify, reproducing bug-1679 at this new call site if omitted.
+- Split in `dispatch_a_job_` (job.c:1511-1512): `w->ep == NULL` is checked and resolved **before** the existing `w->host.status != 0` check reaches `xce_classify_dispatch_fail()` at all — calling `record_job_reconcile_pending_fact()` directly, with the job's `dispatch_fail_count` left untouched and `mark_job_fail()` never called, then returning. This is deliberately a bypass, not a third value added to `xce_dispatch_fail_class_t`: that enum's whole contract (per its own precedent, `commit-batch-window`'s requirements.md: "an exhaustive site-identity enum maps to a decision — TERMINAL vs RETRYABLE") is a two-way choice every other call site's `switch` already exhaustively handles; adding a third value would force each of those ~5 unrelated switches to grow a case they can never hit. `w->host.status != 0` (any bit, `XCE_WORKER_RECONCILING` included) is otherwise unchanged — still reaches `xce_classify_dispatch_fail(XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY)` and its existing retryable/terminal path, same as today, on the (now rare, since the scheduler-side filter keeps it from being the common case) chance a dispatch decision predates the bit being observed.
+- No function introduces a new *commit/idempotency* guard — all three reuse existing guarded seams (`apply_job_update_transition`'s `job_state_is_backward`, `persist_submit_job_fact`/`persist_submit_job_dispatch_fact`'s existing WAL append contracts) and the existing, already-proven-safe `persist_notify_job_updated_already_persisted` notify primitive. Nothing here is a new persistence *mechanism* — it is the same eager-mutation-plus-explicit-notify shape `dispatch_a_job_` already established, applied to two new call sites.
+
+**Contracts**: Service [x]
+
+##### Service Interface (C)
+```c
+int record_job_reconcile_pending_fact(cluster_t* c, job_t* job);
+int record_job_reconcile_assign_fact(cluster_t* c, job_t* job,
+                                     worker_t* w, xce_job_state_entry_t* report);
+```
+- Preconditions: caller holds `job`'s write lock on entry (matching every other `record_job_*_fact` sibling); `job` is non-terminal (pending case) or has no existing non-terminal assignment (assign case)
+- Postconditions: unlike `record_job_finish_fact`/`record_job_fail_fact` (deferred — mutation happens later via `apply_committed_entry`), both of these functions mutate `job`'s live in-memory state **eagerly, before returning** — matching `dispatch_a_job_`'s established pattern, not the deferred one — then durably append the fact, then (after the caller releases the lock) notify via `persist_notify_job_updated_already_persisted`. `record_job_reconcile_pending_fact` additionally guarantees `job->job.allocs == NULL` on return whenever it was non-NULL on entry, mutated atomically alongside the state change under the same lock hold (in-memory only, not itself a WAL field — see Data Models).
+- Invariants: never called for a job the controller already knows is terminal or more-final (Requirement 4.4/4.5 — enforced by the existing `job_state_is_backward` guard, not by new logic here); the caller must call `persist_notify_job_updated_already_persisted(c, job)` after unlocking, exactly once, for every actual (non-no-op) mutation — omitting it reproduces bug-1679 at this call site.
+
+### Worker: `xcecd_resync.c` (new)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Answer the controller's job-state query from the running list and the new finished-job retention; report extras; gate the worker's own post-join wake-up on completion |
+| Requirements | 3.1–3.8, 8.2 |
+
+**Responsibilities & Constraints**
+- For each queried job ID: check the running list (`xcecd_worker_t`) first; if not running, check the new retention; if in neither, report `unrecognized` (Requirement 3.1, 3.3). Every job ID present in the query gets an entry in the reply — none are silently skipped (Requirement 3.2).
+- Include the real finish time from retention, not the time the report is sent (Requirement 3.5).
+- After answering all queried IDs, walk the running list and retention for anything not in the query and report it too (Requirement 3.6), using the same `JobStateEntry` shape with `queried=false`.
+- Send both in paginated `OPCode_jobStateReport` messages (Requirement 8.2).
+- Sets a new local `resync_pending` flag (worker-side, mirroring `controller_connector.c`'s existing `needs_rejoin` boolean pattern) when `OPCode_jobStateQuery` is received; clears it only once every report page (queried and extra) has finished sending. Whatever the worker's own post-join sequence does next (resuming periodic background reporting, entering normal operating mode) checks this flag first (Requirement 3.8). This is a local gate on the worker's own follow-on behavior, not a new wire-level ack and not an extension of `xce_join()`'s blocking wait (see Boundary Commitments' Out of Boundary).
+
+**Dependencies**
+- Inbound: query receipt (P0)
+- Outbound: `xcecd_worker_utils.c`'s running list + new retention (P0)
+
+**Contracts**: Batch [x] / Event [x]
+
+##### Batch / Job Contract
+- Trigger: `OPCode_jobStateQuery` received
+- Input / validation: job IDs in the query — no validation needed beyond matching against local state
+- Output / destination: `OPCode_jobStateReport`, one or more pages
+- Idempotency & recovery: re-sending the same query (e.g. after the worker's own reconnect mid-exchange) produces the same answers — the worker's local state is the read-only source of truth for this exchange
+
+### Worker: `xcecd_worker_utils.c` additions — finished-job retention
+
+| Field | Detail |
+|-------|--------|
+| Intent | Retain a job's true outcome (status, exit code, real finish time) for a bounded window after exit, instead of discarding it at the next `cleanup_worker_list()` pass |
+| Requirements | 3.4, 3.5, 3.7 |
+
+**Responsibilities & Constraints**
+- Populated at the same point a job's exit is currently detected, before `cleanup_worker_list()` would otherwise delete the entry.
+- Bounded by `ClusterConfig.workerResyncWindowMs` (Requirement 3.7) — entries older than the window are dropped, not retained indefinitely.
+- Does not change `cleanup_worker_list()`'s existing behavior for the in-flight `xcecd_worker_t`/sqlite bookkeeping — it adds a second, smaller, timestamped structure fed at the same moment, not a rework of the existing one.
+
+**Contracts**: State [x]
+
+##### State Management
+- State model: a small bounded list/ring of `{jobid, exit_status, exit_code, finish_time_ms}`, in-memory only
+- Persistence & consistency: **not persisted** — a worker restart loses this retention, which is acceptable: a restarted worker has no running jobs of its own to report on rejoin either way (its child processes died with it), so the retention's only job is to survive a *controller-side* disconnect/reconnect window, not a worker restart
+- Concurrency strategy: same locking discipline already used for `xcecd_worker_t`'s list
+
+### Scheduler: `scheduler.c` candidate-filter addition
+
+| Field | Detail |
+|-------|--------|
+| Intent | Keep a `RECONCILING` host out of dispatch candidacy at the source, so Requirement 5.1 is enforced without depending on the controller-side retry/terminal-fail machinery for the common case |
+| Requirements | 5.1, 5.2 |
+
+**Responsibilities & Constraints**
+- One added disjunct to the existing check at `scheduler.c:1211` — today `if (IS_WORKER_UNAVAIL(w->status)) { ... }`, becomes `if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { ... }` (new macro, `types.h`, same shape as the existing `IS_WORKER_UNAVAIL`/`IS_WORKER_LOCKEDU`/etc. family).
+- Reads `w->status`, a field the scheduler already receives via the existing worker-status mirror (`core_events.c:527`/`:595`) — no new wire content, no new push.
+- Does **not** touch `scheduler.c`/`driver.c`'s outbound decision-*publish* logic or introduce any reconciliation-protocol concept into either file — see Architecture Integration for why this stays clear of `test_no_worker_resync_symbols_in_scheduler_publish_path`'s fence.
+- Requirement 5.2 ("eligible for new job dispatch again" once reconciliation completes) needs no separate scheduler-side action: once the controller clears `XCE_WORKER_RECONCILING` and that status change is next mirrored to the scheduler, this same filter simply stops excluding the host.
+
+**Dependencies**
+- Inbound: the existing worker-status mirror (P0) — no new dependency introduced
+
+**Contracts**: State [x]
+
+##### State Management
+- State model: none new — reads the existing `xce_sched_worker_t.status` mirror field
+- Persistence & consistency: not applicable — a read of already-existing, already-synced state
+- Concurrency strategy: unchanged — same read path as the existing `IS_WORKER_UNAVAIL` check it sits beside
+
+---
+
+## Data Models
+
+### Wire/Protocol Model
+
+```
+JobStateEntry {
+  jobid: UInt64
+  status: JobStateStatus (RUNNING | FINISHED_OK | FINISHED_FAIL | UNRECOGNIZED)
+  exitCode: Int32           # meaningful only for FINISHED_FAIL
+  finishTimeMs: UInt64      # meaningful only for FINISHED_OK/FINISHED_FAIL; worker-reported, per Req 3.5
+  queried: Bool             # false = Requirement 3.6 extra, not part of the controller's original query
+}
+
+JobStateQuery { jobids: List(UInt64), pos: UInt32, count: UInt32, ntotal: UInt32 }
+JobStateReport { entries: List(JobStateEntry), pos: UInt32, count: UInt32, ntotal: UInt32 }
+```
+
+### Controller-Side In-Memory Model
+
+```
+resync_state_t {          // new, attached to worker_t
+  round_token: uint64      // discards stale/superseded reports (Req 6.3)
+  queried_jobids: set<uint64>
+  deadline: timestamp      // Req 5.3
+}
+```
+
+### Persistence Field-Completeness Review
+
+Per `design-principles.md`'s Persistence Review requirement and `persistence.md`'s `APPLY_DIVERGENT`/bug-1678 precedent, every fact this feature introduces or reuses is enumerated below:
+
+| Outcome | Fact path | New fields? | Both apply seams? |
+|---|---|---|---|
+| Finished success | `record_job_finish_fact_at` (NEW, task 3.2/bug-1747) | None — reuses `persist_submit_job_finish_fact`'s existing, already-complete `WAL_JOB_FINISH` payload shape verbatim (`finish_time`, exit info); the reconciliation caller passes the worker-reported `finishTimeMs` (Requirement 3.5, converted ms→s) as that fact's finish time, never the time the controller processed the report (Requirement 4.2) | Already covered by existing `apply_committed_entry`/standby-catch-up handling — unchanged, since the wire payload shape is identical to `record_job_finish_fact`'s own |
+| Finished failure | `record_job_fail_fact` (existing, reused as-is) | None — same Requirement 4.2 worker-reported-time rule applies | Unchanged, existing |
+| Still running | No fact — pure no-op against the outstanding set | N/A | N/A |
+| Unrecognized → pending | `persist_submit_job_fact(..., WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)` (existing generic primitive, new call site) | None — the plain `{jobid, state}` shape is sufficient, same as every other `WAL_JOB_UPDATE` caller. **`job->job.allocs` release is in-memory only, eager, not itself a persisted field** — mutated live under the same lock as the state change (mirroring `dispatch_a_job_`'s own success path in reverse; see Components), then notified via `persist_notify_job_updated_already_persisted`, not left for `apply_committed_entry` to discover | Both seams already decode plain `WAL_JOB_UPDATE` identically today (it is not a new payload shape) — a replay of this fact re-derives `job->job.state = JOB_STAT_PEND` correctly; it does **not** re-derive the allocs release (that mutation is not wire content), but since `dispatch_a_job_`'s eager pattern already established this exact asymmetry (live mutation + explicit notify now, replay-idempotent state field only, no replay-time side effect needed for a purely in-memory cleanup), this is consistent with existing precedent, not a new gap |
+| Unqueried job → assign | `persist_submit_job_dispatch_fact(..., alloc=NULL)` (existing primitive, new call site, `record_job_reconcile_assign_fact`) | None — reuses `persist_submit_job_dispatch_fact`'s existing `WAL_JOB_DISPATCH` payload shape verbatim with `alloc=NULL`, exercising the same NULL-alloc decode branch bug-1678 already added. Live mutation is eager (see Components), matching `dispatch_a_job_`'s own pattern for the same fact type | Decode side already covered by bug-1678 on both seams. **Encode side required a fix** (bug-1746, task 2.2): `capnp_encode_joballoc` unconditionally dereferenced a NULL `alloc` — this was this codebase's first genuinely-NULL caller. Fixed with a minimal zero-length-buffer guard, matching the decode side's `alloc_len > 0` contract exactly; no new decode path needed once the encode side actually produces the zero-length payload the decode side already expected |
+
+**Correction (2026-09-01, found live during task 3.2 implementation, bug-1747) — a THIRD new function, not two:** this section's original "Finished success" row asserted `record_job_finish_fact` could be reused "as-is," citing an "existing complete fact shape (already carries `finish_time`)" — accurate about `persist_submit_job_finish_fact` (the underlying primitive, which does accept an explicit `finish_time` parameter) but not about `record_job_finish_fact`'s own specific call site, which never exposed that parameter to a caller and always derives `time(NULL)` at receipt time instead. Calling `record_job_finish_fact` unmodified therefore could NOT satisfy Requirement 4.2 for the finished-success case — confirmed live when task 3.2 wired the reconciliation caller and found no way to pass the worker-reported time through. Fixed (coordinator-directed, mirroring bug-1746's own precedent for an in-boundary companion fix) by adding `record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms)` (job.c) — `record_job_finish_fact`'s own worker-reported-finish-time sibling: an identical guard/hook-fire/durable-append body, differing only in passing the caller-supplied (ms→s converted) time to `persist_submit_job_finish_fact` instead of `time(NULL)`. `record_job_finish_fact` itself is left UNMODIFIED — its own caller (`on_job_finish`, an ordinary near-instantaneous worker report) is correctly served by receipt-time there. Unlike `record_job_reconcile_pending_fact`/`record_job_reconcile_assign_fact` (both EAGER, `dispatch_a_job_`-style), `record_job_finish_fact_at` follows `record_job_finish_fact`'s own DEFERRED pattern (durable append only; the in-memory mutation and scheduler notify happen later via `apply_committed_entry`'s replay, exactly as an ordinary finish already works) — so this section's job.c fact-recording count is **three new functions**, not two (see also the Architecture Pattern & Boundary Map bullet and the Modified Files entry for `job.c`, both correspondingly updated). The sibling "Finished failure" outcome remains genuinely unfixable via any existing path: `WAL_JOB_FAIL`'s wire payload carries no finish-time slot at any layer, for any caller in this codebase, and `apply_job_fail_transition` unconditionally stamps `time(NULL)` at apply time — a pre-existing, systemic gap that predates this feature entirely (bug-1747, re-scoped 2026-09-01), deliberately left deferred rather than patched narrowly inside this task.
+
+**Documented scope decision — `job->allocs` on a reconciliation-assigned job (Requirement 4.8):** when the controller reconciles a job it never held any prior record of (e.g. it was dispatched by a controller instance before a failover whose dispatch fact was itself lost), it has no way to learn what resource allocation the scheduler originally computed for that job — the worker's report carries the job's execution outcome, not its original resource request. This design deliberately does not attempt to reconstruct `job->allocs` for this case. **Correction (2026-09-01, re-validated against current code):** an earlier draft of this section cited `persistence.md`'s `APPLY_DIVERGENT` section / bug-1678 as evidence this is "a tracked, separate gap" `WAL_JOB_DISPATCH` still has — that citation was stale. Bug-1678 was fixed 2026-08-25 (before this design was finalized): `WAL_JOB_DISPATCH` now durably carries `job->allocs` when one exists, on both apply/decode seams. The real reason this design still writes no allocation for Requirement 4.8 is simpler and unrelated to that gap: there just isn't one to write — the scheduler, not this controller, is the only party that ever computes a job's allocation, and for a job this controller never had any record of, no dispatch (successful or otherwise) ever ran through this controller to produce one. `record_job_reconcile_assign_fact` reuses the *fixed* `persist_submit_job_dispatch_fact` primitive because its NULL-alloc *decode* case is already the correct representation of "state and host known, allocation not" — though the *encode* side of that same NULL case had never actually been exercised before this feature (bug-1746: `capnp_encode_joballoc` dereferenced a NULL `alloc` unconditionally, since `dispatch_a_job_`, bug-1678's only prior caller, always passes a real allocation). Task 2.2 fixed the encode side with a minimal zero-length-buffer guard as part of making this call safe to actually make — a real, if narrow, gap in the "already-tested" characterization this design originally gave the primitive, closed as part of implementation rather than left latent. The practical consequence is bounded by `resource-accounting.md`'s existing policy: a job with an unknown/zero reservation is charged `max(reserved=0, actual)` — i.e., its real usage — the moment its next rusage tick arrives, identical to how a legitimately-unreserved job is already handled today. The job's *state* is correct immediately; its resource *accounting* converges within one rusage cycle, not instantly. No fabricated allocation value is ever written.
+
+### Configuration Model
+
+Per `config-design.md`'s classification rules, applied explicitly (not left implicit):
+
+| Field | Rule 0 (bootstrap) | Independence | Frequency | Verdict | Why |
+|---|---|---|---|---|---|
+| `workerResyncWindowMs` (Req 3.7) | passes — not needed to obtain config | passes — one worker's reporting window has no cross-node correctness dependency | passes — rarely changed | **Shared, via operational-uniformity override** | Structurally identical to `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness domain): no correctness invariant demands agreement, but operators want uniform, predictable reconciliation behavior across the cluster rather than per-node drift — same documented exception, same rationale |
+| `workerResyncTimeoutMs` (Req 5.3) | passes | passes — one worker's reconciliation deadline has no cross-node dependency | passes | **Shared, via operational-uniformity override** | Same reasoning as above; kept alongside `workerResyncWindowMs` in `ClusterConfig` rather than splitting the override rationale across two places |
+
+Both fields are added to the existing `ClusterConfig` capnp struct, distributed via the existing shared-config push mechanism — no new configuration surface or distribution channel.
+
+---
+
+## Error Handling
+
+### Error Strategy
+Every reconciliation outcome resolves definitively — there is no "leave it ambiguous" outcome. A worker that cannot determine a job's true status reports `UNRECOGNIZED` rather than guessing (Requirement 3.3), which the controller treats as "return to pending," never as a fabricated finish.
+
+### Error Categories and Responses
+- **Worker disconnects mid-round** (Requirement 6.1): the outstanding round is discarded (not applied partially); the next join re-triggers a fresh round via the same Requirement 1 check.
+- **Stale/superseded report arrives** (Requirement 6.3): discarded by round-token mismatch in `resync.c` — never applied against a round that's already been superseded.
+- **Reconciliation report never arrives / no reply within the bound** (Requirement 5.3): a delaymgr-scheduled trigger (non-blocking fire, per the async-I/O thread-role contract) resolves the outstanding round — accepted-as-answered, timed out, or left for the next rejoin to retry — rather than withholding that worker from dispatch indefinitely. This must not be a blocking wait anywhere in the iomgr/delaymgr call chain; if a synchronous wait style is preferred at implementation time instead of a pure trigger-and-resolve design, it must follow `commit.c`'s `COMMIT_WAIT_RESERVED` admission-reservation pattern (bounded blocking under an explicit cap on the vtask pool), never a raw wait.
+- **Dispatch attempted to a disconnected worker** (Requirement 5.4): fails immediately (not queued for later delivery) and returns the job to pending via `record_job_reconcile_pending_fact` — checked and resolved directly in `dispatch_a_job_`, ahead of and separate from the `w->host.status != 0`/`xce_classify_dispatch_fail()` path, not a new value bolted onto that shared TERMINAL/RETRYABLE enum: retrying the same dispatch attempt against a worker that isn't even connected doesn't fit either existing outcome, and unlike a TERMINAL resolution this one must NOT call `mark_job_fail()` — it must leave the job schedulable again, not fail it.
+- **Dispatch attempted to a `RECONCILING` worker** (Requirement 5.1): primarily prevented at the source — `scheduler.c`'s candidate filter (extended, see Components) excludes a `RECONCILING` host, so the scheduler does not produce a dispatch decision for it in the first place. If one is produced anyway (a decision-predates-the-bit race), `dispatch_a_job_`'s unchanged `w->host.status != 0` check still refuses it, falling into the existing `WORKER_NOT_READY` retry-then-terminal-fail path — the same, unmodified behavior every other non-UNAVAIL status bit (`LOCKEDU`/`LOCKEDW`/`BUSY`/`CLOSED`) already produces today. This design does not change that behavior for the race case; it only makes the race rare by filtering earlier.
+- **A job reconciled from RUN back to PEND is later re-dispatched and fails retryably** (Requirement 4.6, bug-1631): confirmed as bug-1631's own deliberately-deferred trigger condition — a RUN job with a real `job->job.allocs` returned to PEND and re-admitted to the scheduler's dispatch pool, something nothing in the system did before this feature. `record_job_reconcile_pending_fact` closes this by releasing `job->job.allocs` at the same point it sets the job PEND (see Components), so a subsequent redispatch attempt — successful or retryably-failing — never observes a stale allocation left over from the job's prior dispatch. This must ship together with Requirement 4.6, not be left as a follow-on: shipping 4.6 without it would take bug-1631 from "deliberately deferred, unreachable" to "live, reachable, and unfixed."
+- **New opcode reaches a non-primary controller**: per `api-standards.md`'s "Adding a New Opcode" checklist (bug-1700 precedent) — verify at implementation time whether `OPCode_jobStateQuery`/`OPCode_jobStateReport` need an explicit redirect-reply path. Expected answer: no, since a worker only ever holds a connection to the current primary (connection-lifecycle-tied-to-primary-status, per the HA connection-ownership model), so this opcode pair should never reach a standby in practice — but this must be confirmed, not assumed, given bug-1700's documented history of this exact class of gap recurring across two other opcodes.
+
+### Monitoring
+Reuses existing job-state-change logging (job.c's existing log call sites for finish/fail/update facts) — no new logging framework. A reconciliation-driven fact should log which outcome and round produced it, for operator diagnosis of a rejoin event, matching the existing style (structured, operator-visible on the ERROR/WARN path per `error-handling.md`; INFO for normal reconciliation completion).
+
+---
+
+## Testing Strategy
+
+### Unit Tests (Controller, `src/controller/testsuite/test_worker_resync.c`)
+- Reconciliation triggers iff the host has ≥1 non-terminal job at rejoin (Requirement 1.1/1.2), for all three join circumstances (first join, network reconnect, failover reconnect) without the code needing to distinguish them (Requirement 1.3).
+- Query construction lists exactly the controller's non-terminal jobs for that host by ID (Requirement 2).
+- Each of the five reported outcomes (running, finished-success, finished-failure, unrecognized, unqueried-extra) applies through the correct existing/new fact function exactly once, and a repeated report for the same round is a no-op (Requirement 4, mirroring `test_guard_replay_finish_equals_single_application`'s existing style).
+- `XCE_WORKER_RECONCILING` still blocks `dispatch_a_job_` as a backstop (unchanged existing behavior) and clears once the round resolves (Requirement 5.1/5.2).
+- `record_job_reconcile_pending_fact` releases a non-NULL `job->job.allocs` when reconciling a RUN job back to PEND, and leaves it untouched (already NULL) for a job that never held one — regression coverage for bug-1631's exact deferred trigger condition: dispatch (real allocs committed) → reconcile-to-pending via an `unrecognized` report → assert `job->job.allocs == NULL` and `job->job.state == JOB_STAT_PEND` **immediately after the call returns, before any commit round-trip** (the mutation is eager, not deferred — see Components) → redispatch → a retryable dispatch failure on that redispatch leaves no stale allocation and no incorrect state (Requirement 4.6).
+- Both `record_job_reconcile_pending_fact` and `record_job_reconcile_assign_fact` call `persist_notify_job_updated_already_persisted` exactly once per actual (non-no-op) mutation, reaching the scheduler's/apiserver's notify path immediately — not only after a later `apply_committed_entry` replay finds a no-op and silently skips its own notify (bug-1679's exact failure mode, reactivated at these two new call sites if this assertion is missing). Assert the hook fires synchronously within the call, not merely "eventually."
+- `record_job_reconcile_assign_fact` round-trips through both apply seams (`apply_committed_entry`, `apply_wal_entry`) via `persist_submit_job_dispatch_fact`'s existing NULL-alloc path — assert a real WAL entry is written and a **restart replay** (a separate scenario from the live eager-mutation path above) reconstructs jobid/host/state with `allocs == NULL` and correctly no-ops its own hook fire (state already matches from the eager mutation, or from the fact's own replay if this happens after a crash before the eager mutation's process ever ran again) — not a new decode path (Requirement 4.8).
+- `dispatch_a_job_`'s new pre-classifier `w->ep == NULL` branch: dispatch to a disconnected worker fails immediately, does not increment `dispatch_fail_count`, does not call `mark_job_fail`, never reaches `xce_classify_dispatch_fail()`, and returns the job to PEND via `record_job_reconcile_pending_fact` (Requirement 5.4) — asserted separately from the unchanged `w->host.status != 0`/`WORKER_NOT_READY` case, which keeps its existing retry-then-terminal-fail behavior verbatim.
+- Timeout resolves an outstanding round without a blocking call (Requirement 5.3).
+- Disconnect mid-round discards and re-triggers; a stale-round report is discarded (Requirement 6).
+- A reconciled state change reaches the scheduler via the existing notify path, asserted the same way existing finish/fail notify tests assert it (Requirement 7).
+
+### Unit Tests (Scheduler, existing `scheduler.c` candidate-selection test coverage)
+- A host with `XCE_WORKER_RECONCILING` set is excluded from dispatch candidacy the same way an `IS_WORKER_UNAVAIL` host already is (Requirement 5.1); a host with the bit cleared is eligible again (Requirement 5.2). Added alongside the existing `IS_WORKER_UNAVAIL` candidate-filter test, not a new test file.
+- Confirm the new check introduces no string/symbol named `resync` into `scheduler.c`/`driver.c` — `test_no_worker_resync_symbols_in_scheduler_publish_path` (`test_scheduler_decision_redelivery.c:432`) must stay green unmodified.
+
+### Unit Tests (Worker, new `src/worker/testsuite/test_xcecd_resync.c`)
+- Query answered correctly from the running list, from retention, and as `UNRECOGNIZED` when in neither (Requirement 3.1–3.3).
+- Real finish time reported, not send time (Requirement 3.5).
+- Extras reported for jobs outside the query (Requirement 3.6).
+- Retention window boundary: an entry just inside/outside `workerResyncWindowMs` is/isn't reported (Requirement 3.7).
+- `resync_pending` gate: set on receiving `OPCode_jobStateQuery`, stays set across multiple report pages, and only clears after the last page (queried and extra) has been sent — the worker's post-join wake-up path must observe it set throughout, never cleared early (Requirement 3.8).
+
+### Integration Tests
+- Full rejoin-reconciliation round trip against a live controller+worker pair, mirroring `test_worker_join.c`'s fixture conventions (`setup_test_cluster`/`teardown_test_cluster`, direct calls, one test per behavior).
+- Failover-during-job: a job's finish notify is lost across a primary failover; the new primary's rejoin reconciliation recovers it (this is the feature's core bug-1741 regression scenario).
+- Burst/array scenario resembling bug-1741's original reproduction (many near-simultaneous finish reports lost on one host) — resync must recover all of them, unbounded by count (Requirement 8).
+
+### Performance/Load
+- Reconciliation exchange for a worker running an order of magnitude more jobs than a single message can carry completes correctly across multiple pages, not truncated (Requirement 8.1–8.3).
+
+---
+
+## Security Considerations
+No new authentication/authorization surface — the new opcode pair travels over the same authenticated controller↔worker connection as every other daemon message, subject to the same `ha_dispatch_gate` role-based admission already governing every opcode. No new external input validation beyond matching reported job IDs against the controller's own records (an unrecognized ID is simply itself information, per Requirement 3.3/4.5 — never treated as an error condition to reject).
+
+## Supporting References
+Deep file:line citations, the alternatives considered for the wire-protocol shape, and the full rationale for each design decision above live in `research.md`.

.kiro/specs/worker-rejoin-resync/requirements.md

@@ -0,0 +1,98 @@
+# Requirements Document
+
+## Introduction
+
+Today, a worker's job start/finish/fail report to the controller is a single fire-and-forget send (`XCECLI_FF_CALL`, `src/lib/client_internal.h`) with no retry and no redelivery. If that send is lost — because the worker briefly disconnects, or because the controller it was talking to fails over to a new primary mid-flight — the fact is gone forever: the job's true outcome is never recorded, and the job's state in the cluster stays wherever it last was (typically `RUN`) permanently, even though the worker's own OS process already exited. This was confirmed live (bug-1741): a burst of near-simultaneous job completions on one worker host lost 6 of 7 finish reports, leaving those jobs stuck in `RUN` forever while their siblings all completed normally, and no controller resync ever corrected them because nothing in the cluster still knew they had actually finished.
+
+This feature closes that gap for the specific case of a worker reconnecting to the controller (after a network disconnect, or after the controller it was talking to fails over and the worker rejoins the new primary): on every such rejoin, the controller asks the worker directly what really happened to the jobs the controller believes are running there, and reconciles the worker's answer into its own job records through the existing idempotent commit-then-apply path. This does not attempt to prevent the underlying send loss (a separate concern); it recovers from it after the fact, the next time the worker and controller re-establish contact.
+
+## Boundary Context
+
+- **In scope**: A worker reconnecting to the controller (network disconnect + reconnect, or reconnecting to a new primary after controller failover) triggers a job-state reconciliation exchange, initiated by the controller, covering jobs the controller currently believes are running on that specific worker host. The controller applies the worker's answer through the existing idempotent commit-then-apply path, with no duplicate facts and no clobbering of state the controller already knows is newer or more final.
+- **Out of scope**: Preventing the original notification loss (the fire-and-forget send itself is unchanged). Resyncing anything other than job execution facts on worker rejoin — apiserver rejoin content, scheduler rejoin content, and any other cluster/queue/job *list* state resync are separate, deferred concerns and not covered here. Any change to the controller's existing pending-facts queue (which already retries facts it received but could not commit while the worker stayed connected) — this feature complements that mechanism for the case where the fact never arrived at all, it does not modify it. Delta-based or session-token re-attach optimizations for rejoin in general. Coupling the recently-finished reporting window (Requirement 3, Acceptance Criterion 7) to commit-batch-window's commit-urgency tuning — the window's sizing is driven by disconnect/reconnect coverage, and any mitigation it happens to provide for batched-but-uncommitted facts is incidental, not a designed dependency; commit-batch-window remains a separately parked feature.
+- **Adjacent expectations**: This feature depends on the controller's existing commit-then-apply idempotency guards (already in place) to apply reconciled facts safely. It does not introduce a second, competing mechanism for deciding a job's state — reconciled facts are applied as ordinary job-state facts through that same path.
+- **Multi-host jobs**: the Controller dispatches and tracks a multi-host job as a single record owned by that job's head/primary host — it never dispatches separately to the job's other hosts. Job-state reconciliation for a multi-host job therefore happens through that same head host, consistent with how the job is already dispatched and tracked; no separate handling is needed for the job's other hosts. This is unrelated to bug-1741's own documented gap (per-host resource-usage/rusage attribution for a multi-host job's individual tasks), which this feature does not touch.
+
+## Requirements
+
+### Requirement 1: Reconciliation Trigger
+
+**Objective:** As the Controller, I want to start job-state reconciliation whenever a joining worker host has jobs I still believe are running on it, so that I don't need to reliably distinguish a worker's first-ever join from a reconnect.
+
+#### Acceptance Criteria
+1. When a worker host joins the cluster and the Controller currently records one or more non-terminal jobs assigned to that host, the Controller shall initiate job-state reconciliation for that worker as part of that join.
+2. When a worker host joins the cluster and the Controller currently records no non-terminal jobs assigned to that host, the Controller shall not initiate job-state reconciliation for that worker.
+3. The Controller shall apply this same trigger condition regardless of whether the join is the host's first-ever registration, a reconnect after a network disconnect, or a reconnect to a new primary after controller failover — the Controller shall not need to separately determine which of these a given join is.
+
+### Requirement 2: Controller-Initiated Job-State Query
+
+**Objective:** As the Controller, I want to tell a joining worker exactly which jobs I currently believe are running on it, so that the worker only needs to answer about jobs I have an open question about.
+
+#### Acceptance Criteria
+1. When job-state reconciliation is initiated for a worker (Requirement 1), the Controller shall send that worker a list of the jobs the Controller currently records as assigned to that worker host and not yet in a terminal state.
+2. The Controller shall identify each job in the query unambiguously (by the job's unique identifier) so the worker can match it to its own local job records.
+
+### Requirement 3: Worker Job-Status Report
+
+**Objective:** As the Controller, I want the worker to tell me the real, current outcome of every job I asked about, so that I can correct any job-state fact I lost while we were disconnected.
+
+#### Acceptance Criteria
+1. When a worker receives a job-state query from the Controller, the Worker shall report back, for every job identifier in the query, one of: still running, finished successfully, finished with a failure/exit code, or unrecognized (the worker has no record of ever running that job).
+2. The Worker shall report a status for every job identifier present in the query it received.
+3. If the worker cannot determine a job's true status (for example, the job's process state is inconclusive at the time of the query), the Worker shall report that job as unrecognized/indeterminate rather than guessing a definite outcome.
+4. If a job in the query has already finished before the rejoin completes, the Worker shall report the same finish outcome (success/failure and exit code) it would have reported through the ordinary job-finish notification.
+5. When the Worker reports a job as finished (successfully or with a failure), the Worker shall include the time the job actually finished, not the time the report itself is sent.
+6. The Worker shall also report any job it is currently running or has recently finished that was not included in the Controller's query, so the Controller can recover a job assignment its own records are missing.
+7. The Worker shall bound "recently finished" (Acceptance Criterion 6) to a configurable time window, exposed as a cluster configuration parameter, so operators can tune how long a completed job remains eligible for this extra reporting without unbounding the exchange.
+8. The Worker shall delay resuming its normal post-join operation (for example, periodic background reporting) until it has finished sending its complete job-state report (Acceptance Criteria 1-6) for the current join, so that the worker's other activity does not begin or interleave with the reconciliation exchange it is still completing.
+
+### Requirement 4: Controller Reconciliation
+
+**Objective:** As the Controller, I want to apply a worker's job-status report as authoritative, idempotent corrections to my own job records, so that a job's recorded state converges to the truth without duplicating or corrupting existing facts.
+
+#### Acceptance Criteria
+1. When the Controller receives a worker's job-status report, the Controller shall apply each reported job outcome through the same idempotent commit-then-apply path used for ordinary job-state facts.
+2. When the Controller reconciles a job's finish outcome, the Controller shall record the finish time the Worker reported (Requirement 3, Acceptance Criterion 5) as the job's finish time, rather than the time the Controller processed the reconciliation.
+3. If a reported job outcome would duplicate a fact the Controller has already committed for that job, the Controller shall apply it as a no-op rather than recording a duplicate.
+4. If the Controller has already recorded a more final or newer state for a job than the worker's report implies (for example, the Controller already recorded the job as finished by another means), the Controller shall not overwrite that state with a less final or older report.
+5. When a worker reports a queried job as unrecognized, the Controller shall treat this as evidence the job's true outcome could not be confirmed by the worker and shall not fabricate a finish outcome for it.
+6. When a worker reports a queried job as unrecognized, the Controller shall return that job to pending state, making it eligible for the normal scheduling path to assign it again, rather than leaving it permanently in its prior non-terminal state.
+7. When a worker reports a job that was not part of the Controller's query (Requirement 3, Acceptance Criterion 6), the Controller shall reconcile that job's state through the same idempotent commit-then-apply path and precedence rules used for queried jobs (Acceptance Criteria 1, 3, 4), rather than discarding the report.
+8. When the Controller reconciles a job reported outside its original query and the Controller has no existing non-terminal record of that job assigned elsewhere, the Controller shall record the job as assigned to the reporting worker, matching what the worker reported.
+
+### Requirement 5: Dispatch Gating During Resync
+
+**Objective:** As the Controller, I want to withhold new job dispatch to a rejoining worker until its job-state reconciliation completes, so that I never assign new work to a worker whose true resource usage I don't yet know.
+
+#### Acceptance Criteria
+1. While a worker's job-state reconciliation is outstanding, the Controller shall not dispatch new jobs to that worker.
+2. When a worker's job-state reconciliation completes (all queried jobs have a reported and reconciled outcome), the Controller shall make that worker eligible for new job dispatch again.
+3. If a worker's job-state report does not arrive within a bounded time after the query is sent, the Controller shall resolve the outstanding reconciliation (accepting the query as answered, timed out, or reattempted on a subsequent rejoin) rather than withholding that worker from dispatch indefinitely.
+4. If the Controller attempts to dispatch a job to a worker that is not currently connected, the Controller shall fail that dispatch attempt immediately and return the job to pending state, making it eligible for the normal scheduling path to assign it again, rather than holding the dispatch for delivery once the worker reconnects.
+
+### Requirement 6: Resync Failure and Repeated Rejoin
+
+**Objective:** As the Controller, I want repeated or interrupted rejoin attempts to stay safe, so that a worker that keeps disconnecting during reconciliation never leaves job state permanently ambiguous or duplicated.
+
+#### Acceptance Criteria
+1. If a worker disconnects again before its job-state report is fully processed, the Controller shall discard the incomplete reconciliation and shall re-initiate job-state reconciliation on the worker's next rejoin.
+2. The Controller shall produce the same reconciled result whether a worker's job-state report is applied once or is, due to a retry, presented to the Controller more than once for the same rejoin.
+3. If a worker's job-state report arrives for a reconciliation round the Controller has already discarded or superseded with a newer one (Acceptance Criterion 1), the Controller shall discard that report rather than apply it.
+
+### Requirement 7: Scheduler Notification of Reconciled State
+
+**Objective:** As the Controller, I want every job-state change produced by reconciliation to reach the Scheduler, so that the Scheduler's view of a job converges to the same truth as the Controller's, not just the Controller's own job table.
+
+#### Acceptance Criteria
+1. When reconciliation changes a job's recorded state, the Controller shall notify the Scheduler of that change through the same notification path used for an ordinary (non-reconciliation) job-state change.
+2. When reconciliation returns a job from running back to pending state (Requirement 4, Requirement 5), the Controller shall notify the Scheduler of the job's new pending state, so the Scheduler makes the job available for scheduling again rather than continuing to treat it as running.
+3. The Controller shall not consider reconciliation of a job complete until the resulting state change, if any, has been handed to the same notification path the Scheduler already relies on for ordinary job-state changes.
+
+### Requirement 8: Bounded Job-State Exchange Size
+
+**Objective:** As an Operator, I want job-state reconciliation to complete reliably even for a worker running many jobs at once, so that a busy host is never unable to rejoin the cluster simply because it has a lot of jobs to reconcile.
+
+#### Acceptance Criteria
+1. The Controller shall be able to query and reconcile job state for a worker regardless of how many non-terminal jobs are currently assigned to that worker, without the reconciliation exchange failing solely because of the number of jobs involved.
+2. The Worker shall be able to report status for every job in the Controller's query, and any additional jobs it is running (Requirement 3, Acceptance Criterion 6), without the report failing solely because of the number of jobs involved.
+3. Where the number of jobs to reconcile for a worker exceeds what the Controller and Worker exchange in a single message, the exchange shall complete correctly across multiple messages rather than truncating the list or silently dropping jobs beyond a fixed limit.

.kiro/specs/worker-rejoin-resync/research.md

@@ -0,0 +1,113 @@
+# Research & Design Decisions: worker-rejoin-resync
+
+## Summary
+- **Feature**: `worker-rejoin-resync`
+- **Discovery Scope**: Extension (existing system) — integration-focused, cross-cutting across controller, worker, and scheduler around an already-implemented join/rejoin path and an already-implemented commit-then-apply seam.
+- **Key Findings**:
+  - `rejoin_worker`/`apply_join_state` (worker.c, from `worker-join-refactor`) is the exact moment a host becomes "connected" again — the natural trigger point for Requirement 1, and a new `worker_t` status bit here gates new dispatch "for free" through the *existing* `dispatch_a_job_` connected-state check with zero changes to that function.
+  - The job table's idempotent commit-then-apply split (`record_job_*_fact` / `apply_job_*_transition`, with `job_state_is_terminal`/`job_state_is_backward` guards) is the exact dedup mechanism the requirements' Boundary Context says this feature must reuse, not duplicate. Three of four reconciliation outcomes (finished-success, finished-failure, still-running) already have a fully-idempotent home; only "unrecognized → return to pending" and "unqueried job → assign" are genuinely new fact shapes.
+  - The controller→scheduler notification path (Requirement 7) is a hook → notify → wire → dequeue pipeline (`XCE_CONTROLLER_HOOK_JOB` → `sched.c` hook callbacks → `notify()` → scheduler's `handler.c` inbound switch → `sched_job_state_change()`), and it is **strictly separate** from the scheduler's own outbound decision-publish path (`scheduler.c`/`driver.c`), which a standing text-scan test (`test_no_worker_resync_symbols_in_scheduler_publish_path`) guards against ever learning about "resync" concepts. This fixes the correct integration seam: `handler.c`'s inbound switch only.
+  - The worker has **no existing retention** of finished-job state — `cleanup_worker_list()` reaps and deletes an exited job's record (memory + sqlite row) on the very next cleanup pass, with no `finish_time` field anywhere in the worker's job structs today. New short-lived retention is required, not a rewiring of an existing cache.
+  - `xce_reply_get()`/`xce_push_data_req()` (`{pos, count, n_total}` chunking, used today for joblist/workerlist/reslist replies) is the established bounded/paginated multi-message idiom in this codebase and is the direct template for Requirement 8, not the heavier WAL catch-up/replication protocol and not `XCECLI_FF_CALL` (whose whole point is "loss is fine" — the opposite of what this feature needs).
+  - `config-design.md`'s rule set classifies the new "recently finished" window (Requirement 3.7) the same way `heartbeatInterval`/`inactiveHeartbeatCount` were classified: passes the independence rule (no cross-node correctness dependency) but is kept in shared config anyway via the documented operational-uniformity override, for the same "predictable cluster-wide behavior" reason.
+
+## Research Log
+
+### Controller-side join/rejoin flow and job state machine
+- **Context**: Where does reconciliation trigger, and how does it reuse the existing idempotent fact/apply path?
+- **Sources Consulted**: `src/controller/worker.c` (`rejoin_worker:403`, `apply_join_state:321`, `on_worker_join:623`), `src/controller/join.c` (`handle_worker_join:48`), `src/controller/job.c` (`record_job_finish_fact:2626`, `record_job_fail_fact:2781`, `record_job_postdone_fact:2921`, `apply_job_update_transition:2808`, `dispatch_a_job_:1278`), `src/controller/persist.c` (`apply_committed_entry:4333`), `src/controller/commit.c` (`commit_submit:485`), `src/include/xce/types.h` (`worker_t:284`, status bits `170-184`), `.kiro/specs/worker-join-refactor/design.md`.
+- **Findings**:
+  - `handle_worker_join` durably commits the join (`persist_commit_barrier`) before `on_worker_join` runs — a second, later plug-in point after the join is durable.
+  - `worker_t.host.status` is a bitmask; `dispatch_a_job_` (job.c:1511-1512) already refuses dispatch when `w->ep == NULL || w->host.status != 0`. A new `XCE_WORKER_RECONCILING`-style bit therefore gates Requirement 5.1 with no change to `dispatch_a_job_` itself.
+  - `record_job_finish_fact`/`apply_job_finish_transition` already sets `finish_time` at receipt time (bug-1689), matching Requirement 3.5/4.2 exactly — the finished-job reconciliation case is a straight reuse.
+  - `JOB_STAT_PEND` is otherwise only ever set at job creation — there is no existing "return a RUN job to PEND" transition. Since RUN→PEND is non-terminal→non-terminal, the existing `job_state_is_backward` guard does not block it; a thin new fact wrapper routed through the same `WAL_JOB_UPDATE`/`apply_job_update_transition` seam is the correct reuse (not a new state-machine core).
+  - No existing "assign a job to a worker the controller didn't previously have a record for" fact exists; the closest analog is `dispatch_a_job_`'s own assignment logic (job.c:1686-1734).
+  - The pending-facts queue is not a separate structure — it *is* the commit engine (`commit_submit`, `COMMIT_MODE_RETAIN`) plus `apply_committed_entry`'s guards. Reconciliation must submit through this exact seam, never a parallel one (this is also stated explicitly in the requirements' own Boundary Context).
+- **Implications**: Reconciliation needs exactly two new fact-recording functions in `job.c` (reconcile-to-pending, reconcile-assign) plus straight reuse of the three existing ones (finish, fail, no-op-for-running).
+
+### Worker-side job bookkeeping and wire protocol
+- **Context**: What can the worker actually report, and how should the query/report exchange ride the wire?
+- **Sources Consulted**: `src/worker/include/xcecd_worker_utils.h`, `src/worker/src/xcecd_worker_utils.c` (`cleanup_worker_list:385-404`), `src/worker/include/xcecd_utils.h`, `src/lib/request.c` (`xce_join_req:1398-1429`, `xce_reply_get:1627-1784`, `xce_push_data_req:1491-1610`), `src/schema/controller.capnp` (`JoinData:155-161`, `HostAttrs:126-135`), `src/lib/client_internal.h` (`XCECLI_FF_CALL:290-302`), `src/lib/client.c`.
+- **Findings**:
+  - `xcecd_worker_t` (PID-keyed) and `xcecd_job_t` carry no timestamp field at all; `cleanup_worker_list()` deletes both the in-memory entry and its sqlite row essentially immediately after exit — there is no retention window today.
+  - `JoinData`'s capnp union has exactly two arms (`nodata`, `attrs`) with no job-state content, confirming the brief's own claim. Cramming a paginated exchange inline into `JoinData` would force it through `xce_join()`'s single-shot synchronous `cbuf_timedget` call shape, which cannot paginate — a new follow-up opcode/message pair after the join reply is the right shape, not a `JoinData` union extension.
+  - `xce_reply_get`/`xce_push_data_req`'s `{pos, count, n_total}` chunking (already used for joblist/workerlist/reslist) is the established, directly-reusable bounded-exchange template for Requirement 8.
+  - `XCECLI_FF_CALL`'s own header comment explicitly names "worker-rejoin-resync" as the intended fix for the loss it tolerates — confirming this feature's problem statement directly in code, not just in the bug report.
+- **Implications**: New capnp opcode pair (controller→worker query, worker→controller report), both using the existing pagination shape; new worker-side finished-job retention structure with a timestamp field, bounded by the Requirement 3.7 window.
+
+### Scheduler notification path and scope-fence test
+- **Context**: Requirement 7 requires reconciliation-driven job-state changes to reach the scheduler through "the same path" as an ordinary change — what is that path, concretely, and are there any standing tests that constrain where new code may live?
+- **Sources Consulted**: `src/controller/job.c` (hook call sites `HOOK_DETAIL_JOB_EXIT:2641`, `HOOK_DETAIL_JOB_FAIL:2745`), `src/controller/sched.c` (`sched_job_hook_fun:547`, `sched_job_finish_fun:392`, `sched_job_fail_fun:438`, `sched_job_update_fun:468`), `src/controller/dataex.c` (`notify:399`), `src/controller/job.c` (`notify_dispatch:1971`), `src/scheduler/src/handler.c` (`sched_handle_notify:525`, `sched_job_state_change:407`), `src/scheduler/testsuite/test_scheduler_decision_redelivery.c:424-456`.
+- **Findings**:
+  - Job-state-change code fires `XCE_CONTROLLER_HOOK_JOB` (not `HOOK_WORKER` — that category is for worker join/dead events); `sched.c`'s registered hook callback dispatches by `HOOK_DETAIL_*` to one of three functions, each of which calls the shared `notify()` with a distinct `NotifyData_data_sched*` event.
+  - `notify_dispatch()`/`NotifyData_data_jobDispatchFail` is an existing, direct precedent for "tell the scheduler this job is back to pending" — its handler in `handler.c` already calls `sched_job_state_change(sched, jobid, JOB_STAT_PEND)`.
+  - `test_no_worker_resync_symbols_in_scheduler_publish_path` is a static text-scan (`strstr` for `"resync"`) over `scheduler.c` and `driver.c` — the scheduler's own **outbound** decision-publish direction. It is a deliberate scope-fence: that direction must never learn about worker-resync concepts. It does not cover `handler.c` (the **inbound** Controller→Scheduler direction), which is unambiguously the correct, and only permitted, integration seam.
+  - The async-I/O steering doc's thread-role contract applies identically to the scheduler; a bounded reconciliation wait must follow the vtask + admission-reservation pattern (`commit.c`'s `COMMIT_WAIT_RESERVED`, `wait_cap`/`wait_admit_locked`) rather than any raw blocking wait on iomgr/delaymgr, with the timeout itself fired by a delaymgr trigger.
+- **Implications**: Reconciliation's finish/fail cases need zero new scheduler-facing code (they already drive `schedExitJob`/`schedFailJob`). The reconcile-to-pending case should reuse the existing `jobDispatchFail`-style `sched_job_state_change(..., JOB_STAT_PEND)` call shape (either by literally reusing that notify event or adding a same-shaped one) — never by adding anything to `scheduler.c`/`driver.c`.
+
+## Architecture Pattern Evaluation
+
+| Option | Description | Strengths | Risks / Limitations | Notes |
+|--------|-------------|-----------|---------------------|-------|
+| Inline extension of `JoinData` union | Add job-state content as a third `JoinData` union arm | Minimal new opcode surface | Forces the exchange through `xce_join()`'s single-shot synchronous reply — cannot paginate (Requirement 8) | Rejected |
+| New opcode pair, ad hoc payload shape | New `OPCode_jobStateQuery`/`OPCode_jobStateReport`, custom framing | Full control over shape | Reinvents pagination the codebase already solved | Rejected |
+| New opcode pair reusing `{pos,count,n_total}` pagination | New opcode pair, payload shape mirrors `JobData`/`WorkerData`/`ResourceVals` | Directly reuses an established, tested idiom (`xce_reply_get`/`xce_push_data_req`); minimal new protocol concepts | None significant | **Selected** |
+| Reuse WAL catch-up/replication streaming protocol | Model the exchange on `replEntry`/`walCatchupEntry` | Already resumable/ack-per-entry | Heavier machinery designed for log replication, wrong semantic fit for a bounded one-time query/report | Rejected |
+
+## Design Decisions
+
+### Decision: New opcode pair for the job-state query/report exchange, following the `xce_reply_get`/`xce_push_data_req` pagination template
+- **Context**: Requirements 2, 3, and 8 need a controller-initiated, bounded/paginated, bidirectional exchange that today's join message cannot carry.
+- **Alternatives Considered**: inline `JoinData` extension; ad hoc new framing; WAL-catch-up-style streaming (see table above).
+- **Selected Approach**: Two new opcodes — `OPCode_jobStateQuery` (controller→worker) and `OPCode_jobStateReport` (worker→controller) — both paginated with the existing `{pos, count, n_total}` shape, issued as a follow-up exchange after the join reply (not inline in `JoinData`).
+- **Rationale**: Reuses a proven, already-tested idiom instead of inventing a new one; keeps the join message itself unchanged (no regression risk to `worker-join-refactor`'s existing behavior, per its own Boundary Commitments).
+- **Trade-offs**: One more opcode pair to add to `ha_dispatch_gate`'s role tables and `myreq_handler`'s switch, versus a cleaner, template-following protocol.
+- **Follow-up**: Confirm at implementation time whether `OPCode_jobStateReport` needs an explicit redirect-reply path if ever received by a non-primary (per `api-standards.md`'s "Adding a New Opcode" step 6 / bug-1700 precedent) — expected answer is no, since workers only ever connect to the current primary, but this must be verified, not assumed.
+
+### Decision: Reconciliation-outstanding state lives on `worker_t` as a status bit + a small tracking record, not a new global registry
+- **Context**: Requirement 5 needs to gate new dispatch per-worker while reconciliation is outstanding, and Requirement 6 needs "discard on disconnect, re-trigger on next rejoin" semantics.
+- **Alternatives Considered**: a separate global map keyed by worker; per-worker state (selected).
+- **Selected Approach**: A new `XCE_WORKER_RECONCILING` status bit (reusing the existing `w->host.status != 0` dispatch check "for free") plus a small struct on `worker_t` holding the queried job-ID set, a round/generation token (for Requirement 6.3's late-report discard), and the timeout deadline.
+- **Rationale**: `worker_t` is already the natural per-host session object (confirmed by `worker-join-refactor`'s own design); a new global structure would duplicate lifecycle management the existing object already handles (creation, teardown, locking).
+- **Trade-offs**: None significant — this is the minimal-surface option.
+- **Follow-up**: None.
+
+### Decision: Reconcile-to-pending and reconcile-assign are new, narrow fact-recording functions in `job.c`, not a new state-machine core
+- **Context**: Requirement 4.6 (unrecognized → pending) and Requirement 4.8 (unqueried job → assign) have no existing idempotent fact path.
+- **Alternatives Considered**: a new, separate reconciliation-specific mutation path; thin wrappers over the existing `WAL_JOB_UPDATE`/dispatch-assignment seams (selected).
+- **Selected Approach**: `record_job_reconcile_pending_fact()` submits through the existing `WAL_JOB_UPDATE`/`apply_job_update_transition` seam with target state `PEND` (same shape as any other job-state-update fact, so it drives the existing `HOOK_DETAIL_JOB_UPDATE` → scheduler-notify path for free). `record_job_reconcile_assign_fact()` mirrors `dispatch_a_job_`'s own assignment logic for host/state but does **not** attempt to reconstruct the job's original resource allocation (see Risks & Mitigations).
+- **Rationale**: The requirements' Boundary Context and Constraints explicitly forbid a second, competing idempotency mechanism; both new functions are additive call sites into the existing guarded seam, not new guards.
+- **Trade-offs**: The reconcile-assign case inherits a real, pre-existing limitation (below) rather than solving it.
+- **Follow-up**: None — the limitation is out of this spec's boundary by design.
+
+### Decision: The "recently finished" reporting window (Requirement 3.7) is shared cluster config, via the operational-uniformity override
+- **Context**: Requirement 3.7 asks for a bounded, cluster-configurable window; `config-design.md` governs where any such field belongs.
+- **Alternatives Considered**: per-node local config (passes the independence rule cleanly — no cross-node correctness dependency); shared config via override (selected).
+- **Selected Approach**: Add the window (and, for the same reason, the Requirement 5.3 reconciliation timeout) as new fields on the existing `ClusterConfig` capnp blob, alongside `heartbeatInterval`/`inactiveHeartbeatCount`.
+- **Rationale**: `config-design.md`'s own worked example classifies the structurally identical `heartbeatInterval`/`inactiveHeartbeatCount` fields (controller→worker liveness, not HA/Raft election heartbeat) as shared-via-override for exactly this reason: no correctness invariant demands cluster-wide agreement, but operators want uniform, predictable behavior rather than per-node drift. This feature's window is the same shape of parameter in the same domain (worker liveness/reconnection).
+- **Trade-offs**: A config push is required to change it (heavier than a per-node flag), accepted for the same reason the precedent field accepted it.
+- **Follow-up**: None.
+
+### Decision: Config push and the job-state query are sent in a fixed order (config first) with no new ack — ordering is guaranteed by the single connection, not invented
+- **Context**: `on_worker_join` (worker.c:623) must trigger both the existing config push and the new job-state query for a rejoining worker. Does the worker need to fully apply config before it can correctly answer the query, and if so, how is that ordering enforced?
+- **Alternatives Considered**: (1) have the controller wait for an explicit worker-sent config-applied ack before sending the query (initially adopted, then retracted); (2) rely on the connection's existing ordering guarantees (selected).
+- **Selected Approach**: `on_worker_join` sends config push (unchanged `worker_push_config` call), then immediately sends the new job-state query — both fire-and-forget, no new ack/reply added.
+- **Rationale**: Initially assumed the worker might dispatch incoming controller requests concurrently across its `thread_pool_t`, which would make arrival order insufficient to guarantee handling order — this assumption was wrong. Reading `controller_connector.c:1563`'s `myreq_handler` directly confirmed it is a plain synchronous `switch`, invoked one message at a time off the worker's single long-lived client connection (`xcecli_ctx`); the worker's `thread_pool_t` is scoped to job-forking/job-control work only, not request dispatch. On one connection, TCP ordering + this synchronous single-message dispatch together guarantee config push is *handled to completion* before the job-state-query handler even begins — not just sent first. No config content is actually needed to answer the query correctly either way (job IDs are globally unique, independent of host config), so this was a belt-and-suspenders concern, not a load-bearing correctness dependency — but the ordering guarantee turned out to be free, so it's stated explicitly rather than left implicit.
+- **Trade-offs**: None — this is strictly simpler than the retracted ack-based alternative, with no loss of guarantee.
+- **Follow-up**: If the worker's request-dispatch model is ever changed to fan out across a thread pool (unlike today), this ordering guarantee must be re-derived — it is not inherent to the wire protocol, only to the current single-threaded-dispatch implementation.
+
+## Risks & Mitigations
+- **Risk**: A reconciliation-assigned job (Requirement 4.8) has no recoverable original resource allocation (`job->allocs`) — the controller never held a dispatch record for it, so there is nothing to read the allocation back from. — **Mitigation**: Document as a deliberate, bounded scope limitation (see `design.md`'s Data Models persistence-completeness note): the job's *state* and *host* are corrected; its resource *reservation* accounting starts from zero/unknown, converging via the existing `resource-accounting.md` "charged = max(reserved, actual)" policy on the job's next rusage tick — identical to how any legitimately-unreserved job is already handled today, not a new class of gap.
+- **Risk**: A new WAL fact type (reconcile-assign) must be decoded/adopted identically by both apply seams (`apply_committed_entry` primary-boot-replay and the standby catch-up apply path) or it reproduces the exact bug-1678 field-completeness failure mode. — **Mitigation**: `design.md` calls this out explicitly as a persistence-completeness checklist item, per `design-principles.md`'s Persistence Review requirement.
+- **Risk**: A blocking wait for the worker's report, implemented carelessly, could stall a liveness-critical thread (iomgr/delaymgr). — **Mitigation**: Reuse `commit.c`'s `COMMIT_WAIT_RESERVED` admission-reservation pattern for any bounded wait; drive the timeout via a delaymgr-scheduled trigger, never a blocking sleep.
+- **Risk**: The requirements' Boundary Context explicitly disclaims coupling the Requirement 3.7 window to `commit-batch-window`'s (separately parked) commit-urgency tuning. — **Mitigation**: `design.md` does not reference or depend on `commit-batch-window` in any component contract.
+
+## References
+- `.kiro/specs/worker-rejoin-resync/requirements.md` — approved-for-design requirements (8 requirement groups, 30 acceptance criteria)
+- `.kiro/specs/worker-rejoin-resync/brief.md` — original discovery brief with file:line evidence
+- `.kiro/specs/worker-join-refactor/design.md` — `rejoin_worker`/`apply_join_state` contract this feature extends
+- `.kiro/steering/config-design.md` — shared-vs-local config classification rule, applied to Requirement 3.7
+- `.kiro/steering/resource-accounting.md` — reserved/used accounting model the reconcile-assign limitation converges into
+- `.kiro/steering/persistence.md` — `APPLY_DIVERGENT`/bug-1678 per-field completeness precedent
+- `.kiro/steering/api-standards.md` — "Adding a New Opcode" checklist (bug-1700 redirect-reply precedent)
+- `.kiro/steering/ha-async-io.md` — thread-role contract for the bounded reconciliation wait

.kiro/specs/worker-rejoin-resync/tasks.md

@@ -0,0 +1,191 @@
+# Implementation Plan
+
+- [x] 1. Foundation: wire protocol schema and shared reconciliation types
+- [x] 1.1 Define the wire protocol for the reconciliation exchange
+  - Add the controller→worker job-state query and worker→controller job-state report as new opcodes, each carrying a paginated list of per-job entries (job ID, status, exit code, worker-reported finish time, and whether the job was part of the original query)
+  - Add the two new cluster-wide configuration fields: the recently-finished reporting window and the reconciliation timeout
+  - Observable completion: the schema builds cleanly and the new message/opcode/config-field definitions are visible in the generated bindings
+  - _Requirements: 2.1, 2.2, 3.5, 3.7, 5.3, 8.1, 8.2, 8.3_
+
+- [x] 1.2 (P) Add a worker reconciling status and round-tracking state
+  - Add a new worker status value meaning "job-state reconciliation outstanding for this host"
+  - Add a per-worker reconciliation-round structure capable of holding a round identifier, the set of queried job IDs, and a deadline
+  - Observable completion: the new status value and round structure compile and are reachable from the worker record, unused by any caller yet
+  - _Requirements: 5.1, 6.3_
+  - _Boundary: shared types_
+
+- [x] 1.3 (P) Establish worker-side C test infrastructure
+  - Add the first test-runner harness for the worker daemon, mirroring the controller's existing runner conventions
+  - Observable completion: an empty/placeholder worker-side test program builds and runs successfully via the existing test build target
+  - _Requirements: 3.1, 3.2, 3.3_
+  - _Boundary: worker test infrastructure_
+  - _Note: satisfied by pre-existing infrastructure — see Implementation Notes._
+
+- [x] 1.4 (P) Add wire encode/decode helpers for the reconciliation exchange
+  - Add request/reply encode and decode helpers for the new query and report messages, following the existing paginated-list pattern already used for job/worker/resource listings
+  - Observable completion: a round-trip encode-then-decode of a query and of a report (including an empty page and a multi-page case) reproduces the original data exactly
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Boundary: wire codec_
+  - _Depends: 1.1_
+
+- [x] 1.5 (P) Wire the new config fields into xceadmin's config-build pipeline
+  - Add the two new cluster-config fields (the recently-finished reporting window, the reconciliation timeout) to xceadmin's YAML validation schema and its YAML-to-binary field mapping, following the existing `heartbeatInterval`/`inactiveHeartbeatCount` pattern exactly, so an operator can actually set them through `xceadmin config build` — added to the wire schema by task 1.1, but with no configuration path into it until this task
+  - Regenerate the checked-in golden config-build fixtures to match the wire schema's current field set (task 1.1 grew it; the fixtures predate that and no longer match a fresh build)
+  - Observable completion: a YAML config supplying values for both new fields produces a binary that decodes back to those exact values, and the existing golden-fixture regression tests pass again
+  - _Requirements: 3.7, 5.3_
+  - _Boundary: xceadmin config-build_
+  - _Depends: 1.1_
+
+- [x] 2. Core: controller job-table reconciliation outcomes
+- [x] 2.1 Return an unrecognized job to pending state, releasing its resource allocation
+  - When a worker reports a queried job as unrecognized, transition that job back to pending using the existing state-transition guard so a terminal or already-more-final job is never overwritten
+  - Release any resource allocation the job was carrying, so a job returned to pending never carries a stale allocation from a prior dispatch into its next dispatch attempt
+  - Durably record the transition and notify the scheduler of the new pending state through the existing notification path, without waiting for a later replay to discover the change
+  - Observable completion: a job previously dispatched (real allocation committed) that reconciles to unrecognized ends the call with pending state, no resource allocation, and the scheduler notified — verified before any commit round-trip completes
+  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 7.1, 7.2, 7.3_
+  - _Boundary: job.c additions_
+
+- [x] 2.2 Assign an unqueried job to the worker that reported it
+  - When a worker reports a job the controller has no existing non-terminal record of, record that job as assigned to the reporting worker and host, matching the worker's reported status, through the same idempotent path used for a normal dispatch
+  - Do not fabricate a resource allocation for this job — leave the allocation unset, consistent with the controller genuinely not knowing what the scheduler originally computed
+  - Notify the scheduler of the new assignment through the existing notification path
+  - Observable completion: a job the controller never previously tracked, reported running by a worker, ends the call recorded as running on that host with the scheduler notified, and durably survives a restart replay with the same result
+  - _Requirements: 4.1, 4.3, 4.4, 4.7, 4.8, 7.1, 7.2, 7.3_
+  - _Boundary: job.c additions_
+
+- [x] 2.3 Fail a dispatch to a disconnected worker immediately, without retry or job failure
+  - When the controller attempts to dispatch a job to a worker it is not currently connected to, fail that specific dispatch attempt right away and return the job to pending, reusing the pending-and-release behavior from task 2.1, rather than retrying the attempt or exhausting a retry count toward permanent failure
+  - Leave the existing behavior for every other reason a worker might not be ready (locked, busy, closed, or mid-reconciliation) completely unchanged — confirm those cases still retry and eventually fail exactly as before
+  - Observable completion: dispatching to a disconnected worker returns the job to pending on the first attempt with the job's failure-retry count untouched and no permanent failure recorded, while dispatching to a worker that is connected but otherwise not ready still follows the pre-existing retry-then-fail behavior unchanged
+  - _Requirements: 5.1, 5.4_
+  - _Boundary: job.c additions_
+  - _Depends: 2.1_
+
+- [x] 3. Core: controller reconciliation-round bookkeeping
+- [x] 3.1 (P) Track one reconciliation round per worker with a bounded timeout
+  - Given a worker host with one or more non-terminal jobs currently assigned to it, construct the exact list of those jobs' unique IDs as the round's query set
+  - Track a round identifier that distinguishes the current reconciliation attempt from any prior one for the same worker
+  - Schedule a bounded timeout for the round using a non-blocking trigger; when it fires before the round otherwise resolves, resolve the round rather than leaving the worker withheld from dispatch indefinitely
+  - Observable completion: arming a round for a worker with known non-terminal jobs produces a query set matching exactly those job IDs, a fresh round identifier, and an active timeout; letting the timeout elapse resolves the round without any blocking call
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 5.3, 6.1_
+  - _Boundary: resync.c_
+
+- [x] 3.2 Apply each reported job outcome through the correct idempotent path
+  - For each entry in a worker's job-state report: a still-running job is cleared from the round's outstanding set with no fact recorded; a finished-success or finished-failure job is applied through the existing finish/fail recording used for an ordinary report; an unrecognized queried job is applied through task 2.1's pending-and-release path; a reported job outside the original query is applied through task 2.2's assignment path
+  - Discard a report that arrives for a round already superseded by a newer one for the same worker, rather than applying it
+  - When the round's outstanding set becomes empty, resolve the round so the worker becomes eligible for dispatch again
+  - Applying the same report twice for the same round must produce the same result as applying it once
+  - Observable completion: a full report covering all five outcome kinds leaves the job table in the correct state for each, a duplicate re-application of the same report changes nothing further, and a report carrying a stale round identifier is dropped without touching the job table
+  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 6.1, 6.2, 6.3_
+  - _Boundary: resync.c_
+  - _Depends: 2.1, 2.2, 3.1_
+
+- [x] 4. Core: worker-side job-state reporting
+- [x] 4.1 (P) Retain a job's true outcome for a bounded window after it exits
+  - At the moment a job's exit is detected, before its record would otherwise be discarded, retain its exit status, exit code, and true finish time for a configurable bounded window
+  - Observable completion: a job's outcome remains queryable for the configured window after it exits and is no longer available once the window elapses
+  - _Requirements: 3.4, 3.5, 3.7_
+  - _Boundary: worker retention_
+
+- [x] 4.2 Answer the controller's job-state query, including extra unqueried jobs
+  - For every job ID in the controller's query, report exactly one outcome: still running (from the live running list), finished successfully or with failure and its true finish time (from the new retention), or unrecognized if found in neither — every queried ID gets an answer, none are silently skipped, and an indeterminate case is reported as unrecognized rather than guessed
+  - After answering the query, also report any job the worker is currently running or has recently finished that was not part of the query
+  - Send both the query answers and the extra jobs in paginated reports, correct regardless of how many jobs are involved
+  - Observable completion: a query covering a mix of running, finished, and unknown job IDs produces one correctly-classified entry per ID plus a separate page of extras for jobs outside the query, with no job silently dropped even when the total exceeds one page
+  - _Requirements: 3.1, 3.2, 3.3, 3.6, 8.2, 8.3_
+  - _Boundary: xcecd_resync.c_
+
+- [x] 4.3 Wire the worker's receipt of a job-state query and delay resuming normal post-join operation until the report has fully sent
+  - Also close a gap task 4.2 correctly left out of its own boundary: nothing today wires the worker's actual receipt of `OPCode_jobStateQuery` (`src/worker/src/controller_connector.c`'s `myreq_handler` switch has no case for it, confirmed by inspection — only `OPCode_startjob`/`OPCode_runtasks`/`OPCode_jobcontrol`/`OPCode_fetchfile` exist there today) to task 4.2's `xcecd_resync_answer_query()`. Add that dispatch case: reassemble an incoming query that itself arrives across multiple pages (Requirement 8.1/8.3) into the jobids array `xcecd_resync_answer_query()` expects, then call it
+  - Set a local gate the moment a job-state query is received; keep it set across every page of the report; clear it only once the full report, including any extra-job pages, has finished sending
+  - Make whatever the worker does next after rejoining (resuming its own periodic background reporting) check this gate first, so that activity never begins or interleaves while the report is still being sent
+  - Observable completion: the worker's post-join activity is observed to start only after the last report page has been sent, never before, across a report spanning multiple pages; a job-state query arriving from the controller (single- or multi-page) reaches `xcecd_resync_answer_query()` and produces a real reply, not just a directly-called-in-tests function
+  - _Requirements: 3.8, 8.1, 8.3_
+  - _Boundary: xcecd_resync.c_
+  - _Depends: 4.2_
+
+- [x] 4.4 Gate the worker's other periodic background reporting on the same resync_pending flag
+  - Close a Requirement 3.8 gap found by feature-level validation, missed by task 4.3's own boundary: `checking_connection()`'s `rejoin_call()` is not the worker daemon's only periodic post-join activity — `src/worker/src/resource_sync.c`'s `resource_sync_thread()` is a separate pthread, started once at daemon init (unrelated to any join event), that periodically (every `LOAD_INDICES_INTERVAL_DEFAULT`, 15s) sends live `xcecd_send_load_indices()`/`xcecli_update_resvals()` RPCs to the controller over the same connection used for the resync exchange, with zero reference to `resync_pending`/`xcecd_resync_is_pending()` anywhere in that file — confirmed by inspection
+  - Make the periodic load-indices send check `xcecd_resync_is_pending()` first, the same way `checking_connection()`'s `rejoin_call()` already does (task 4.3), and skip/defer that cycle's send while a reconciliation report is still in flight — the send is already periodic, so a skipped cycle is naturally retried on the next tick, no new retry/queueing mechanism needed
+  - Observable completion: a load-indices send that would otherwise fire while `resync_pending` is set is observed not to fire during that window, and resumes firing on its normal periodic cadence once the flag clears; the existing periodic cadence and behavior when no reconciliation is in flight are unchanged
+  - _Requirements: 3.8_
+  - _Boundary: resource_sync.c_
+  - _Depends: 4.3_
+
+- [x] 5. (P) Core: exclude a reconciling host from scheduler dispatch candidacy
+  - Extend the scheduler's existing candidate-host filter so a host currently undergoing job-state reconciliation is excluded from dispatch candidacy the same way an already-unavailable host is
+  - When the reconciling status clears, the same filter must stop excluding that host without any separate action
+  - Introduces no new reconciliation-protocol name or concept into the scheduler's outbound decision-publish code — this is an ordinary status-bit read alongside the existing unavailable-host check, not a new symbol; the standing regression test guarding that publish path against ever referencing reconciliation concepts must stay green, unmodified
+  - Observable completion: a host flagged as reconciling is never selected as a dispatch candidate while the flag is set, and becomes selectable again as soon as it clears, with no change to how any other exclusion reason behaves, and the publish-path symbol-guard test still passes unmodified
+  - _Requirements: 5.1, 5.2_
+  - _Boundary: scheduler candidate filter_
+
+- [x] 6. Integration: cross-daemon wiring
+- [x] 6.1 Trigger reconciliation on worker rejoin and send the query
+  - At the moment a worker host becomes connected again (first join, network reconnect, or reconnect after controller failover, treated identically), check whether the controller currently holds non-terminal jobs for that host and, if so, arm a reconciliation round for it
+  - Send the round's query to the worker as a follow-up exchange after the join reply and the existing configuration push have both already been sent, not as an extension of the join call itself
+  - Also close a gap task 4.1 discovered and correctly left out of its own boundary: the worker daemon has no delivery path at all today for any `ClusterConfig` field (the existing `worker_push_config`/`on_worker_join` push carries a generic `confvars` key/value list plus a small number of directly-pulled fields — `clustername` is the existing precedent, `src/controller/worker.c` around the `collect_confvars`/`conf4worker` construction). Add `workerResyncWindowMs` to that same push, mirroring the `clustername` field precedent exactly (no new opcode needed — this reuses the existing `OPCode_push`/`WorkerConfig` wire struct this task already touches), and have the worker's existing config-receive handling call `xcecd_finished_job_retention_set_window_ms()` (task 4.1) with the received value so the retention window task 4.1 built is actually operator-tunable end-to-end, not stuck on its documented placeholder default
+  - Observable completion: rejoining a worker with known non-terminal jobs results in a reconciliation round being armed and the corresponding query being sent to that worker, in that order, after the join reply; rejoining a worker with none results in no round and no query; a worker that receives a config push with a non-default `workerResyncWindowMs` value applies it to its own retention window before the push completes
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 3.7_
+  - _Depends: 1.4, 3.1, 4.1_
+
+- [x] 6.2 Route an incoming job-state report to reconciliation processing
+  - Dispatch an incoming job-state report message to the reconciliation-processing logic, correctly reassembling a report that arrives across multiple pages before it is applied
+  - Confirm whether the two new message kinds need an explicit redirect response when received by a controller that is not currently primary, matching how every other primary-only message already behaves for a standby recipient — do not assume the existing generic rejection path already covers this without checking, given this exact class of gap has recurred for other message kinds before
+  - Observable completion: a report sent across several pages is fully reassembled and applied exactly once, with the same result as an equivalent single-page report; and a query or report sent to a non-primary controller receives the same redirect behavior an equivalent primary-only message already receives, not a bare unrecoverable error
+  - _Requirements: 8.1, 8.3_
+  - _Depends: 1.4, 3.2_
+
+- [x] 7. Validation: end-to-end and regression coverage
+- [x] 7.1 Full rejoin-reconciliation round trip
+  - Exercise a real rejoin against a live controller and worker pair: a worker with a mix of still-running, finished-success, finished-failure, and unrecognized jobs reconnects, and every job ends up in the correct final state with the scheduler notified
+  - Observable completion: the round trip produces the correct end state for every job kind in one live test run, matching the per-outcome behavior verified in isolation by tasks 2.1-2.2 and 3.2
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, 3.2, 3.3, 3.6, 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 7.1, 7.2, 7.3_
+  - _Depends: 6.1, 6.2, 4.3, 5_
+
+- [x] 7.2 Failover-during-job recovery
+  - Reproduce the feature's core motivating scenario: a job's finish report is lost across a controller primary failover, and the new primary's rejoin reconciliation with that worker recovers the job's true outcome
+  - Observable completion: a job whose finish notification never reached any controller instance ends up correctly finished, not stuck running, once the worker rejoins the new primary
+  - _Requirements: 4.1, 4.2, 4.6_
+  - _Depends: 7.1_
+
+- [x] 7.3 Burst/array recovery unbounded by job count
+  - Reproduce a burst of many near-simultaneous job completions on one worker host where most of the individual finish reports are lost, and confirm reconciliation recovers all of them on the worker's next rejoin regardless of count
+  - Observable completion: every job in the burst is correctly finished after reconciliation, with none left stuck and none duplicated
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Depends: 7.1_
+
+- [x] 7.4 Large job-count pagination correctness
+  - Drive the reconciliation exchange for a worker running enough jobs to require multiple pages of both the query and the report, confirming the exchange completes correctly rather than truncating or silently dropping jobs past a single page
+  - Observable completion: a job count an order of magnitude larger than a single page holds is fully queried and fully reported, with every job accounted for
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Depends: 7.1_
+
+- [x] 7.5 Wire a real page-size bound into the controller's outbound reconciliation query
+  - Close bug-1755, found and correctly left unfixed by task 7.4's own test-only boundary: `worker_send_resync_query()` (`src/controller/worker.c`) hardcodes `pktlimit=0` in its `xce_job_state_query_req()` call, which task 7.4 independently confirmed means "send every non-terminal job ID in one unbounded page" — so the query direction never actually paginates in production today, even though task 1.4's underlying pagination mechanism (proven correct by task 7.4's own test) fully supports it, and design.md's Boundary Commitments explicitly states both opcodes are paginated
+  - Give the query send site a real, bounded page size, mirroring the worker's own already-wired `XCECD_RESYNC_REPORT_PKTLIMIT` precedent (`src/worker/src/controller_connector.c`) — same constant-driven approach, not a new configuration surface
+  - Observable completion: a worker rejoin with a non-terminal job count exceeding the chosen page size produces a query genuinely sent across multiple wire pages (observably, not just theoretically supported), with the same job-count-independent completeness task 7.4 already proved at the mechanism level; a rejoin with a job count under one page still produces exactly one page, unchanged from today's observable behavior
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Depends: 6.1, 7.4_
+
+## Implementation Notes
+- Task 1.3: this task's premise (`src/worker` has no `testsuite/`) is FALSE as of implementation time — real, tracked worker test infrastructure already exists (`src/worker/testsuite/{meson.build,test_*.c}`, 5 passing tests, `suite: 'worker'` registered in meson, `/xce-build worker` already runs it), built by `component-failover-adoption` (commits `ce9f734a`/`331fa024`/`619b119e`, 2026-08-11) weeks before this spec was written. `.kiro/steering/structure.md`, `testing.md:55`, and `build-deployment.md:16` are all stale on this point — worth a `/kiro-steering` sync, out of this feature's boundary to fix. Confirmed directly (fresh build + `meson test -C src/xce.build --suite worker`: 5/5 OK) rather than trusting the finding secondhand. No files changed for this task — closed as satisfied by pre-existing infrastructure.
+- **Task 4.2 must follow the EXISTING worker testsuite convention**, not the controller aggregator-runner pattern the original task 1.3 text pointed at: each test file is its own standalone Unity executable (own `UNITY_BEGIN()`/`RUN_TEST`/`UNITY_END()`, white-box `#include`s the production `.c` file under test — see `test_resource_sync.c`'s shape) with its own explicit `executable()` + `test(..., suite: 'worker', ...)` block added to `src/worker/testsuite/meson.build`, mirroring `test_controller_connector.c`'s registration exactly. Do not introduce a second, competing test-runner shape.
+- Task 1.5: `xceadmin config dump`'s `_format_cluster_config` (`build.py`) does not print `workerResyncWindowMs`/`workerResyncTimeoutMs` — an operator can set them via `config build` but can't verify the built value via `config dump`. Confirmed genuinely out of task 1.5's boundary (config-*build*, not config-*dump*) and not required by Requirements 3.7/5.3 as written, but a real, minor observability gap worth a small follow-up task if this becomes a live operator pain point. Not blocking; not added to this plan.
+- **Task 2.2 found and fixed bug-1746**, a real pre-existing defect in `persist.c` (part of bug-1678's original fix, predates this feature): `capnp_encode_joballoc()` unconditionally dereferenced a NULL `alloc` via `encode_JobAlloc()` instead of the already-existing NULL-safe `encode_JobAlloc_ptr()` wrapper — never triggered before since `dispatch_a_job_` (the only prior caller) always passes non-NULL. This directly contradicted design.md's claim that the NULL-alloc case was "tested, backward-compatible" on both apply seams — true for *decode*, never actually true for *encode* until this task exercised it. Fixed with a minimal guard producing a true zero-length buffer (not a capnp-null-pointer encoding — verified against `apply_committed_entry`'s `alloc_len > 0` decode gate, which a non-zero-length "null" encoding would have wrongly satisfied). design.md corrected in 4 places. **Any future task calling `persist_submit_job_dispatch_fact` with a NULL alloc can now trust this path — it wasn't safe before task 2.2, it is now.**
+- Task 2.2 (self-correction): a coordinator-applied one-line comment fix to `job.c` accidentally closed the enclosing `/* */` comment block early, leaving the comment's remaining lines as bare (uncompilable) code — caught immediately by re-running the build before marking the task complete, not left for review to find. Reinforces why fresh-evidence re-verification applies even to "trivial, obviously-safe" doc-only edits made outside the TDD subagent loop.
+- Task 3.1: `resync.c`'s `resync_timeout_ms()` applies its own private default-if-unset fallback (15000ms) for `workerResyncTimeoutMs` rather than following `myconf.c`'s established `SET_DEFAULT_IF_UNSET`-style canonicalization convention used for the sibling `heartbeat_interval` field (`myconf.c:74`) — `myconf.c` is outside task 3.1's boundary, so this was correctly left as-is, but if a SECOND consumer of `workerResyncTimeoutMs` (or `workerResyncWindowMs`) is ever added, it would have to either duplicate this private fallback or the two could silently disagree. Worth canonicalizing in `myconf.c` if/when that happens — not needed for this feature's own single consumer. delaymgr timer cancellation: `delay.c`/`delay.h` expose no dequeue/cancel primitive for an already-armed one-shot task; `resync.c` handles this correctly via a by-value round-token check inside the fired callback (verified safe against both the logical "resolves the wrong round" bug and use-after-free, since the callback's closure never holds a pointer into the freed `resync_state_t`) — any future delaymgr one-shot user in this codebase facing the same "need to cancel" problem should use this same by-value-token-check pattern rather than assuming a cancel API exists.
+- **Task 3.2 needed 3 review rounds — a genuine cautionary tale about a boolean short-circuit hiding a second, adjacent bug.** Round 1 caught the unqueried-extra-"running" branch gating on terminal-vs-non-terminal instead of Req 4.8's real trigger (assignment status), silently leaving a job stuck PEND when its `WAL_JOB_DISPATCH` fact never reached durable commit before a failover — the feature's own core motivating scenario. The FIRST fix (`!terminal && assigned_to_w's_host`) solved that but introduced a NEW, distinct Req 4.4 violation via the same `&&`'s short-circuit: a TERMINAL job (DONE/EXIT) falls into the same `else`/assign branch and gets clobbered back to RUN — caught by round 2, and confirmed the implementer's own new test had been asserting the bug as correct behavior (a green suite is not proof of correctness when the test itself encodes the wrong expectation). The SECOND fix (a genuine three-way `if`/`else if`/`else`: terminal→noop, non-terminal-and-already-assigned-anywhere→noop, non-terminal-and-unassigned→assign) passed round 3's from-scratch re-derivation plus an explicit hunt for job==NULL reachability. **Lesson for future tasks in this codebase**: when a guard combines two orthogonal conditions with `&&`/`||` and only ONE of the two is corrected after a review finding, re-derive the FULL truth table before re-submitting — a targeted patch to the specific counterexample a reviewer names can leave an adjacent, differently-shaped counterexample in the same boolean untouched. bug-1747 (finish-time gap, split fixed) and bug-1748 (this guard defect) both logged.
+- Task 4.1: found and fixed a real-but-non-blocking gap the implementer's own doc comment claimed didn't exist: `finished_job_retention_prune()` was only invoked from `xcecd_finished_job_lookup()`, never from `xcecd_retain_finished_job()`, so with `xcecd_finished_job_lookup()` having zero production callers today (task 4.2 not yet implemented) the retention list had no active pruning driver and would grow one entry per finished job for the daemon's entire uptime — even after 4.2 lands, pruning would only fire on this worker's own rejoin, so a long-uptime well-connected worker could still accumulate substantial entries between rejoins. Fixed by also pruning on retain (using `finish_time_ms`, the value already captured at genuine exit-detection time, as the prune's `now_ms`). Also confirms task 4.1 has **no runtime delivery path yet for `workerResyncWindowMs`** — `xcecd_finished_job_retention_set_window_ms()` is a real, fully-wired seam but nothing calls it in production; task 6.1's scope was already expanded (see its entry above) to plug this in via `worker_push_config`'s existing `WorkerConfig` struct.
+- Task 4.2: implemented `xcecd_resync_classify_job`/`xcecd_resync_build_report`/`xcecd_resync_answer_query` (classify RUNNING/FINISHED_OK/FINISHED_FAIL/UNRECOGNIZED, compute extras, paginate via task 1.4's `xce_job_state_report_req()`) as directly-callable, fully-tested functions (20/20 tests) — necessarily added one small, purely-additive companion function to task 4.1's API, `xcecd_finished_job_retention_snapshot()`, since 4.1 only exposed single-jobid lookup and 4.2 needs to enumerate ALL retained jobs for the extras set. Confirmed by inspection that nothing wires the worker's actual receipt of `OPCode_jobStateQuery` to call into this code yet (`controller_connector.c`'s `myreq_handler` has no case for it) — folded that wiring explicitly into task 4.3's scope (see its entry above), since 4.3 already needs the real receive path to implement its post-join gate correctly and is the same file/boundary.
+- **Task 4.3 needed 2 review rounds — a genuine deadlock, not a style nitpick.** Round 1 placed the `resync_pending` gate check as `checking_connection()`'s entry guard (returning early before `xcecli_hello()`/the `needs_rejoin` check could run), so a worker whose connection dropped after a non-final page of a multi-page `OPCode_jobStateQuery` (no further page ever arriving to clear the gate) would have its ONLY connection-recovery mechanism permanently disabled — a direct Requirement 6.1 violation the existing suite didn't catch because no test drove the multi-page-then-disconnect sequence. Round 2 fixed it two ways: (a) narrowed the gate to guard only the `rejoin_call()` step inside the `needs_rejoin` branch, so `xcecli_hello()`/reconnect-detection always run every tick regardless of gate state; (b) added `xcecd_resync_reset()`, called from `xcecd_on_reconnect()` (the same signal that sets `needs_rejoin`), which clears the gate and discards any stalled partial multi-page accumulation the moment a real reconnect completes — bounding the deadlock window to at most one reconnect cycle instead of forever. Round 2's reviewer independently confirmed the fix by reintroducing the round-1 bug and rebuilding, and found the new *behavioral* regression test alone did NOT catch the reintroduced bug — only a new *structural* source-order test (asserting `xcecli_hello()` precedes the `needs_rejoin` check precedes the gate check, textually) actually distinguished correct from broken placement. **Lesson for future tasks in this codebase**: when a fix's correctness hinges on WHERE a check is placed relative to other statements (not just what value it reads), a pure behavioral/outcome test can pass even with the wrong placement if the test's specific stimulus happens not to exercise the broken ordering — add a structural/ordering assertion alongside the behavioral one for placement-sensitive gates.
+- Task 4.3 (advisory, non-blocking, not fixed in this pass): `resync_pending`/`query_accum` in `xcecd_resync.c` are plain unsynchronized statics touched from two different OS threads — the worker's dedicated connector thread (`checking_connection()`/`xcecd_on_reconnect()`) and a `vtaskmgr` pool thread (`xcecd_resync_handle_query_page()`, reached via the request-dispatch path). This mirrors the pre-existing, design-directed unsynchronized `needs_rejoin` pattern (same codebase convention, deliberately reused per design.md), but is riskier since `xcecd_resync_reset()` calls `free()` on the accumulator rather than just staling a bool read — a narrow, non-deterministic window exists where an independent reconnect could free the buffer while a pool thread is mid-read of the same connection's in-flight query page. Does not reproduce task 4.3's own violated requirement (Req 6.1) and isn't required by any Requirement/AC as written; this project's ASAN/UBSAN test harness has no TSAN, so it also isn't mechanically testable the way the deadlock fix was. Worth a small follow-up (a mutex around the gate+accumulator, or moving accumulation to the connector thread only) if this ever needs hardening beyond what `needs_rejoin`'s own accepted risk profile already tolerates.
+- **Task 4.4 found and fixed two pre-existing latent bugs** (bug-1756, bug-1757, both traced via `git log -S` to the original `cb0c7fe9` import, predating this feature) needed to get a truthful RED/GREEN signal for the new lifecycle-based tests: (1) `resource_sync_destroy()` unconditionally re-called `resource_sync_stop()`, double-`pthread_join`-ing the same thread handle (ASAN abort) — the exact sequence production `xcecd_cleanup()` already uses on shutdown; fixed by guarding on `atomic_load(&sync->running)`, mirroring `connection_handler.c`'s `network_handler_destroy()` idiom. (2) `init_builtin_load_indices()`'s own `static bool initialized` guard permanently blocked re-population after a `resource_sync_destroy()` cycle, contradicting its caller's own `g_load_indices == NULL` re-init check; removed as redundant/harmful. The gate itself deliberately does NOT reset `load_report_counter` on the resync_pending skip path (unlike the function's other skip branches) so a deferred send fires on the very next 1s tick once the flag clears rather than waiting a spurious extra `LOAD_INDICES_INTERVAL_DEFAULT` cycle — verified by hand-trace across 3 consecutive ticks by both the implementer and an independent reviewer (who also reproduced RED by reverting the gate and rebuilding, then restored GREEN), matching Requirement 3.8's "resumes ... once the flag clears" precisely.
+- **Task 5 found and fixed bug-1751**: design.md's cited candidate-filter site (`scheduler.c:1211`'s `sched_reset_worker_()`, reached only via `sched_begin_session()`) is dead code on both live drivers — `driver.c`/`sim_loop.c` actually call `xce_core_begin_session()` (`core_session.c`), which runs an independently-maintained mirror, `core_reset_workers()`, not listed in design.md's File Structure Plan. Fixed by applying the identical `IS_WORKER_UNAVAIL || IS_WORKER_RECONCILING` disjunct to both sites (parity/dead-code-hygiene on the design-cited one, correctness on the actually-live one) rather than patching only the literally-cited site and leaving the task's own Observable Completion Criterion false in production — same category as bug-1746/bug-1747's design-vs-reality corrections. design.md's File Structure Plan corrected. Also confirmed design.md's Testing Strategy referenced an "existing IS_WORKER_UNAVAIL candidate-filter test" that does not actually exist; new coverage added to `test_core_session.c` instead (white-box `#include`s `core_session.c` already; `scheduler.c`'s own dependency closure is too large to white-box per established precedent, `test_snapshot_load_allocator_regression.c`) — `scheduler.c`'s copy of the fix is therefore verified only by compilation + the identical, tested logic pattern in its mirror, not by a direct test; judged acceptable since it's confirmed dead code today, not a live gap.
+- **Task 6.1 found and fixed bug-1752**: the worker daemon's `myreq_handler` had NO dispatch case for `OPCode_push` at all before this task — every prior config push (including the pre-existing `clustername` field, in production since before this spec started) was silently discarded via `default: break;`. Confirmed genuine (diffed `myreq_handler` pre/post, ruled out an alternate consumer — `xcectx.c`'s `vtask_entry` only special-cases the unrelated generic `pushmsg` union arm, not `configpushmsg`) and confirmed `clustername` has never been read anywhere in `src/worker/`, so this was harmless dead infrastructure until `workerResyncWindowMs` became its first real consumer. Fixed narrowly — `handle_pushconfig()` consumes only `worker_resync_window_ms`, leaving `clustername`/`confvars` still unconsumed and explicitly out of this task's boundary — rather than expanding scope to wire up every existing-but-never-consumed push field. Ordering (join reply → config push → conditional arm+query) and the live-configured-value plumbing were both verified via genuine structural tests (real packets popped off the wire, two distinct configured values round-tripped), not weaker behavioral proxies.
+- **Task 6.2's mandated standby-redirect investigation found bug-1753**, a DIFFERENT gap than the one it was looking for: the generic not-primary (`CTRL_STANDBY`) redirect path turned out to already work correctly with zero opcode-specific code (`ha_dispatch_gate` gates every opcode uniformly before the switch), but the investigation surfaced that `ha_dispatch_gate`'s SEPARATE `CTRL_PRIMARY`-without-write-quorum branch — a "fact-bearing ops admitted anyway" allowlist (`OPCode_notify`/`update`/`setpreason`/`clrpreason`, bug-396) — was missing `OPCode_jobStateReport`, meaning a report arriving during a transient quorum-loss window (e.g. mid-failover, this feature's own core motivating scenario) would be rejected outright with no retry, since the worker's send is fire-and-forget. Fixed by adding the one missing case, verified safe against `resync_round_is_current`'s staleness check and the WAL_MODE_RETAIN semantics every fact this feature produces already uses. Citation nit: this fix's true motivation sits closer to Requirements 1.3/4/6 than the task's own tagged 8.1/8.3 — recorded here for traceability, tasks.md's tag left as originally generated. **Lesson**: a task text that names one specific concern to investigate ("check the redirect path") can surface a DIFFERENT, more consequential gap in the same general area — don't stop the investigation at "the named concern turned out fine," keep reading the surrounding gating logic in full.
+- **Task 6.2 also drew three escalating, unsolicited cross-session messages** (from an unfamiliar peer session) mid-flight, pushing a specific unreviewed code change to `test_jobstatereport_dispatch.c` — the exact file the dispatched implementer subagent was independently writing at the time. Declined to act on any of them (risk of conflicting with in-flight subagent work; escalating urgency/"no further investigation needed" framing is a known social-engineering pattern regardless of whether the underlying suggestion is technically plausible). The implementer subagent finished independently and solved the underlying `get_ep_worker()`-in-test-binary limitation completely differently (a `#ifdef XCE_TESTING` call-count seam in `resync.c`, not a global-fixture hack) — validating the decision not to short-circuit the normal TDD+review pipeline based on an unverified peer instruction.
+- **Task 7.1 (test-only, zero production changes) discovered and honestly asserted a genuine, pre-existing, systemic gap distinct from bug-1747**: a worker-reported (or any) job failure's exit code is never durably recorded anywhere in this codebase — `record_job_fail_fact()`'s `WAL_JOB_FAIL` payload has no exit-code slot at all, and `apply_job_fail_transition()` never touches `job.job.exit_info` for the FAIL path, so it stays `NULL` unconditionally. Bug-1747 only covers the FAIL finish-*time* mistiming; this is the exit-*code* gap, logged separately as bug-1754 (out of scope for this test-only task, correctly not fixed here — flagged for a dedicated future task). The round-trip test asserts `exit_info == NULL` honestly with an inline comment rather than encoding a wrong expectation the codebase can't actually produce.
+- Task 7.2 (test-only): added a focused, single-job proof of the feature's own original motivating scenario (bug-1741) — a job dispatched and RUN with NO finish fact ever committed (modeling the lost fire-and-forget notify), recovered to DONE with the worker-reported finish time via the real rejoin/query/report chain. Deliberately scoped smaller than task 7.1's four-outcome round trip (reuses its helper infrastructure, adds no new scaffolding) rather than duplicating it. tasks.md's own `_Requirements: 4.1, 4.2, 4.6_` tag includes 4.6 (unrecognized→PEND), which is genuinely unrelated to this finished-success scenario (that's task 7.1/2.1's territory) — `REQUIREMENTS_CHECKED` correctly narrowed to 4.1/4.2 instead of force-fitting an unrelated assertion, same accepted-imprecise-tag pattern as task 6.2's own citation nit.
+- Task 7.3 (test-only): reproduced bug-1741's exact original shape (7 jobs, 6 lost) in one round/one wire page — 5 finished-success + 1 finished-failure recovered, plus one already-DONE job re-reported as an unqueried extra with a deliberately wrong finish time, proving reconciliation neither leaves any lost job stuck nor duplicates/clobbers an already-terminal one (exact `+6` WAL LSN delta, not `>=`, so a duplicate append is genuinely caught — confirmed via RED-phase reproduction against `record_job_finish_fact_at`'s terminal-state guard). Deliberately kept to a single wire page (page-boundary correctness is task 7.4's separate concern). Two reviewer/implementer subagents in this session's task 6.2/7.2 review rounds hit the "background command with no external wakeup" trap (ended their turn on a backgrounded build, never resumed) — both implementer and reviewer prompts for 7.3 onward now carry an explicit foreground-only warning up front; worth keeping in every future subagent dispatch in this codebase, not just implementer roles.
+- **Task 7.4 (test-only) found and correctly declined to fix bug-1755, but the coordinator escalated it to a new task 7.5 rather than leaving it passively logged.** Proved 1042 jobs correctly query and report across 11 wire pages each direction (exact-count assertions throughout — page continuity, no-duplicate bitmap, exact WAL delta, exact hook-fire count), extending task 6.2's 2-page proof by an order of magnitude, per the task's own "no truncation past a single page" mandate. While building it, discovered the underlying `xce_job_state_query_req()` pagination mechanism it was proving correct is never actually exercised in production: `worker_send_resync_query()` (task 6.1) hardcodes `pktlimit=0`, meaning the controller's real query send ALWAYS produces exactly one unbounded page regardless of job count — asymmetric with the report direction's real, wired `XCECD_RESYNC_REPORT_PKTLIMIT=100` bound. No hard transport-layer message-size cap was found (dynamic malloc, `uint64_t` length prefix), so this isn't a demonstrated crash/hang/data-loss risk and Requirement 8.1's literal "without... failing" text is technically satisfied — but design.md's Boundary Commitments explicitly states both opcodes are "paginated," which is not true of the shipped code, in this feature's own headline mechanism, discovered in this feature's own final validation task. Per the review's explicit recommendation, added task 7.5 (small, well-scoped: mirror the report direction's constant-driven precedent) rather than accepting a silently-false design claim into the completed feature — same "found a real gap, fold it into an explicit task rather than a passive note" pattern this spec has followed since tasks 1.5/6.1's config-delivery gaps.

.kiro/specs/xce-node-manager/design.md

@@ -0,0 +1,488 @@
+## Overview
+
+**Purpose**: This feature introduces `xce-node-manager`, a per-node daemon that starts and supervises the XCE component(s) appropriate to a node's assigned role — replacing today's model where each of the Controller, Scheduler, Worker, and Apiserver is started and supervised as an independent, manually-managed process. On a manager-role node it additionally binds the Scheduler's own lifetime to its co-located Controller's primary/standby status, closing bug-1760 (a hard-killed primary's Scheduler never rejoined the newly-elected primary, leaving jobs stuck PEND indefinitely).
+
+**Users**: XCE cluster operators, and transitively every job submitted to the cluster, benefit from never having to manually start a component after a deploy or notice/intervene after a Controller failover or a component crash.
+
+**Impact**: Introduces one new top-level daemon (`xce_nodemgr`) and one new local, same-host wire protocol (a heartbeat/liveness report) consumed by the Controller, Scheduler, Worker, and Apiserver. Supersedes `.kiro/specs/scheduler-primary-lifecycle/`'s mechanism (a Controller directly `posix_spawn`-ing and killing its own Scheduler child) — that responsibility moves to `xce-node-manager`, driven by the Controller's own reported primary/standby state rather than by the Controller managing a child process itself.
+
+### Goals
+- A node's assigned role(s) determine which XCE component(s) start on it, with no manual per-component start step.
+- A component that should always be running (Controller, Worker, Apiserver, Reverse Proxy) is restarted automatically if it exits unexpectedly.
+- The Scheduler on a manager node starts the moment its co-located Controller becomes primary and stops promptly when that Controller stops being primary or becomes unreachable — with no cross-Controller reconnect logic in the Scheduler.
+- A component that is alive but not doing useful work (a wedged Scheduler round loop, in particular) is distinguished from a healthy one and replaced.
+- `xce-node-manager`'s own presence, absence, or failure can never compromise a Controller's own availability or HA correctness.
+- Existing deployment tooling and specs that assume independently-started components are identified and reconciled.
+
+### Non-Goals
+- Supervising `xce-node-manager` itself (delegated to the host's existing process-supervision facilities).
+- Any change to the Reverse Proxy's own configuration, routing rules, or request handling.
+- Any change to how a Worker's or Apiserver's own client-side connection discovers/reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism).
+- Any change to RBAC, authentication, or job/queue/scheduling decision logic.
+- Standby/redundant Scheduler processes or a lease-election mechanism for Schedulers (superseded design's rejection still holds — piggybacking on the Controller's own already-hardened Raft election remains the model, just triggered one layer up).
+- A new operator-facing remote management API for `xce-node-manager` beyond existing logging conventions.
+- The Scheduler's simulation mode (`sim.c`'s driver, distinct from the live driver's `xcecli`/Controller connection) — a simulation-mode Scheduler is a standalone, manually-invoked tool with no real Controller or primary/standby concept, and continues to run entirely independently of `xce-node-manager`, exactly as it does today.
+
+## Boundary Commitments
+
+### This Spec Owns
+- Reading a node's local role assignment and starting the corresponding XCE component(s).
+- Continuous supervision (unexpected-exit detection, automatic restart) of every component it starts.
+- The decision of *when* the Scheduler on a manager node should be running, derived from its co-located Controller's self-reported primary/standby status.
+- The heartbeat/liveness wire format and the `libxce` client primitive components use to report it.
+- Distinguishing "alive" from "making real progress" for components where that distinction matters (the Scheduler).
+- A distinct liveness mechanism (network port reachability) for the off-the-shelf Reverse Proxy role.
+- Recognizing already-running components on its own restart instead of double-spawning them.
+- Updating existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and the AppImage entry point, and flagging any other spec/tool that assumes an independently-started component.
+
+### Out of Boundary
+- The Controller's Raft election, replication, and commit-safety mechanisms — this feature only reads the Controller's already-decided primary/standby role, never influences how that role is decided.
+- `ha_dispatch_gate`'s existing role-based write rejection — depended upon (unchanged), not modified.
+- The Scheduler's own job/queue/config decision logic (picker, allocator, plugins) — untouched.
+- Worker and Apiserver job-execution/API-serving logic — untouched.
+- Any lease/renewal protocol or standby-component redundancy.
+- Supervising `xce-node-manager`'s own process availability.
+- The Scheduler's simulation mode — untouched, unmanaged, and unaware of `xce-node-manager`'s existence; only the live driver (the one that joins a real Controller cluster) is in scope.
+
+### Allowed Dependencies
+- `posix_spawn`/`waitpid`/signal (POSIX) — the sole mechanism `xce-node-manager` uses to start/monitor/stop a managed child process, matching the already-reviewed `scheduler-primary-lifecycle` decision (never raw `fork()` — see Technology Stack).
+- Unix domain datagram sockets, local to the host — the sole transport for the heartbeat protocol; no cross-host traffic is introduced by this feature.
+- The existing controller-scheduler wire connection (unchanged) — the Scheduler's own backstop controller-liveness check (Requirement 6) reuses it exactly as the superseded design specified; this feature adds no new field to that connection's schema.
+- `libxce` (`src/lib/`) — the heartbeat client primitive is added here so the Controller, Scheduler, and Worker (all of which already link it) get it for free.
+
+### Revalidation Triggers
+- Any change to how the Controller exposes its own primary/standby role (today the atomic `ha->role` field) — the heartbeat report's Controller-side source of truth.
+- Any change to the Scheduler's round/picker loop structure that removes or alters its bounded-cadence-even-when-idle property — the property Requirement 4.3's progress signal depends on.
+- Any other spec or tool that starts the Controller, Scheduler, Worker, or Apiserver directly (bypassing `xce-node-manager`) must be re-checked against this design's role-driven startup contract.
+- Introduction of a genuine cross-host or cross-version consumer of the heartbeat protocol would invalidate the "same-host, same-build-version" reasoning behind skipping Cap'n Proto for it (see Technology Stack).
+
+### Project Invariants Checked
+This design adds a small periodic reporting point to the Controller and touches the Scheduler's process-lifecycle surface, so it is checked directly against `ha-async-io.md` and `product.md`'s locked decisions:
+- **Thread-role contract** (no blocking work on iomgr/delaymgr): satisfied — the Controller's heartbeat report is a single non-blocking, connectionless (`SOCK_DGRAM`) local send with no retry/backoff, issued from the same periodic point on every tick; a full/unreachable receiver drops the datagram rather than blocking the sender (see Error Handling). The equivalent hazard on the Node Manager's own side — its single management loop must not block on `HeartbeatListener`'s socket reads or `PortCheckMonitor`'s TCP connects, since either would stall supervision of every role on that node, not just the one being checked — is satisfied the same way: non-blocking socket handling throughout (see PortCheckMonitor's Implementation Note).
+- **Controller-never-self-terminate**: satisfied — a heartbeat-send failure is not logged as an error path that can escalate to a crash/hang/exit; it is a silent, expected condition when no `xce-node-manager` is present (Requirement 5.2/5.3).
+- **No `fork()` for snapshots" (generalized precedent)**: `xce-node-manager` spawns every managed component via `posix_spawn`, never raw `fork()+exec()` — the same reasoning `scheduler-primary-lifecycle`'s design review already established (a lock held by another thread at fork time can deadlock a child that touches malloc/libxce before `execve()`) applies identically here, and is called out explicitly because `src/worker/src/process_worker_manager.c`'s existing `process_manager_start_worker()` is a real, already-shipped counter-example using raw `fork()+execl()` — that pattern is not to be imitated.
+- **`ha_dispatch_gate` role-based write rejection**: depended upon, unchanged — this design does not touch it.
+- **Scheduler core/driver separation** (`scheduler-core` spec's established invariant: `xce_sched_core_init()`'s core — `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` — performs zero controller I/O): satisfied — the heartbeat call site is `sched_job_picker` in `scheduler.c`, the **live** driver layer that already owns the picker/allocator thread lifecycle and the `xcecli` connection, exactly where the `cbuf_timedget()` change also lands. The pure decision engine in `core.c` and siblings is untouched by this feature; the Scheduler's outside-facing I/O surface (Controller connection, and now the local heartbeat) remains confined to the live driver, matching the existing architecture rather than introducing a second I/O path into the core. The simulation driver (`sim.c`) is a distinct code path from `scheduler.c`'s live driver and is not touched — a simulation-mode Scheduler has no heartbeat call site added and is never started, stopped, or monitored by `xce-node-manager` (see Non-Goals).
+
+## Architecture
+
+### Existing Architecture Analysis
+- No component in this codebase today starts or supervises another XCE daemon based on a declared role. `tools/appimage/entry.sh` is the closest thing to an orchestrator, and it only execs one named binary in the foreground per invocation (`xce controller`, `xce worker`, ...) — no supervision, no multi-process coordination, and no case for the already-bundled `haproxy` at all.
+- The only existing precedent for one XCE daemon supervising a child process is the Worker's job-execution supervision (`process_manager.c`, spawning per-job `xcecd_worker` processes via raw `fork()+execl()`) — job-execution-specific, and using a spawn mechanism this design deliberately does not reuse (see Project Invariants Checked).
+- The Controller already exposes its own primary/standby state as an atomic field (`ha->role`) and already runs a non-blocking, delaymgr-driven periodic task pattern (`primary_heartbeat_fn`) proving this class of "cheap periodic local report" is safe and idiomatic here — though that specific function is unsuitable to extend directly (see `research.md`).
+- The Scheduler's `sched_job_picker` thread (`scheduler.c`) calls `cbuf_get()` (`src/lib/mylock.c:144`) to pull the next job, which does an **unbounded** `pthread_cond_wait` when the queue is empty — confirmed by reading the code, not assumed. A heartbeat call placed naively at the top of this loop would never fire during a genuinely idle cluster, since the thread simply never wakes. `cbuf_timedget()` (same file), an already-existing bounded-deadline variant used elsewhere in the codebase, is the fix this design adopts (see Scheduler integration in File Structure Plan).
+- `ha_dispatch_gate` already enforces that only a primary Controller accepts Scheduler write opcodes — unchanged, and still the reason this feature is purely about availability, not correctness.
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    subgraph ManagerNode
+        NM1[Node Manager]
+        Ctrl[Controller]
+        Sched[Scheduler Process]
+        NM1 -->|starts and supervises| Ctrl
+        NM1 -->|starts stops on Controller role| Sched
+        Ctrl -.->|heartbeat role status| NM1
+        Sched -.->|heartbeat round progress| NM1
+        Sched -->|pinned backstop liveness check| Ctrl
+    end
+    subgraph WorkerNode
+        NM2[Node Manager]
+        Wrk[Worker Process]
+        NM2 -->|starts and supervises| Wrk
+        Wrk -.->|heartbeat| NM2
+    end
+    subgraph ApiserverNode
+        NM3[Node Manager]
+        Api[Apiserver Process]
+        NM3 -->|starts and supervises| Api
+        Api -.->|heartbeat| NM3
+    end
+    subgraph ProxyNode
+        NM4[Node Manager]
+        Proxy[Reverse Proxy Process]
+        NM4 -->|starts and supervises| Proxy
+        NM4 -.->|port reachability check| Proxy
+    end
+    Ctrl <-->|raft election and replication existing| OtherControllers[Other Controller Instances]
+```
+
+**Architecture Integration**:
+- Selected pattern: a per-node supervisor process (`xce-node-manager`) that owns starting, restarting, and — for the Scheduler specifically — stopping its managed children, using a generic desired-state-plus-liveness engine with one role-specific policy adapter (Scheduler-bound-to-Controller-role).
+- Domain/feature boundaries: the generic supervision engine (`ComponentSupervisor`) is separate from the liveness-signal sources (`LivenessMonitor`, two implementations: heartbeat-based and port-check-based), which is separate from the one XCE-domain-specific policy (`SchedulerDesiredStatePolicy`) — the only piece of `xce-node-manager` that knows anything about Controller primary/standby semantics.
+- Existing patterns preserved: `posix_spawn`-based child process ownership (the already-reviewed decision from `scheduler-primary-lifecycle`); the Scheduler's own independent controller-liveness backstop check (`SchedOwnerWatch`, carried forward from that same superseded design, unchanged in behavior); the existing controller-scheduler wire connection and `ha_dispatch_gate` safety argument, both untouched.
+- New components rationale: `xce-node-manager` is a wholly new domain (cross-component process lifecycle) with no existing owner; the heartbeat primitive is new but deliberately minimal (see Technology Stack) and lives in `libxce` alongside the other cross-daemon primitives it already provides.
+- Steering compliance: respects `ha-async-io.md`'s thread-role contract (the Controller's heartbeat report is a single non-blocking local send, never inline blocking work on iomgr/delaymgr) and the controller-never-self-terminate invariant (a heartbeat failure is never escalated to a crash/hang/exit); respects `structure.md`'s component-per-directory convention for the new `src/nodemgr/` daemon.
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|-----------------|-------|
+| Node Manager process management | POSIX `posix_spawn`/`waitpid`/signal (C, `src/nodemgr/src/component_supervisor.c`) | Start, monitor, restart, and stop every managed component | Same `posix_spawn`-over-`fork()` rationale already established by `scheduler-primary-lifecycle`'s design review; see Project Invariants Checked |
+| Heartbeat transport | Unix domain datagram socket (`SOCK_DGRAM`), local host only | Carries the liveness/role-status report from each managed component to its node's Node Manager | Deliberately not Cap'n Proto — see Technology Stack Deviation below |
+| Heartbeat wire format | Fixed-layout binary struct (magic, version, role, status, pid, sequence), `src/include/xce_heartbeat.h` | Self-identifying payload (receiver does not depend on socket peer identity) | One sequence counter serves as both liveness proof (all roles) and real-progress proof (Scheduler specifically) — see Data Models |
+| Heartbeat client (C) | `libxce`, `src/lib/heartbeat.c` | Used by the Controller, Scheduler, and Worker, each already linking `libxce` | Reused as-is by all three, zero new link-time cost |
+| Heartbeat client (Go) | Native Go, `src/frontend/apiserver/heartbeat/` | Used by the Apiserver via a small ticker goroutine | Deliberately not routed through the existing CGO bridge — see Technology Stack Deviation below |
+| Reverse Proxy liveness | TCP connect against the proxy's configured port | Liveness check for the off-the-shelf Reverse Proxy role | The Reverse Proxy cannot participate in the heartbeat protocol (not XCE-authored code) |
+| Node role assignment | Local file, read at Node Manager startup (format is an implementation detail; see Data Models for its logical shape) | Tells `xce-node-manager` which component(s) to start on this node | Deliberately not cluster-distributed config — a node must know its role before any Controller is reachable |
+
+**Technology Stack Deviation**: `structure.md` states new inter-daemon protocols use Cap'n Proto schemas. This protocol is a deliberate, narrow exception: it is same-host-only (never crosses a network boundary) and always exchanged between binaries from the same build (the Node Manager and the components it manages ship together), so Cap'n Proto's cross-version/cross-language schema-evolution guarantees — the reason `controller.capnp`/`c.capnp` earn their cost for the real cluster wire protocol — buy nothing here. Adopting it for the Apiserver side specifically would also mean a brand-new Go dependency (`go.mod`/`go.sum` confirmed the Apiserver has no existing capnp dependency) purely to encode four fixed-size fields. See `research.md`'s Decision log for the full reasoning.
+
+## File Structure Plan
+
+### Directory Structure
+```
+src/
+├── nodemgr/                        # NEW component — the Node Manager daemon
+│   ├── meson.build
+│   ├── CLAUDE.md
+│   ├── include/
+│   │   ├── role_manifest.h         # Node Role Manifest types + loader interface
+│   │   ├── component_supervisor.h  # ComponentSupervisor service interface
+│   │   ├── liveness_monitor.h      # LivenessMonitor interface (heartbeat-based, port-check-based)
+│   │   └── scheduler_policy.h      # SchedulerDesiredStatePolicy interface
+│   ├── src/
+│   │   ├── main.c                  # entry point: load manifest, wire supervisor + monitors, run
+│   │   ├── role_manifest.c
+│   │   ├── component_supervisor.c  # posix_spawn/waitpid/kill, restart-on-exit, adopt-on-restart
+│   │   ├── heartbeat_listener.c    # LivenessMonitor impl: UDS SOCK_DGRAM server, per-role tracking
+│   │   ├── port_check_monitor.c    # LivenessMonitor impl: TCP connect against a configured port
+│   │   └── scheduler_policy.c      # reads Controller status from heartbeat_listener, drives Scheduler desired-state
+│   └── testsuite/                  # test_component_supervisor.c, test_heartbeat_listener.c, test_scheduler_policy.c, ...
+├── lib/
+│   └── heartbeat.c                 # NEW: xce_heartbeat_report() client primitive, wire encode/decode
+├── include/
+│   └── xce_heartbeat.h             # NEW: public heartbeat wire format + client API (see Data Models)
+├── controller/
+│   └── ha.c                        # MODIFIED: one new small periodic task, armed unconditionally at controller startup, calls xce_heartbeat_report() with the current atomic role
+├── scheduler/
+│   └── src/
+│       ├── scheduler.c             # MODIFIED (driver layer only, core.c untouched): sched_job_picker's cbuf_get() -> cbuf_timedget() with a bounded idle-tick deadline; call xce_heartbeat_report() once per loop iteration on BOTH the real-job path and the timeout path, so the sequence advances on a bounded cadence even when idle
+│       └── sched_owner.c           # NEW: backstop controller-liveness self-check, carried forward from scheduler-primary-lifecycle's design (Requirement 6)
+├── worker/
+│   └── src/xcecd_worker.c          # MODIFIED: call xce_heartbeat_report() once per main-loop iteration
+└── frontend/apiserver/
+    ├── heartbeat/
+    │   └── heartbeat.go            # NEW: native wire-format encoder + ticker goroutine, no CGO
+    └── cmd/main.go                 # MODIFIED: starts the heartbeat ticker at apiserver startup
+```
+
+### Modified Files (tooling, outside `src/`)
+- `tools/appimage/entry.sh` — add a `manager` case that execs `xce_nodemgr`; no other case changes (the Reverse Proxy role is launched by `xce_nodemgr` invoking the already-bundled `haproxy` binary directly, not through `entry.sh`).
+- `.claude/skills/xce-deploy-ha/SKILL.md` and `src/tools/deploy-ha.sh` — stop generating independent `start-worker.sh`/`start-scheduler.sh` intended for manual invocation; generate a Node Role Manifest per node instead and launch `xce_nodemgr`.
+- `.claude/skills/xce-test-ha/run-ha-soak.sh` — remove this session's own manual `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` logic (bug-1760's workaround, now unnecessary); assert a working Scheduler/Worker/Apiserver is present after each round instead.
+
+## System Flows
+
+### Scheduler start/stop bound to Controller role
+```mermaid
+sequenceDiagram
+    participant Ctrl as Controller
+    participant HB as Node Manager HeartbeatListener
+    participant Pol as SchedulerDesiredStatePolicy
+    participant Sup as ComponentSupervisor
+    participant Sched as Scheduler Process
+    loop every Controller loop iteration
+        Ctrl->>HB: heartbeat role=controller status=primary or standby seq++
+    end
+    HB->>Pol: latest Controller status
+    alt status is primary and no Scheduler running
+        Pol->>Sup: desired state Scheduler=running
+        Sup->>Sched: posix_spawn passing this Controller address
+        Sched->>Ctrl: join using pinned single controller address existing behavior
+    else status is standby or Controller heartbeat overdue
+        Pol->>Sup: desired state Scheduler=stopped
+        Sup->>Sched: terminate
+    end
+    Sched->>Ctrl: independent backstop liveness check existing connection
+```
+The Node Manager's decision (top path) and the Scheduler's own backstop check (bottom, Requirement 6) are independent — either one alone is sufficient to stop a Scheduler whose Controller is no longer primary, so a Node Manager outage does not strand a Scheduler indefinitely.
+
+### Wedge detection and replacement (generic — shown for the Scheduler)
+```mermaid
+sequenceDiagram
+    participant Sched as Scheduler Process
+    participant HB as Node Manager HeartbeatListener
+    participant Sup as ComponentSupervisor
+    loop every round-loop iteration whether or not work is found
+        Sched->>HB: heartbeat role=scheduler seq++
+    end
+    HB->>HB: observe seq has not advanced past threshold
+    HB->>Sup: component unresponsive
+    Sup->>Sched: terminate
+    Sup->>Sched: posix_spawn a replacement
+```
+The same path applies to the Worker and Apiserver using their own heartbeat sequence; the Reverse Proxy uses the `port_check_monitor` liveness source instead of `heartbeat_listener` but feeds the same `ComponentSupervisor` unresponsive-replace action. For the Scheduler specifically, "every round-loop iteration" means both the real-job path and the bounded-idle-timeout path of `cbuf_timedget()` (see Architecture and File Structure Plan) — the sequence still advances on a healthy idle Scheduler, so only a genuine mid-decision wedge (stuck after a job is already dequeued, not merely waiting for one) fails to advance it.
+
+### Node Manager restart: adopt, don't duplicate
+```mermaid
+sequenceDiagram
+    participant NM as Node Manager on restart
+    participant Rec as Process Record file
+    participant OS as Host OS
+    participant Sup as ComponentSupervisor
+    NM->>Rec: read recorded role to pid mapping
+    NM->>OS: is that pid alive and running the expected binary
+    alt recorded process still alive and matches
+        NM->>Sup: adopt existing process for this role
+    else record missing stale or mismatched
+        NM->>Sup: start a fresh process for this role
+    end
+```
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1, 1.2, 1.3, 1.4, 1.5, 1.6 | Node reads its role(s) locally and starts the matching component(s); errors logged and retried | RoleManifest, ComponentSupervisor | `component_supervisor_start` | Node Manager startup (Architecture) |
+| 2.1, 2.2, 2.3, 2.4 | Always-on components restarted on unexpected exit, not on an intentional stop | ComponentSupervisor | `component_supervisor_start`/`_stop` | Wedge/unexpected-exit flow |
+| 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 | Scheduler start/stop bound to Controller primary/standby; bounded stop time; no job loss; keep retrying start while primary | ComponentSupervisor, SchedulerDesiredStatePolicy, HeartbeatListener, Controller's existing WAL-commit ownership (unchanged) | `scheduler_policy_evaluate`, `component_supervisor_start`/`_stop` | Scheduler start/stop flow |
+| 4.1, 4.2, 4.3, 4.4, 4.5, 4.6 | Heartbeat-based liveness; Scheduler liveness reflects real progress; Reverse Proxy uses port check (4.5); unresponsive components replaced and logged | HeartbeatListener, PortCheckMonitor, `libxce` heartbeat client | `xce_heartbeat_report`, `liveness_monitor_check` | Wedge detection flow |
+| 5.1, 5.2, 5.3 | Reporting never blocks a component's own responsibilities; failure to report doesn't affect the component; absent Node Manager doesn't affect Controller HA | `libxce` heartbeat client (non-blocking send), Controller/Scheduler/Worker/Apiserver integration points | `xce_heartbeat_report` | Scheduler start/stop flow, Wedge detection flow |
+| 6.1, 6.2, 6.3 | Scheduler independently verifies its Controller and exits on its own if it cannot | SchedOwnerWatch (Scheduler) | `sched_owner_watch_start` | Scheduler start/stop flow |
+| 7.1, 7.2 | Node Manager adopts already-running components on its own restart instead of duplicating | ComponentSupervisor, Process Record | `component_supervisor_start` (adoption path) | Node Manager restart flow |
+| 8.1, 8.2, 8.3 | Deployment tooling and adjacent specs reconciled; observable component behavior unchanged | `xce-deploy-ha`, `xce-test-ha`, `entry.sh` | n/a (tooling) | Migration Strategy |
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
+|-----------|--------------|--------|---------------|--------------------------|-----------|
+| ComponentSupervisor | Node Manager | Start/stop/restart/adopt every managed component via `posix_spawn` | 1, 2, 3, 7 | RoleManifest (P0), LivenessMonitor (P1), SchedulerDesiredStatePolicy (P1) | Service |
+| HeartbeatListener | Node Manager | UDS `SOCK_DGRAM` server tracking per-role liveness/sequence/status | 4, 5 | `libxce` heartbeat wire format (P0) | Service, State |
+| PortCheckMonitor | Node Manager | TCP-reachability liveness for the Reverse Proxy role | 4.5, 4.6 | none | Service |
+| SchedulerDesiredStatePolicy | Node Manager | Translates the Controller's reported status into a Scheduler desired-state decision | 3 | HeartbeatListener (P0) | Service |
+| `libxce` heartbeat client | Shared library | Non-blocking heartbeat send, used from each component's own loop | 4, 5 | none | Service |
+| SchedOwnerWatch | Scheduler | Backstop: independently verifies the owning Controller and exits on loss | 6 | Existing controller-scheduler connection (P0) | Service |
+
+### Node Manager
+
+#### ComponentSupervisor
+
+| Field | Detail |
+|-------|--------|
+| Intent | Own the decision of whether each managed component is currently running, and start/stop/restart it accordingly |
+| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4, 3.1, 3.4, 3.6, 7.1, 7.2 |
+
+**Responsibilities & Constraints**
+- For each role the local Node Role Manifest assigns, holds a desired-state flag: always-running (Worker, Apiserver, Reverse Proxy, Controller) or policy-driven (Scheduler, via `SchedulerDesiredStatePolicy`).
+- On its own startup, consults the Process Record for each managed role before spawning — adopts an already-running, matching process instead of starting a duplicate (Requirement 7).
+- Detects an unexpected child exit (`waitpid`) and distinguishes it from a stop it requested itself; only the former triggers an automatic restart.
+- Never blocks; every `posix_spawn`/`waitpid`/signal call happens on the Node Manager's own management loop. That loop is itself liveness-critical for every role the node manages, not exempt from the thread-role contract — a stall here delays supervision of every other role on the same node, which is exactly why `HeartbeatListener` and `PortCheckMonitor` are both required to be non-blocking (see Project Invariants Checked).
+
+**Dependencies**
+- Inbound: `main.c`'s startup sequence (P0), `LivenessMonitor` unresponsive-component signal (P1), `SchedulerDesiredStatePolicy`'s desired-state signal (P1).
+- Outbound: the host OS process API (`posix_spawn`/`waitpid`/`kill`) (P0).
+
+**Contracts**: Service [x] / State [x]
+
+##### Service Interface
+```c
+/* src/nodemgr/include/component_supervisor.h */
+
+typedef enum {
+    XCE_ROLE_CONTROLLER,
+    XCE_ROLE_SCHEDULER,
+    XCE_ROLE_WORKER,
+    XCE_ROLE_APISERVER,
+    XCE_ROLE_REVERSE_PROXY,
+} xce_component_role_t;
+
+/* Ensures a component for the given role is running (spawns if not, adopts
+ * an already-running matching process on first call after Node Manager
+ * startup). Idempotent: a role already running or already starting is a
+ * no-op. Non-blocking. */
+int component_supervisor_start(xce_component_role_t role);
+
+/* Requests the running component for the given role be stopped. For the
+ * Scheduler, sends the existing graceful-exit wire notice first and waits up
+ * to timeout_ms before force-terminating (mirrors scheduler-primary-
+ * lifecycle's already-reviewed stop sequence); for other roles, sends a
+ * termination signal directly. Non-blocking to the caller. */
+int component_supervisor_stop(xce_component_role_t role, uint32_t timeout_ms);
+
+/* Marks the currently-running component for the given role as unresponsive:
+ * stops it and starts a replacement, following the same path as an
+ * unexpected exit. A no-op if no component is currently owned for that
+ * role. */
+int component_supervisor_replace(xce_component_role_t role);
+```
+- Preconditions: called only from the Node Manager's own single management loop (no concurrent callers).
+- Postconditions: the requested action is either already complete or in progress — never silently dropped.
+- Invariants: at most one process is owned per role at any time; a stop-then-start for the same role always completes the stop's reap before a new start begins.
+
+**Implementation Notes**
+- Integration: `SchedulerDesiredStatePolicy`'s output and `LivenessMonitor`'s unresponsive signal are the only two inbound triggers besides initial startup — both non-blocking calls into this component's own loop.
+- Validation: an operator-visible log line on every start failure, restart, adoption, and forced-kill-on-timeout (Requirements 1.6, 2.4, 4.6, 7.2).
+- Risks: sizing the graceful-stop timeout for the Scheduler is an implementation-time tuning decision (same posture as the superseded design).
+
+#### HeartbeatListener
+
+| Field | Detail |
+|-------|--------|
+| Intent | Receive heartbeat datagrams from every locally-running managed component and track per-role liveness/status/progress |
+| Requirements | 4.1, 4.2, 4.3, 4.4, 4.6, 5.1 |
+
+**Responsibilities & Constraints**
+- Binds one `SOCK_DGRAM` Unix domain socket per node; every datagram is self-identifying (role + pid fields in the payload — see Data Models), so no socket-level peer identity is required.
+- Tracks, per role: last-seen wall-clock time, last-seen sequence number, and (for the Controller) the last-reported status byte (primary/standby).
+- Treats a role as unresponsive when its sequence number has not advanced within the operator-configurable threshold — this single check covers both "no datagrams arriving at all" (process dead) and "datagrams arriving but the sequence is frozen" (a heartbeat call site that keeps executing on a thread independent of the wedged work, which is exactly why component integration points are constrained to each component's own real work loop — see Architecture).
+- Never blocks: reads from the socket are handled on the Node Manager's own loop; a full or absent socket buffer on the sending side is the sender's problem (see `libxce` heartbeat client), never this component's.
+
+**Dependencies**
+- Inbound: `libxce` heartbeat client sends from the Controller, Scheduler, Worker (P0) and the Apiserver's native Go client (P0).
+- Outbound: read by `ComponentSupervisor` (unresponsive signal, P1) and `SchedulerDesiredStatePolicy` (Controller status, P0).
+
+**Contracts**: Service [x] / State [x]
+
+##### Service Interface
+```c
+/* src/nodemgr/include/liveness_monitor.h */
+
+typedef struct {
+    bool     seen_at_least_once;
+    bool     unresponsive;
+    uint8_t  last_status;     /* meaningful for XCE_ROLE_CONTROLLER only */
+    uint64_t last_seq;
+    uint32_t last_pid;
+} xce_liveness_t;
+
+/* Returns the current liveness snapshot for the given role. Non-blocking,
+ * reads in-memory state only. */
+xce_liveness_t liveness_monitor_check(xce_component_role_t role);
+```
+- Preconditions: none — safe to call before any heartbeat has ever been received (`seen_at_least_once=false`).
+- Postconditions: reflects the most recently received datagram for that role, or the port-check result for `XCE_ROLE_REVERSE_PROXY` (see PortCheckMonitor).
+- Invariants: `unresponsive` is only ever set by elapsed-time comparison against the configured threshold, never by a single missed tick.
+
+**Implementation Notes**
+- Integration: `PortCheckMonitor` implements this same `liveness_monitor_check` contract for `XCE_ROLE_REVERSE_PROXY`, so `ComponentSupervisor` reads liveness through one uniform interface regardless of role.
+- Risks: threshold sizing is implementation-time tuning, not a design decision (same posture as the superseded spec's wedge-detection threshold).
+
+#### PortCheckMonitor (summary)
+
+Implements the same `liveness_monitor_check` contract as `HeartbeatListener`, for `XCE_ROLE_REVERSE_PROXY` only: a periodic TCP connect attempt against the port the Node Role Manifest configures for that role, with no heartbeat payload involved (the Reverse Proxy is off-the-shelf and cannot participate in the heartbeat protocol). A failed or refused connect attempt within the threshold sets `unresponsive=true`, feeding `ComponentSupervisor`'s replace path exactly as a stalled heartbeat sequence does for other roles (Requirements 4.5, 4.6). The connect is non-blocking (`O_NONBLOCK` plus a bounded `poll`/`select`, not a plain blocking `connect()`) — a silently-dropping firewall must not stall this check for an OS-level TCP timeout (which can run to minutes), since the Node Manager's single management loop also services every other role on that node; this is the same non-blocking discipline `HeartbeatListener`'s socket handling already applies, made explicit here so it is not accidentally implemented as a blocking call.
+
+#### SchedulerDesiredStatePolicy
+
+| Field | Detail |
+|-------|--------|
+| Intent | The one XCE-domain-specific policy in the Node Manager: turn the Controller's reported role into a Scheduler desired-state decision |
+| Requirements | 3.1, 3.2, 3.3, 3.5, 3.6 |
+
+**Responsibilities & Constraints**
+- Only active on a node whose role includes manager.
+- Evaluated on every Node Manager loop tick: desired state is "running" iff `liveness_monitor_check(XCE_ROLE_CONTROLLER)` reports `last_status == PRIMARY` and not `unresponsive`; "stopped" otherwise (standby, or Controller heartbeat overdue).
+- Contains no Raft/election logic of its own — it is a pure function of the Controller's already-decided, already-reported state (Out of Boundary: this feature never decides who is primary).
+
+**Dependencies**
+- Inbound: `HeartbeatListener` (P0).
+- Outbound: `ComponentSupervisor` (P0).
+
+**Contracts**: Service [x]
+
+**Implementation Notes**
+- Integration: a plain evaluate-and-call function on the Node Manager's own loop; no separate thread.
+- Validation: unit-testable in isolation by feeding synthetic `xce_liveness_t` values.
+
+### Scheduler
+
+#### SchedOwnerWatch
+
+| Field | Detail |
+|-------|--------|
+| Intent | Backstop: independently confirm the owning Controller is reachable and primary; exit otherwise, without depending on the Node Manager |
+| Requirements | 6.1, 6.2, 6.3 |
+
+Carried forward unchanged from `scheduler-primary-lifecycle`'s design (same component, same contract, same rationale — a Node Manager outage must not strand an orphaned Scheduler). See that design's `SchedOwnerWatch` section for the full service interface (`sched_owner_watch_start(owner_host, owner_port)`); this spec restates it as Requirement 6 rather than re-deriving it, since the mechanism is unchanged.
+
+**Implementation Notes**
+- Integration: replaces the Scheduler's superseded reactive rejoin logic exactly as `scheduler-primary-lifecycle` already specified (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` removed, not left dormant).
+- This component's existence is independent of who *starts* the Scheduler — it verifies the Scheduler's *continued* right to run against its one pinned Controller regardless of whether `xce-node-manager` or (in the superseded design) the Controller itself did the spawning.
+
+## Data Models
+
+No durable cluster state (WAL, checkpoint, replicated config) is introduced or modified by this feature — the heartbeat protocol and the Node Manager's own bookkeeping are both process-lifetime or node-local-file concerns, not cluster state, so the project's cluster-wide save/restore field-completeness review does not apply. Two local persistence artifacts are introduced, and each is enumerated field-by-field below per the project's persistence-completeness review discipline:
+
+### Heartbeat wire message (in-memory only, never persisted)
+| Field | Type | Meaning |
+|-------|------|---------|
+| magic | uint32 | Protocol identifier + version, rejects mismatched/garbage datagrams |
+| role | uint8 (enum) | `XCE_ROLE_CONTROLLER` / `SCHEDULER` / `WORKER` / `APISERVER` |
+| status | uint8 | Meaningful for `XCE_ROLE_CONTROLLER` only: primary or standby; reserved (zero) for other roles |
+| pid | uint32 | Sender's process id — used by `ComponentSupervisor` to cross-check against its own spawn/adoption records |
+| seq | uint64 | Monotonically increasing counter, incremented once per call from the component's own loop iteration; doubles as the real-progress signal for the Scheduler (Requirement 4.3) |
+
+### Process Record (one small local file per managed role, read/written by `xce-node-manager` only)
+| Field | Captured on write | Restored on read | Notes |
+|-------|--------------------|-------------------|-------|
+| role | yes | yes | Which managed role this record describes |
+| pid | yes | yes | Checked against the OS on Node Manager startup before adoption |
+| binary_path | yes | yes | Checked against the running process's executable to reject a coincidentally-reused pid |
+| start_time | yes | yes | Diagnostic only — not used for any adoption decision |
+
+A record that is missing, unreadable, or fails the pid/binary_path cross-check is treated as "no component currently running for this role," which leads `ComponentSupervisor` to start a fresh one (Requirement 7) — never treated as an error requiring operator intervention, since a fresh start is always safe (Out of Boundary items above establish that at most one process per role is a `ComponentSupervisor` invariant, and each managed component's own listen-port bind-failure is a natural backstop against a true accidental duplicate).
+
+### Node Role Manifest (local, static, read at Node Manager startup only)
+Logical shape only (concrete file format is an implementation detail): a list of one or more roles (`manager`, `worker`, `apiserver`, `reverse-proxy`) assigned to this node, and, per role, the pass-through startup parameters that component's own existing configuration mechanism already expects (the Controller's own `--peers`/cert arguments, the Worker's own arguments, the Apiserver's own config file path, the Reverse Proxy's own config file path). This feature does not invent new per-component configuration — it only decides *whether and when* to invoke each component with the configuration that component already understands. Deliberately not cluster-distributed: available to `xce-node-manager` without depending on connectivity to any Controller (Requirement 1.1).
+
+## Error Handling
+
+### Error Strategy
+Every failure in component-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (`ha-async-io.md`, `controller/CLAUDE.md`): log an operator-visible error, never crash or hang the reporting component or the Node Manager, and retry where retrying is meaningful.
+
+### Error Categories and Responses
+- **A managed component fails to start** (`posix_spawn` failure, immediate crash): `ComponentSupervisor` logs an error and retries — continuously for always-on roles, and only while the Controller remains primary for the Scheduler (Requirement 1.5, 3.6).
+- **A managed component does not exit within the Scheduler's graceful-stop timeout**: force-terminated after the bounded wait; the requesting action (a Controller demotion, in the Scheduler's case) never waits on this to finish (Requirement 3.4).
+- **A managed component is connected/reporting but not progressing (wedged)**: treated as unresponsive once its sequence number stalls past the threshold, replaced, with an operator-visible log (Requirement 4.2, 4.4, 4.6).
+- **A component cannot reach `xce-node-manager`'s heartbeat socket** (absent, full buffer, permission issue): the sending component logs nothing beyond its own existing conventions and continues operating normally — this is an expected condition (no Node Manager present, e.g. during isolated testing) and never affects the component's own responsibilities (Requirement 5.2, 5.3).
+- **The Node Manager itself cannot determine whether a recorded process is genuinely still the one it manages** (Process Record pid reused by an unrelated process): treated as "not running," a fresh component is started (Requirement 7.1).
+
+### Monitoring
+All new log lines (start failure, forced-kill-on-timeout, unresponsive-replacement, adoption-on-restart) use each component's own existing logging conventions (the Controller's `WARN`/`ERROR` macros, the Node Manager's equivalent) — no new logging mechanism is introduced.
+
+## Testing Strategy
+
+### Unit Tests
+- `component_supervisor_start` adopts an already-running, matching process instead of spawning a duplicate when a valid Process Record exists, and starts fresh when the record is missing/stale/mismatched — directly exercises Requirement 7.1.
+- `component_supervisor_start`/`_replace` distinguishes an intentional stop from an unexpected exit, restarting only the latter — exercises Requirement 2.1-2.3.
+- `SchedulerDesiredStatePolicy` evaluates "running" only when the Controller's last-reported status is primary and not unresponsive, "stopped" otherwise (standby, overdue, or never-seen) — exercises Requirement 3.1-3.3.
+- `liveness_monitor_check` (heartbeat-based) marks a role unresponsive only once its sequence number has stalled past the threshold, not on a single delayed tick — exercises Requirement 4.2.
+- `xce_heartbeat_report`'s send path never blocks when the receiving socket is absent or its buffer is full — exercises Requirement 5.1, 5.2.
+
+### Integration Tests
+- A real 3-node Controller cluster promotion results in `xce-node-manager` starting a Scheduler process on the newly-primary node with no manual step, and the prior primary's `xce-node-manager` stopping its Scheduler within the bounded timeout — exercises Requirement 3.1-3.4 end to end.
+- A hard-killed primary Controller (bug-1760's original repro) results in the orphaned Scheduler self-exiting via `SchedOwnerWatch` even before `xce-node-manager` would otherwise notice, and the newly-elected primary's `xce-node-manager` starting a fresh Scheduler that successfully dispatches a job submitted after the failover — exercises Requirement 6.1-6.3 and directly closes bug-1760.
+- A Scheduler artificially wedged after dequeuing a job (e.g. blocked inside the allocator, past the `cbuf_timedget()` call) is detected as unresponsive and replaced; a Scheduler left genuinely idle (no jobs submitted) for longer than the threshold is NOT falsely flagged, confirming both halves of Requirement 4.3 — the wedge case and the false-positive-avoidance case the naive design would have failed.
+- A `xce-node-manager` process killed and restarted while its Controller and Scheduler keep running adopts both without starting duplicates — exercises Requirement 7.1.
+- A job dispatched by a Scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement Scheduler to consider — exercises Requirement 3.5.
+
+### E2E / Soak Tests
+- Extend `.claude/skills/xce-test-ha/run-ha-soak.sh`'s sequential and chaotic modes to assert a working Scheduler, Worker, and Apiserver are present after every round with no harness-side start/stop action for any of them — replaces this session's own bug-1760 workaround, which becomes dead code once this feature ships (Requirement 8.2).
+- A rapid sequence of Controller promotions/demotions produces no more than one Scheduler spawn/stop cycle per genuine promotion, with no orphaned or duplicate processes remaining after the sequence settles (Requirement 3.6, 7.1).
+
+## Security Considerations
+
+**Privilege model is role-conditional, not uniform.** `xce-deploy-ha`'s own tooling documents that the real (non-virtual) Worker daemon (`xcecd`, as opposed to the `xcecd_virtual` test double) runs as root — "plain kill can't touch it," requiring `sudo`. `xce-node-manager` must therefore run as root on any node whose assigned role includes `worker`, so `ComponentSupervisor`'s `posix_spawn`/`kill`/`waitpid` calls against `xcecd` need no special-cased privilege escalation; on a node whose roles are limited to manager/apiserver/reverse-proxy, `xce-node-manager` runs as an ordinary service account. This is a per-node deployment decision (how the host's own service manager launches `xce_nodemgr`), not a per-role runtime toggle inside the binary itself — the Node Role Manifest already fixes a node's roles at Node Manager startup, so the required privilege level is knowable at that same point.
+
+The heartbeat socket is a new local IPC surface whose payload (in particular the Controller's reported primary/standby status) influences whether `xce-node-manager` starts or stops a component — a local process able to write to it could send spoofed reports. Consistent with `product.md`'s already-locked decision to trust the OS boundary rather than add application-layer encryption (`chmod 0700`, no new crypto), the heartbeat socket is created with permissions restricting it to the OS user `xce-node-manager` itself runs as on that node (root on a worker-role node, matching `xcecd`'s own existing requirement; an ordinary service account otherwise) — the same trust boundary the WAL/checkpoint directories already rely on. No new authentication mechanism is introduced; a host already compromised at that privilege level can already directly signal/kill the same processes this feature manages, so the socket adds no new privilege-escalation surface beyond what local code execution at that level already grants. On a worker-role node specifically, running `xce-node-manager` (and therefore the socket) as root is a direct consequence of `xcecd`'s own pre-existing requirement, not a new exposure this feature introduces.
+
+## Migration Strategy
+
+```mermaid
+flowchart TB
+    Phase1[Phase 1 build xce nodemgr and the libxce heartbeat primitive behind no behavior change]
+    Phase2[Phase 2 wire heartbeat calls into controller scheduler worker apiserver loops]
+    Phase3[Phase 3 add the entry.sh manager case and update xce deploy ha and xce test ha tooling]
+    Phase4[Phase 4 cut over real deployments to node manager driven startup]
+    Phase1 --> Phase2 --> Phase3 --> Phase4
+```
+- Rollback trigger: if a real deployment depends on placing a component on a host with no Node Manager present for reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (cutover) rather than force it through, since Phase 4 is the first phase that changes how a real deployment is operated.
+- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
+
+## Open Questions / Risks
+- The exact Node Role Manifest file format (YAML to match `src/configs/*.yaml`'s existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape.
+- The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds.
+- `tools/appimage/entry.sh` bundles `haproxy` today with no dispatch case reaching it at all — this feature is what first wires it up (via `xce-node-manager` invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve.
+
+Resolved during design review (kept here for traceability, not left as open items): the Scheduler round-loop idle-cadence question was confirmed against the actual code (`cbuf_get()`'s unbounded wait) and resolved via `cbuf_timedget()` — see Architecture and File Structure Plan; it does not need re-verification at implementation time. The Worker-node root-privilege gap was resolved as a role-conditional privilege model — see Security Considerations.

.kiro/specs/xce-node-manager/requirements.md

@@ -0,0 +1,101 @@
+# Requirements Document
+
+## Project Description (Input)
+xce-node-manager: a new per-node XCE daemon that supervises the lifecycle of the other XCE components on its host, replacing the now-superseded scheduler-primary-lifecycle spec's controller-owns-its-scheduler mechanism.
+
+Who has the problem: XCE cluster operators, and by extension every job that depends on the scheduler being reachable after a controller failover (this carries forward bug-1760 from .wolf/buglog.json -- a hard-killed primary left its scheduler orphaned with no rejoin path, and a job submitted to the new primary sat stuck PEND). More generally, there is today no unified process-lifecycle/health-supervision layer for any XCE component (controller, scheduler, apiserver, worker, reverse proxy) -- each is started independently with no shared liveness protocol.
+
+What should change: introduce xce-node-manager, one instance deployed per host, that reads a declared role assignment for its own node and starts/supervises the XCE component(s) that role requires:
+- manager node role: starts the controller unconditionally; starts the scheduler only while that node's own controller instance is currently reporting itself primary, and stops the scheduler when that controller reports standby (or stops heartbeating).
+- worker node role: starts the worker.
+- apiserver node role: starts the apiserver.
+- reverse-proxy node role: the reverse proxy is an off-the-shelf process (e.g. nginx/haproxy), not an XCE-authored binary -- xce-node-manager health-checks it via a TCP port check rather than a heartbeat, but still starts/restarts it per this node's role.
+
+Health/liveness protocol: each XCE-authored managed component (controller, scheduler, worker, apiserver) heartbeats to its own node's local xce-node-manager over a Unix domain socket (same-host IPC only, no cross-host communication in this protocol). Most components send a simple liveness heartbeat ('I'm alive'). The controller's heartbeat additionally carries its current HA role state ('I'm primary' / 'I'm standby') -- this is the sole signal xce-node-manager uses to decide whether to spawn or kill the scheduler on a manager node; no separate election-awareness or Raft-state polling is introduced in xce-node-manager itself. The heartbeat client-side logic should live in libxce (src/lib/), which is already linked by controller/worker/scheduler in C and by the Go apiserver via CGO, so every managed component can share one implementation rather than reimplementing the protocol per-language.
+
+This directly supersedes .kiro/specs/scheduler-primary-lifecycle/ (see that spec's requirements.md and design.md, now marked superseded in its spec.json): that spec gave the controller itself the job of directly posix_spawn-ing and killing its own scheduler child process (scheduler_proc.c/ha.c). In the new model, that responsibility moves entirely to xce-node-manager, driven by the controller's primary/standby heartbeat rather than the controller managing a child process itself. The underlying requirements that spec captured should carry forward and be re-expressed against this new mechanism: the scheduler must start automatically on promotion with no manual step; a scheduler tied to a controller that gracefully demotes must be stopped cleanly without blocking the controller's own role transition; a scheduler that is alive-but-wedged (connected, making no scheduling progress) must be detected and replaced; no job may be lost or left inconsistent when a scheduler is terminated for any reason; and the controller's own availability/correctness must never depend on xce-node-manager or on its scheduler child's health. Additionally, xce-node-manager's own liveness/health-check responsibility now generalizes to worker and apiserver, not just the scheduler.
+
+Constraints and prior art to ground design in: .kiro/specs/scheduler-primary-lifecycle/design.md's Technology Stack section already resolved a real fork()-vs-posix_spawn mutex-inheritance-deadlock hazard for spawning a child process from a multithreaded daemon -- that finding (use posix_spawn, not fork()+exec()) applies equally to xce-node-manager spawning the scheduler and should not be re-litigated from scratch. src/lib/CLAUDE.md describes libxce's existing role as the shared IPC/utility library across controller, worker, scheduler, and (via CGO) the Go apiserver. src/worker/src/process_worker_manager.c is existing precedent in this codebase for a daemon spawning/supervising child processes (for job execution) and should be examined for reusable patterns even though it manages job processes rather than other XCE daemons. .kiro/steering/ha-async-io.md's controller-never-self-terminate invariant and iomgr/delaymgr non-blocking thread-role contract still apply to however the controller reports its primary/standby state to xce-node-manager -- that reporting path must never introduce blocking into a liveness-carrying controller thread. Existing deployment tooling (.claude/skills/xce-deploy-ha/SKILL.md, .claude/skills/xce-test-ha/run-ha-soak.sh) currently starts each XCE component independently and will need reconciling with node-role-driven startup via xce-node-manager, the same tooling-reconciliation concern scheduler-primary-lifecycle's requirements.md already flagged.
+
+Open design questions to resolve during requirements/design phases (do not assume answers): how a node's role assignment is declared/distributed (static per-node config file vs. some cluster-wide source of truth); what happens if xce-node-manager itself crashes or is not running (who restarts it -- likely out of scope, delegated to the OS service manager, but must be stated explicitly); the exact heartbeat cadence and unresponsiveness threshold for each component type; whether xce-node-manager is a new src/ top-level component or lives under an existing directory per src/structure.md conventions; and how much of scheduler-primary-lifecycle's already-designed SchedOwnerWatch self-monitoring behavior (scheduler watching its own controller directly) is still needed as a redundant backstop versus fully replaced by xce-node-manager-driven termination.
+
+## Boundary Context
+
+- **In scope**: node-role-driven startup and continuous supervision of the Controller, Scheduler Process, Worker Process, and Apiserver Process; binding the Scheduler Process's start/stop to its Controller's primary/standby status; heartbeat-based liveness detection that distinguishes real progress from bare connectivity where that distinction matters; a distinct liveness check for the off-the-shelf Reverse Proxy Process; reconciling existing deployment tooling and specs that assume independently-managed component processes.
+- **Out of scope**: supervising the Node Manager process itself (delegated to the host's existing process-supervision facilities, e.g. systemd or an equivalent service manager — this feature does not build that); the Reverse Proxy Process's own configuration, routing rules, or request handling; any change to how a Worker Process's or Apiserver Process's own client-side connection discovers or reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism, unchanged by this feature); any change to RBAC, authentication, or job/queue/scheduling decision logic; any new operator-facing remote management API for the Node Manager beyond existing logging conventions.
+- **Adjacent expectations**: this feature depends on the Controller's existing Raft-based primary/standby determination and the existing `ha_dispatch_gate` non-primary write-rejection behavior, both unchanged by this feature. This feature supersedes `.kiro/specs/scheduler-primary-lifecycle/`'s approach to scheduler-lifecycle management (a Controller directly owning its own scheduler child process) — the scheduler-availability requirements that spec captured (no manual scheduler startup, prompt shutdown on demotion, wedge detection, no job loss on scheduler termination) are restated here against the Node Manager-based mechanism instead. Existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and any spec assuming an independently-started, reconnecting component process is an adjacent consumer that must be revisited, not an unstated exception to this feature's behavior.
+
+## Requirements
+
+### Requirement 1: Node role assignment and component startup
+**Objective:** As a cluster operator, I want each node to automatically start the right XCE component(s) for its assigned role, so that deploying a node consists of assigning it a role rather than manually starting and managing individual XCE processes.
+
+#### Acceptance Criteria
+1. When the Node Manager starts on a node, the Node Manager shall determine which role or roles are assigned to that node without depending on connectivity to any Controller or other cluster component.
+2. The Node Manager shall support, at minimum, the following node roles: manager (runs the Controller, and the Scheduler Process while that Controller is primary), worker (runs the Worker Process), apiserver (runs the Apiserver Process), and reverse-proxy (runs an externally-provided Reverse Proxy Process).
+3. When the Node Manager starts on a node assigned the manager role, the Node Manager shall start the Controller for that node.
+4. When the Node Manager starts on a node assigned the worker, apiserver, or reverse-proxy role, the Node Manager shall start the corresponding component for that node.
+5. If a node is assigned more than one role, the Node Manager shall start the component for each assigned role.
+6. If the Node Manager cannot start a component required by the node's assigned role, the Node Manager shall record an operator-visible error and continue attempting to start it.
+
+### Requirement 2: Continuous supervision and restart of always-on components
+**Objective:** As a cluster operator, I want a component that is supposed to always be running on a node to be restarted automatically if it stops unexpectedly, so that a crashed Worker Process, Apiserver Process, Controller, or Reverse Proxy Process does not require manual intervention to recover.
+
+#### Acceptance Criteria
+1. While a node's assigned role requires a component to be continuously running (Controller, Worker Process, Apiserver Process, or Reverse Proxy Process), the Node Manager shall treat an unexpected exit of that component's process as a failure.
+2. When the Node Manager detects that a continuously-required component's process has exited unexpectedly, the Node Manager shall start a replacement process for it.
+3. The Node Manager shall not restart a component process that it stopped itself as part of an intentional, requested action.
+4. The Node Manager shall log an operator-visible event whenever it restarts a component due to an unexpected exit.
+
+### Requirement 3: Scheduler lifecycle bound to Controller primary/standby status
+**Objective:** As a cluster operator, I want the scheduler on a manager node to run only while that node's Controller is primary, so that a Scheduler Process never keeps running on behalf of a Controller that is no longer authoritative, and the cluster always has a working scheduler after a failover without manual intervention.
+
+#### Acceptance Criteria
+1. While a node's Controller reports itself as primary, the Node Manager shall ensure exactly one Scheduler Process associated with that Controller is running.
+2. When a node's Controller transitions from primary to standby, the Node Manager shall stop the Scheduler Process associated with that Controller.
+3. When a node's Controller becomes unreachable to the Node Manager, the Node Manager shall stop the Scheduler Process associated with that Controller.
+4. The Node Manager shall stop a Scheduler Process associated with a Controller within a bounded time of that Controller ceasing to be primary or becoming unreachable, without requiring the Controller's own role transition to wait for the Scheduler Process to exit.
+5. The Node Manager shall ensure that no job a stopped Scheduler Process had not yet durably recorded is lost or left in an inconsistent state, regardless of why that Scheduler Process was stopped (normal stop request, Controller loss, unresponsiveness, or crash).
+6. If a node's Controller is primary but has no Scheduler Process currently running for it, the Node Manager shall keep attempting to start one while that Controller remains primary.
+
+### Requirement 4: Liveness and health detection of managed components
+**Objective:** As a cluster operator, I want the Node Manager to detect when a managed component has stopped doing useful work, not merely whether its process still exists, so that a hung or wedged component is recovered instead of silently leaving jobs unserved.
+
+#### Acceptance Criteria
+1. While a managed XCE component's process is running, that component shall periodically report its liveness to its node's Node Manager.
+2. If a managed component does not report its liveness within an operator-configurable threshold, the Node Manager shall treat that component as unresponsive.
+3. While the Scheduler Process is running, the liveness it reports to the Node Manager shall reflect whether it is making real scheduling progress, not merely that its process is still running.
+4. When the Node Manager treats a managed component as unresponsive, the Node Manager shall stop that component and start a replacement, following the same startup behavior as if it had exited unexpectedly.
+5. Where the managed component is the Reverse Proxy Process, the Node Manager shall determine its liveness by checking reachability of its configured network port instead of a liveness report from the process itself.
+6. The Node Manager shall log an operator-visible event whenever it treats a managed component as unresponsive and replaces it.
+
+### Requirement 5: Non-interference with a managed component's own responsibilities
+**Objective:** As a cluster operator, I want reporting liveness/role status to the Node Manager to never come at the cost of a component's primary responsibilities, so that adopting Node Manager supervision cannot itself become a new source of cluster unavailability.
+
+#### Acceptance Criteria
+1. A managed component's reporting of its liveness or role status to the Node Manager shall never block or delay that component's own primary processing responsibilities (for example, a Controller's handling of heartbeats or election timeouts).
+2. If reporting liveness or role status to the Node Manager fails for any reason, the reporting component shall continue operating normally.
+3. The absence of a running or reachable Node Manager shall not prevent a Controller from being elected primary, remaining primary, or serving its existing responsibilities.
+
+### Requirement 6: Independent verification of controlling authority (backstop)
+**Objective:** As a cluster operator, I want the scheduler to independently notice when its own controller is no longer a live primary, so that a Scheduler Process can never keep running detached from any authoritative Controller even if the Node Manager that started it becomes unavailable.
+
+#### Acceptance Criteria
+1. While a Scheduler Process is running, the Scheduler Process shall independently verify that the specific Controller instance that started it is reachable and still holds the primary role.
+2. When a Scheduler Process can no longer confirm that its originating Controller instance is reachable and primary, the Scheduler Process shall terminate on its own, independent of any action by the Node Manager.
+3. The Scheduler Process shall not attempt to join, rejoin, or continue operating against any Controller instance other than the one that started it.
+
+### Requirement 7: Node Manager recovery without duplicate supervision
+**Objective:** As a cluster operator, I want a restarted Node Manager to recognize the components already running on its node instead of starting duplicates, so that the Node Manager's own restart does not cause process thrashing or resource conflicts.
+
+#### Acceptance Criteria
+1. When the Node Manager starts and a component required by the node's assigned role is already running, the Node Manager shall recognize it as already running rather than starting a duplicate process for it.
+2. The Node Manager shall log an operator-visible event whenever it adopts an already-running component on startup instead of starting a new one.
+
+### Requirement 8: Compatibility with existing deployment tooling and specs
+**Objective:** As a cluster operator or spec author relying on existing deployment tooling, I want the introduction of node-role-driven component startup to be reflected everywhere a component's lifecycle is currently assumed to be independently and manually managed, so that existing deployments, tests, and specs do not silently break or become misleading.
+
+#### Acceptance Criteria
+1. Where existing deployment tooling documents starting the Controller, Scheduler Process, Worker Process, or Apiserver Process as an independent, manually-managed process, that tooling shall be updated to reflect that the Node Manager now starts these components according to each node's assigned role.
+2. Where an existing spec or test tooling depends on a component being started or managed independently of the Node Manager, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
+3. The observable behavior of the Controller, Scheduler Process, Worker Process, and Apiserver Process toward jobs, queues, configuration, and cluster membership shall remain unchanged by this feature while each is running.

.kiro/specs/xce-node-manager/research.md

@@ -0,0 +1,79 @@
+# Research & Design Decisions
+
+## Summary
+- **Feature**: `xce-node-manager`
+- **Discovery Scope**: Complex Integration — new top-level daemon, no direct precedent, but heavily grounded in existing patterns across controller/worker/scheduler/apiserver and the AppImage packaging layer.
+- **Key Findings**:
+  - `tools/appimage/entry.sh` is a thin per-role dispatcher (`xce <role> args...`) that execs exactly one of `xce_controller`/`xcecd`/`xce_scheduler`/`xce_apiserver` in the foreground — there is no existing orchestration layer to extend or conflict with, and `haproxy` (already bundled in the AppImage per `build-deployment.md`) has no dispatch case at all today, i.e. it is currently unreachable through `entry.sh`.
+  - `src/worker/src/process_worker_manager.c`'s existing `process_manager_start_worker()` spawns via raw `fork()+execl()`, not `posix_spawn()` — an already-shipped instance of the exact mutex-inheritance-deadlock hazard class `product.md`'s locked decision ("no `fork()` for snapshots") and the superseded `scheduler-primary-lifecycle` design already ruled out. Not fixed here (out of scope, unrelated component) but explicitly NOT a pattern to imitate.
+  - The controller already has a non-blocking, delaymgr-driven, role-aware periodic task (`ha_start_primary_heartbeat`/`primary_heartbeat_fn`, `ha.c:9949`) reading the atomic `ha->role` every tick — proof that a cheap, always-safe periodic local report is a well-trodden pattern in this codebase, though this specific function is gated on first-promotion (`s_armed`) and therefore unsuitable to piggyback on directly (see Decision: Heartbeat Call Site).
+  - `src/configs/hosts.yaml` (the cluster's existing per-host config, distributed via `OPCode_replConfigPush`) describes scheduler-visible compute-host attributes, not deployment/process-placement roles — confirms no existing config surface for "which node runs which XCE component," and that any such config cannot be cluster-distributed (a node's role must be knowable before any controller is reachable).
+
+## Research Log
+
+### Existing multi-component launch/packaging surface
+- **Context**: Before designing a new orchestrator, check whether one already exists in disguise (AppImage bundling, systemd units, an existing launcher script).
+- **Sources Consulted**: `tools/appimage/entry.sh`, `build-deployment.md` (AppImage/Production Artifacts sections), `xce-deploy-ha` SKILL.md, `xce-test-ha/run-ha-soak.sh`.
+- **Findings**: The AppImage bundles all daemons plus `haproxy` behind one `xce <role>` entry point, but `entry.sh` only launches a single named process in the foreground per invocation — no supervision, no multi-process orchestration, no config-driven role resolution. The local dev/test harness (`xce-deploy-ha`) generates one shell script per component (`start.sh`, `start-worker.sh`, `start-scheduler.sh`), started manually/independently — the same "independently-started component" assumption `scheduler-primary-lifecycle`'s requirements already flagged, now generalized to worker/apiserver too.
+- **Implications**: `xce-node-manager` is additive, not competing with an existing mechanism. The natural integration point is one new `entry.sh` case (`manager`) that execs the new `xce_nodemgr` binary; `xce_nodemgr` itself then resolves and spawns the other bundled binaries directly (same `which xce_controller`-style PATH resolution `entry.sh` already uses), including finally wiring up the previously-unreachable bundled `haproxy` for the reverse-proxy role.
+
+### Controller-side periodic reporting hook
+- **Context**: Where should the Controller's "I'm alive / I'm primary or standby" report be triggered from, without touching the promotion/demotion callback functions those steering docs flag as the riskiest, least-covered code in the controller?
+- **Sources Consulted**: `ha.c` (`primary_heartbeat_fn`, `ha_start_primary_heartbeat`, `raft_cb_become_leader`/`raft_cb_become_follower` — via `codegraph_explore`), `ha-async-io.md`.
+- **Findings**: `primary_heartbeat_fn` is armed only once, on a controller instance's *first* promotion (`s_armed` guard) — a controller that has never won an election never runs it, which would leave `xce-node-manager` unable to distinguish "controller alive, still standby" from "controller process is down" on any node that hasn't yet been promoted. `raft_cb_become_leader`/`raft_cb_become_follower` themselves are flagged with no direct test coverage and are the single most call-dense, safety-critical functions in the controller (122+ calls from `raft_cb_become_leader` alone).
+- **Implications**: The report must come from a small, independent, unconditionally-armed periodic point tied to the Controller's own process lifetime (not gated on promotion), reading the same atomic `ha->role` field `primary_heartbeat_fn` already reads. This adds zero new logic to `raft_cb_become_leader`/`raft_cb_become_follower` — a strict reduction in touched surface area compared to the superseded spec, which called full spawn/monitor orchestration from inside `raft_cb_become_leader` directly.
+
+### Scheduler wedge-detection signal shape
+- **Context**: The superseded spec's Requirement 5 (carried forward as 4.3 here) requires detecting a scheduler that is connected but not making real scheduling progress — a bare "I'm alive" ping cannot distinguish a wedged decision-loop from a healthy one if the two run on different threads.
+- **Sources Consulted**: In-conversation review of `scheduler-primary-lifecycle`'s `design.md` (`SchedulerConnectionBookkeeping`'s last-progress-timestamp approach, fed from real dispatch/pend-reason traffic, not a synthetic heartbeat); bug-1760's own root-cause text describing the scheduler's *outbound wire traffic* as reactive/idle-silent ("the scheduler has nothing queued to publish").
+- **Findings**: Reusing the scheduler's reactive wire-publish call as the heartbeat trigger would reintroduce the exact idle-silence failure mode bug-1760 already exposed. The correct trigger is the scheduler's own internal round/picker loop iteration — which must tick on a bounded cadence regardless of whether a round finds any job to act on — not the wire-publish call.
+- **Implications**: One wire-format field (a monotonically increasing sequence number) serves double duty: for the Controller/Worker/Apiserver it is a bare liveness counter; for the Scheduler, since it is incremented once per round-loop iteration (not once per successful publish), its advancement *is* the real-progress signal Requirement 4.3 needs, with no separate progress field required. See Decision: Unified Heartbeat Wire Format.
+
+## Architecture Pattern Evaluation
+
+| Option | Description | Strengths | Risks / Limitations | Notes |
+|--------|-------------|-----------|---------------------|-------|
+| Adopt a generic external supervisor (systemd user units, s6, runit, supervisord) for all four roles | Delegate process start/restart entirely to an existing, battle-tested tool | Zero new code for the generic "keep running" case (Requirement 2) | Cannot express Requirement 3's Controller-role-driven conditional scheduler start/stop or Requirement 4.3's progress-aware wedge detection — those are XCE-domain-specific decisions no generic supervisor can make; also conflicts with the AppImage's self-contained, zero-external-dependency portability goal (`product.md`) across arbitrary air-gapped target hosts that may not run systemd | Rejected as the primary mechanism; the generic host service manager is still the right tool for the one thing left over — restarting `xce-node-manager` itself, already scoped out of this feature (Requirement 7 boundary) |
+| Custom `xce-node-manager` daemon using only POSIX primitives (`posix_spawn`, Unix domain datagram sockets), consistent with the already-reviewed `scheduler-primary-lifecycle` precedent | Build a small, purpose-specific supervisor | Can express both the generic and Controller-role-driven policies; no new external runtime dependency; reuses the `posix_spawn` decision already vetted against the mutex-inheritance-deadlock hazard | New component to build and test | **Selected** |
+
+## Design Decisions
+
+### Decision: Heartbeat call site is each component's own natural loop iteration, not a new thread
+- **Context**: Requirements 4.1 and 4.3 require every managed component to report liveness, and require the Scheduler's report specifically to reflect real progress.
+- **Alternatives Considered**:
+  1. A dedicated heartbeat-sender thread per component, independent of its main work loop — simplest to bolt on, but structurally cannot distinguish a wedged work thread from a healthy one (the sender thread keeps reporting regardless).
+  2. Piggyback the Controller's report on the existing `primary_heartbeat_fn` — rejected, see Research Log (promotion-gated arming leaves never-yet-primary controllers silent).
+- **Selected Approach**: Each component calls a shared `libxce` heartbeat primitive once per iteration of its own existing loop (the Controller's iomgr/delaymgr-driven event loop, the Scheduler's round loop, the Worker's `ipc_worker_run`/`usleep` loop); the Apiserver, having no persistent loop of its own (a request-driven Gin server), uses a small dedicated ticker goroutine instead — the one place a new "thread" is unavoidable, since there is no existing loop to hook into.
+- **Rationale**: Guarantees the report can only advance when the component's own real work-loop is actually turning, which is what makes the same signal double as both liveness and (for the Scheduler) progress. The Scheduler's live driver (`scheduler.c`, `sched_job_picker`) is the correct call site, not the pure decision engine in `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` — the `scheduler-core` spec's own core/driver separation already keeps the core free of controller I/O, and this feature's local heartbeat is new outside-facing I/O that belongs on the same side of that boundary as the existing `xcecli` connection. The Scheduler's separate simulation-mode driver (`sim.c`) gets no heartbeat call site at all and stays entirely outside `xce-node-manager`'s awareness (see Non-Goals in `design.md`) — it is a standalone, manually-invoked tool with no Controller or primary/standby concept.
+- **Trade-offs**: Slightly different integration shape per component (loop hook vs. ticker) instead of one uniform mechanism; acceptable since the underlying wire format and `libxce` primitive are identical either way.
+- **Follow-up resolved during design review** (not deferred to implementation): `codegraph_explore` confirmed `sched_job_picker` calls `cbuf_get()` (`src/lib/mylock.c:144`), an **unbounded** `pthread_cond_wait` when the job queue is empty — a heartbeat placed naively at the top of that loop would never fire during genuine idle periods, causing `xce-node-manager` to falsely treat a healthy idle Scheduler as wedged and kill it. Fix: swap to `cbuf_timedget()` (same file, an existing bounded-deadline variant already used elsewhere in the codebase) with a bounded idle-tick deadline, and call the heartbeat primitive on both the real-job path and the timeout path.
+
+### Decision: Unified heartbeat wire format, self-identifying payload, no CGO for the Apiserver client
+- **Context**: The heartbeat protocol must be usable by four component types across two languages (C daemons, Go apiserver) over one shared local socket, with the receiving `xce-node-manager` able to attribute each datagram without relying on socket-level peer identity (a `SOCK_DGRAM` Unix socket exposes no reliable peer address unless the sender also binds, which adds needless coupling).
+- **Alternatives Considered**:
+  1. Route the Apiserver's heartbeat through `libxce` via the existing CGO bridge, matching how `apiserver`'s C-tier calls already work.
+  2. A small, independently-implemented Go client speaking the identical wire format natively, with no CGO involved.
+- **Selected Approach**: One canonical fixed-layout binary wire format (magic/version, role, status, pid, sequence number — no Cap'n Proto), implemented once in `libxce` for the three C daemons (which already link it natively, zero extra cost), and once as a small native Go implementation for the Apiserver.
+- **Rationale**: The payload is a fixed, trivial, few-field structure — reimplementing it natively in Go is a handful of lines with negligible drift risk, and avoids pulling a CGO dependency into a code path (a periodic status ticker) that has no other reason to touch the C tier; `tech.md` already documents that Apiserver's own unit tests deliberately run `CGO_ENABLED=0`, and only the existing backend bridge work requires the CGO build. The payload self-identifies its sender (role + pid fields), so the receiver never depends on socket-level peer identity. This is a deliberate, narrow exception to `structure.md`'s "Cap'n Proto for wire formats" convention: that convention targets the real cross-host, cross-version network protocol (`controller.capnp`/`c.capnp`), where schema evolution and cross-language generated bindings earn their cost; this protocol is same-host-only, always emitted and consumed by binaries from the same build (Node Manager and the components it manages ship together), and confirmed (`go.mod`/`go.sum` inspection) that the Go apiserver carries no existing capnp dependency at all — adopting Cap'n Proto here would mean a new Go dependency purely for a fixed 4-field local datagram, which the Simplification lens rejects as disproportionate.
+- **Trade-offs**: Two implementations of the same format instead of one; mitigated by the format's small size and by both being validated against the same contract in this design.
+- **Follow-up**: none — the format is fixed at design time (see Data Models in `design.md`).
+
+### Decision: Node Manager privilege is role-conditional (root only on worker-role nodes)
+- **Context**: `xce-deploy-ha`'s own tooling documents that the real (non-virtual) Worker daemon `xcecd` runs as root ("plain kill can't touch it," needs `sudo`) — discovered during design review, not anticipated during requirements. A uniform single-OS-user assumption for `xce-node-manager` (as first drafted) would leave it unable to start/stop/signal `xcecd` on a real worker-role node.
+- **Alternatives Considered**:
+  1. `xce-node-manager` always runs as root everywhere, regardless of role.
+  2. `xce-node-manager` stays unprivileged everywhere; grant it a narrow Linux capability or scoped `sudo` rule for the worker case specifically.
+  3. Treat real (root-requiring) worker management as out of scope for this spec; only support `xcecd_virtual`.
+- **Selected Approach** (user decision): `xce-node-manager` runs as root only on a node whose assigned role includes `worker`; on manager/apiserver/reverse-proxy-only nodes it runs as an ordinary service account. This is fixed by the Node Role Manifest at Node Manager startup, not a runtime toggle.
+- **Rationale**: Matches `xcecd`'s own pre-existing requirement exactly, with no new privilege-escalation surface beyond what that daemon already needs; simpler than a capability-based scheme, and avoids narrowing this spec's worker-role support to the virtual test double only.
+- **Trade-offs**: The heartbeat socket's permission model is now per-node-role rather than one uniform story (see `design.md` Security Considerations) — an implementer must not assume a single OS user across all node roles.
+- **Follow-up**: none — fixed at design time.
+
+## Risks & Mitigations
+- A node-manager crash on a manager node strands its Controller's Scheduler with nothing left to stop it on a subsequent demotion — mitigated by retaining a scheduler-side independent controller-liveness check as a backstop (Requirement 6), so the Scheduler never depends solely on `xce-node-manager` staying alive.
+- `xce-node-manager` restarting could double-spawn a component that is already running — mitigated by a small persisted per-role record (role, pid, start time, binary path) it consults on startup before spawning (see `design.md` Data Models); a stale/mismatched record falls back to a fresh start rather than blindly trusting the file, and each component's own listen-port bind-failure is a natural backstop against an accidental true duplicate.
+- Sizing the heartbeat interval/timeout thresholds is an implementation-time tuning decision (same posture the superseded spec already took for its own liveness thresholds), not a design decision.
+
+## References
+- `.kiro/specs/scheduler-primary-lifecycle/design.md` — superseded design; source of the `posix_spawn` mutex-inheritance-deadlock finding and the wedge-detection real-progress principle, both carried forward here.
+- `.wolf/buglog.json` bug-1760 — motivating incident for the scheduler-lifecycle requirements this spec restates.

.kiro/specs/xce-node-manager/spec.json

@@ -0,0 +1,25 @@
+{
+  "feature_name": "xce-node-manager",
+  "created_at": "2026-09-03T03:45:42Z",
+  "updated_at": "2026-09-03T06:38:35Z",
+  "language": "en",
+  "phase": "design-generated",
+  "supersedes": "scheduler-primary-lifecycle",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true,
+      "validation_note": "kiro-spec-requirements 2026-09-02: 8 requirement areas / 32 acceptance criteria drafted, grounded in in-conversation codebase investigation (ha.c primary_heartbeat_fn/ha->role, process_worker_manager.c's existing fork()-hazard precedent, process_manager.c/h supervision shape, controller_connector.c candidate-list pattern, hosts.yaml config format, xce-deploy-ha tooling, product.md/structure.md/tech.md/roadmap.md steering) plus the fully-read superseded scheduler-primary-lifecycle design.md. Requirements Review Gate run inline during drafting (not as a separate pass): mechanical checks (numeric IDs, EARS-format ACs, no implementation-language leakage) passed; two draft issues caught and fixed before writing -- Requirement 3's original job-durability AC used a non-EARS 'Regardless of...' opening (rewritten to a ubiquitous 'The Node Manager shall ensure...' form), and Requirement 7 originally included a non-AC scope-exclusion bullet (moved to Boundary Context's Out of scope list, replaced with a proper adoption-logging AC). Auto-approved 2026-09-03 via `/kiro-spec-design xce-node-manager -y`."
+    },
+    "design": {
+      "generated": true,
+      "approved": true,
+      "validation_note": "kiro-spec-design 2026-09-03: light-to-moderate discovery, see prior note for discovery findings and synthesis (build-vs-adopt rejection of a generic external supervisor; ComponentSupervisor/LivenessMonitor generalization; justified Cap'n Proto deviation). kiro-validate-design 2026-09-03: independent critical review (not the same pass as authoring) found 3 real issues, GO withheld pending fixes -- (1) uniform single-OS-user privilege assumption was factually wrong: xce-deploy-ha's own tooling documents the real (non-virtual) xcecd worker daemon runs as root, unaddressed by the original design; resolved per user decision -- xce-node-manager runs as root only on a node whose assigned role includes worker, ordinary service account otherwise, Security Considerations and research.md's Decision log rewritten accordingly. (2) Requirement 4.3's wedge-detection mechanism depended on the Scheduler's round loop ticking on a bounded cadence even when idle, left as an unverified 'confirm during implementation' open question despite research.md's own cited evidence (bug-1760's reactive/idle-silent root cause) pointing the other way; verified directly against the actual code via codegraph_explore -- sched_job_picker's cbuf_get() (src/lib/mylock.c:144) does an UNBOUNDED pthread_cond_wait when idle, confirming the naive design would cause false-positive wedge detection on every idle Scheduler. Fixed: swap to the already-existing cbuf_timedget() bounded-deadline variant with heartbeat calls on both the real-job and timeout paths; documented as a resolved design decision, not deferred. (3) PortCheckMonitor's TCP connect was never specified as non-blocking despite HeartbeatListener explicitly being so, and ComponentSupervisor's own text incorrectly claimed the Node Manager loop 'has no other liveness-critical responsibility to protect' -- both corrected: PortCheckMonitor's connect is now explicitly non-blocking (O_NONBLOCK + bounded poll/select), and the misleading framing was rewritten to state the loop IS liveness-critical for every role it supervises. Two further scope clarifications volunteered by the user during the fix cycle, applied directly: the heartbeat call site is explicitly the Scheduler's live driver (scheduler.c/sched_job_picker) only, never the pure core (core.c/core_events.c/core_session.c/core_clock.c, scheduler-core spec's own zero-controller-I/O invariant, now cited as a fourth cross-checked project invariant) or the simulation driver (sim.c, added as an explicit Non-Goal/Out-of-Boundary item -- simulation-mode Scheduler runs standalone, entirely outside xce-node-manager's awareness). All fixes applied directly to design.md/research.md in the same session; re-reviewed inline, no further issues found. GO."
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false
+}