docs(worker-rejoin-resync): fix design review findings; add tasks.md

Design validation (2026-09-01) found and fixed 4 issues: a stale bug-1678
citation blocking reuse of persist_submit_job_dispatch_fact, missing
XCE_WORKER_RECONCILING accounting against scheduler-dispatch-commit's
dispatch-fail retry/terminal-fail classification, bug-1631 reactivation
(job.allocs never released on RUN->PEND reconciliation), and an eager-vs-
deferred mutation inconsistency between the new job.c functions and the
bug-1679 explicit-notify precedent. design.approved and tasks.approved
both set; tasks.md drafted (7 major/19 sub-tasks) and passed two rounds
of independent task-graph sanity review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku

.kiro/specs/worker-rejoin-resync/design.md

@@ -6,7 +6,7 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
 
 **Users**: Internal controller/worker/scheduler subsystems — there is no end-user-facing surface. Cluster operators benefit indirectly: jobs self-heal out of a stuck-`RUN` state without manual intervention.
 
-**Impact**: Extends `rejoin_worker`/`apply_join_state` (`worker-join-refactor`, implemented) with a job-state reconciliation round; adds one new opcode pair to the wire protocol; adds a short-lived finished-job retention to the worker daemon (`xcecd`), which retains none today; adds two narrow new fact-recording functions to the controller's job table, reusing its existing idempotency guards for everything else; adds two new shared-config fields.
+**Impact**: Extends `rejoin_worker`/`apply_join_state` (`worker-join-refactor`, implemented) with a job-state reconciliation round; adds one new opcode pair to the wire protocol; adds a short-lived finished-job retention to the worker daemon (`xcecd`), which retains none today; adds two narrow new fact-recording functions to the controller's job table, reusing its existing idempotency guards for everything else; adds two new shared-config fields; adds one line to the scheduler's existing candidate-host filter; and, because Requirement 4.6 is the first path in this codebase that re-admits an already-dispatched job back to the scheduler's PEND pool, closes a previously-dormant gap (bug-1631) in `dispatch_a_job_`'s retryable-failure handling that this feature would otherwise make reachable.
 
 ### Goals
 - Trigger reconciliation on every worker join where the controller holds non-terminal jobs for that host, without needing to distinguish first-join from reconnect (Requirement 1).
@@ -31,6 +31,7 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
 - The wire protocol for the reconciliation exchange: two new opcodes, `OPCode_jobStateQuery` (controller → worker) and `OPCode_jobStateReport` (worker → controller), both paginated.
 - Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign); gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly *after* the worker-join RPC (`xce_join`/`xcecli_join`) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
 - Worker-side: a new short-lived retention of recently-finished job outcomes (jobid, exit status, real finish time), bounded by a configurable window; answering the controller's query from that retention plus the still-running job list; reporting extra jobs outside the query; a new local gate flag that delays the worker's own resumption of normal post-join operation until its report has finished sending (Requirement 3.8).
+- Scheduler-side: one narrow addition — the existing candidate-host filter (`scheduler.c:1211`, today `IS_WORKER_UNAVAIL`-only) also excludes a host with `XCE_WORKER_RECONCILING` set, so a RECONCILING host is never proposed as a dispatch target in the first place (Requirement 5.1). No other scheduler-side change; the worker-status mirror this reads from already exists (`core_events.c:527`/`:595`) and needs no new wire content for this bit.
 - Two new shared-config fields (the Requirement 3.7 reporting window; the Requirement 5.3 reconciliation timeout).
 
 ### Out of Boundary
@@ -39,7 +40,7 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
 - Delta/session-token rejoin optimization.
 - `commit-batch-window`'s commit-urgency tuning — no dependency, no reference.
 - Recovering `job->allocs` (the scheduler's resource allocation) for a job reconciled via Requirement 4.8 that the controller never held a dispatch record for — the data was never known to this controller and cannot be reconstructed from the worker's report. Converges via the existing reserved/actual accounting policy on the job's next rusage report (see Data Models), same as any legitimately-unreserved job today.
-- Fixing `WAL_JOB_DISPATCH`'s own pre-existing, separately-tracked `job->allocs` field-completeness gap (`persistence.md`'s `APPLY_DIVERGENT` section; roadmap.md's `raft-apply-decoupling` item #4) — this design does not make that gap worse, but does not fix it either.
+- A parallel WAL fact/decode path for the reconcile-assign case: bug-1678 (fixed 2026-08-25, predates this design) already gave `WAL_JOB_DISPATCH` a `persist_submit_job_dispatch_fact(persist_t*, jobid, state, ts, alloc)` primitive with both apply seams (`apply_committed_entry`, `apply_wal_entry`) decoding an *optional* trailing `JobAlloc` — a `NULL` alloc is explicitly the documented, tested backward-compatible case. `record_job_reconcile_assign_fact` calls this primitive directly with `alloc=NULL` (see Data Models) rather than inventing a second payload shape — out of boundary is building anything new here, not reusing what already exists.
 - Extending `xce_join()`/`xcecli_join()`'s own blocking wait (`cbuf_deadline_10s`, `xceapi.c`) to also cover config push or job-state reconciliation. That call is a shared, role-agnostic API (worker, apiserver, scheduler, and `xce_test`/`xceadmin` clients all use it) and already returns as soon as the join reply arrives — well before `on_worker_join` (config push) even runs, unchanged by this feature. Reconciliation has its own, independent, server-side timeout (`resync.c`'s `delaymgr` trigger against `workerResyncTimeoutMs`) and is not gated on, or gating of, the join RPC's own deadline in any way.
 
 ### Allowed Dependencies
@@ -51,7 +52,7 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
 - `config-design.md`'s shared-config classification rule, applied to the two new config fields.
 
 ### Revalidation Triggers
-- Any change to `worker_t.host.status` bitmask semantics or `dispatch_a_job_`'s connected-state check (job.c:1511-1512) — the dispatch-gating reuse (Requirement 5.1) depends on both staying exactly as observed.
+- Any change to `worker_t.host.status` bitmask semantics or `dispatch_a_job_`'s connected-state check (job.c:1511-1512) — the dispatch-gating reuse (Requirement 5.1) depends on both staying exactly as observed. **This trigger fired**: `scheduler-dispatch-commit` (implemented 2026-08-24/25, after this design's original research) added `xce_classify_dispatch_fail()`'s retry/terminal-fail machinery behind this exact check, and exposed bug-1631 (job.allocs never released on a retryable-failure path, deliberately left unfixed pending a re-admission path this design provides). Re-validated 2026-08-31/09-01 — see Architecture, Components, Error Handling for the resulting design changes (scheduler-side candidate exclusion, split disconnected-worker branch, explicit allocs release in `record_job_reconcile_pending_fact`).
 - Any change to `apply_job_update_transition` / `job_state_is_backward`'s RUN↔PEND handling.
 - Any change to the `XCE_CONTROLLER_HOOK_JOB` fan-out shape or the `NotifyData` capnp union.
 - Any change to `JoinData`'s capnp union arms (this design deliberately does not touch it, but a future change there should re-check this design's choice to use a separate opcode pair instead).
@@ -67,7 +68,9 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
 
 The job table's existing commit-then-apply split is the load-bearing idempotency mechanism this feature must reuse (per requirements.md's Boundary Context): `record_job_*_fact()` functions append a WAL fact (durable, no mutation), and `apply_job_*_transition()` functions are the guarded mutation core, invoked both as a peek at receipt time and later by `apply_committed_entry` once the fact commits. `job_state_is_terminal`/`job_state_is_backward` reject duplicate or backward (terminal→non-terminal) transitions — this is the exact "no duplicate facts, no clobbering newer state" guarantee Requirement 4 asks for, already built and tested (`test_guard_replay_finish_equals_single_application`).
 
-Dispatch gating (`dispatch_a_job_`, job.c:1278) already refuses to dispatch when `w->ep == NULL || w->host.status != 0` — any nonzero status bit blocks it. A new bit reuses this check with zero changes to `dispatch_a_job_` itself.
+Dispatch gating has two layers today, and both matter to this design. `dispatch_a_job_` (job.c:1278) refuses to dispatch when `w->ep == NULL || w->host.status != 0` (job.c:1511-1512) — but since `scheduler-dispatch-commit` (implemented 2026-08-24/25, after this design's original research), hitting that check no longer just returns an error: it routes through `xce_classify_dispatch_fail()` into a bounded-retry-then-`mark_job_fail()` outcome (job.c:1538-1569, site `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY`) for `w->ep == NULL` and any nonzero status bit alike, with no distinction between them. Separately, bug-1631 (confirmed live, deliberately left unfixed) documents that none of `dispatch_a_job_`'s retryable-failure branches release an already-committed `job->job.allocs` — scoped as unreachable *only* because nothing in the current system re-admits an already-`RUN` job back into the scheduler's PEND pool. Requirement 4.6 is exactly such a re-admission path, so this design cannot treat that gap as pre-existing/out-of-scope (see Components, Error Handling).
+
+Given both findings, `XCE_WORKER_RECONCILING` dispatch-gating is primarily enforced one layer up, at scheduler candidate selection (`scheduler.c:1211`'s existing `IS_WORKER_UNAVAIL`-style filter, extended — see Components), so the scheduler simply never proposes a RECONCILING host as a dispatch target. `dispatch_a_job_`'s status check remains an unchanged defensive backstop for the rare race where a decision predates the bit being observed, correctly falling into the same retry-then-terminal-fail path every other non-UNAVAIL status bit already takes today — unmodified. Requirement 5.4 (a truly disconnected worker, `w->ep == NULL`) needs a genuinely new branch in `dispatch_a_job_`, split out from `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY`.
 
 The controller→scheduler notification path is a hook→notify→wire→dequeue pipeline: job-state-change code fires `XCE_CONTROLLER_HOOK_JOB` (job.c), `sched.c`'s registered callback (`sched_job_hook_fun`) dispatches by `HOOK_DETAIL_*` to `sched_job_finish_fun`/`sched_job_fail_fun`/`sched_job_update_fun`, each of which calls the shared `notify()` (dataex.c:399) with a distinct `NotifyData_data_sched*` event over `c->sched.ep`. On the scheduler side, `sched_handle_notify()` (handler.c:525) is the **inbound** receipt switch that calls `sched_job_state_change()`; this is architecturally distinct from `scheduler.c`/`driver.c`'s own **outbound** decision-publish direction, which a standing test (`test_no_worker_resync_symbols_in_scheduler_publish_path`) mechanically guards against ever referencing "resync." This fixes the only permitted integration seam: `handler.c`'s inbound switch.
 
@@ -96,7 +99,8 @@ graph TB
     end
     subgraph Scheduler
         HANDLER[handler.c sched_handle_notify calls sched_job_state_change]
-        PUBLISH[scheduler.c / driver.c: outbound decision publish — NOT touched]
+        CANDFILT[scheduler.c:1211 candidate filter<br/>NEW: also skips IS_WORKER_RECONCILING<br/>same shape as existing IS_WORKER_UNAVAIL check]
+        PUBLISH[scheduler.c / driver.c: outbound decision publish — NOT touched otherwise]
     end
 
     JC --> RW --> RS
@@ -109,13 +113,16 @@ graph TB
     RQ --> JOBC
     JOBC --> COMMIT --> HOOK --> SCHEDC --> NOTIFY --> HANDLER
     HANDLER -.->|never referenced from| PUBLISH
+    RS -.->|existing worker-status mirror, core_events.c:527/595<br/>no new wire content| CANDFILT
+    CANDFILT -.->|ordinary status-bit read, not a resync concept —<br/>no resync symbol added to this file| PUBLISH
 ```
 
 **Architecture Integration**:
 - Selected pattern: additive extension of an existing daemon-to-daemon RPC + existing fact/apply pipeline — no new subsystem, no new process.
-- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (two new functions, rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is).
-- Existing patterns preserved: commit-then-apply idempotency, `{pos,count,n_total}` pagination, hook→notify scheduler path, per-worker write-lock discipline.
+- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (two new functions, rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is) / scheduler candidate-filter (one added disjunct to an existing check, see below — not zero, but not new machinery either).
+- Existing patterns preserved: commit-then-apply idempotency, `{pos,count,n_total}` pagination, hook→notify scheduler path, per-worker write-lock discipline, and — newly leaned on — the scheduler's existing worker-status-bit candidate filter (`IS_WORKER_UNAVAIL` at `scheduler.c:1211`).
 - New components rationale: reconciliation bookkeeping and worker retention have no existing analog to extend; everything else is reuse.
+- **Scheduler-side touch, and why it doesn't cross `test_no_worker_resync_symbols_in_scheduler_publish_path`'s fence**: that standing test guards `scheduler.c`/`driver.c`'s *outbound decision-publish* direction against ever learning reconciliation *protocol* concepts (round tokens, query/report messages, retention windows). The one line this design adds there — skip a host if `IS_WORKER_RECONCILING(w->status)` — is not that: it's the same *category* of check as the pre-existing `IS_WORKER_UNAVAIL(w->status)` line right next to it, reading an ordinary worker-status bit the scheduler already mirrors for an unrelated reason (`core_events.c:527`/`:595`). No opcode, no reconciliation-round concept, and no symbol named "resync" is introduced into either file. This is called out explicitly, not left for an implementer to discover the fence and guess whether it applies.
 - Steering compliance: async-I/O thread-role contract (bounded wait via admission reservation, not raw blocking); persistence per-field completeness review (see Data Models); config-design.md classification (see Data Models).
 
 ### Technology Stack
@@ -136,9 +143,10 @@ graph TB
 - `src/schema/controller.capnp` — add `OPCode_jobStateQuery`, `OPCode_jobStateReport` to the `OPCode` enum; add `JobStateQuery{jobids: List(UInt64), pos, count, ntotal}`, `JobStateReport{entries: List(JobStateEntry), pos, count, ntotal}`, `JobStateEntry{jobid, status: JobStateStatus, exitCode, finishTimeMs, queried: Bool}` (`queried=false` marks a Requirement 3.6 extra job); add both as `Request`/`Reply` union arms; add `workerResyncWindowMs`/`workerResyncTimeoutMs` fields to `ClusterConfig`.
 - `src/controller/worker.c` — `rejoin_worker`/`apply_join_state`: after clearing `XCE_WORKER_UNAVAIL`, check for non-terminal jobs assigned to this host (Requirement 1) and, if any, set `XCE_WORKER_RECONCILING` and initialize the per-worker reconciliation record before releasing the write lock. No change to admin-bit preservation or attribute/endpoint update behavior. `on_worker_join` (worker.c:623) — extended, not `handle_worker_join`: it already calls `worker_push_config` as its one existing step (the config the worker needs — hosttype/hostmodel/resource variables — is foundational baseline state, and its own stub comment, `/* consider to push jobs on hosts to */`, already anticipates this exact follow-on). The reconciliation query send is added right after it, in that fixed order (config push, then query) — both sent fire-and-forget, no new ack/reply invented for this. This ordering is safe, not just conventional: the worker's incoming-request dispatch (`myreq_handler`, `controller_connector.c:1563`) is a plain synchronous `switch` invoked one message at a time off its single long-lived client connection (`xcecli_ctx`) — confirmed by reading it, not assumed — so the two requests are not just sent in order, they are *handled* to completion in that same order on the worker side; no thread-pool fan-out is in the path (the worker's own `thread_pool_t` is for job-forking work, unrelated to request dispatch). The actual wait in this design belongs at the *end* of the sequence, not in the middle: the controller does not consider the worker's rejoin/reconciliation complete until the job-state *report* arrives (or the Requirement 5.3 timeout fires) — see Error Handling.
 - `src/controller/join.c` — **no change**. `handle_worker_join` itself is untouched; the reconciliation query is sent from the (extended) `on_worker_join`, which `handle_worker_join` already calls unconditionally after the join reply.
-- `src/controller/job.c` — two new functions: `record_job_reconcile_pending_fact()` (Requirement 4.6, routes through the existing `WAL_JOB_UPDATE`/`apply_job_update_transition` seam), `record_job_reconcile_assign_fact()` (Requirement 4.8, mirrors `dispatch_a_job_`'s assignment logic for host/state only — see Data Models for the allocation limitation). Existing `record_job_finish_fact`/`record_job_fail_fact` are called as-is for the finished-success/finished-failure cases.
+- `src/controller/job.c` — two new functions, both following `dispatch_a_job_`'s eager-mutation-plus-explicit-notify pattern (not `record_job_finish_fact`'s deferred one — see Components for why): `record_job_reconcile_pending_fact()` (Requirement 4.6, eagerly calls `apply_job_update_transition(job, JOB_STAT_PEND)` under the write lock, releases `job->job.allocs` — `free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;` — on an applied (non-no-op) transition, durably appends via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)`, then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking; this is bug-1631's own documented candidate fix, applied here because Requirement 4.6 is exactly the "RUN job re-admitted to the PEND pool" precondition bug-1631's fix was deferred pending), `record_job_reconcile_assign_fact()` (Requirement 4.8, eagerly sets `job->job.state`/host assignment under the write lock, calls the existing `persist_submit_job_dispatch_fact(c->persist, jobid, state, ts, alloc=NULL)` — reusing bug-1678's already-tested NULL-alloc decode path on both apply seams verbatim, not a new fact shape — then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking, same as above and for the same reason). Existing `record_job_finish_fact`/`record_job_fail_fact` are called as-is for the finished-success/finished-failure cases. Also: split `dispatch_a_job_`'s `w->ep == NULL || w->host.status != 0` check (job.c:1511-1512) into two branches, checked in order — `w->ep == NULL` is resolved directly via `record_job_reconcile_pending_fact()` with **no retry-counter increment and no `mark_job_fail()`**, *before* the code ever reaches `xce_classify_dispatch_fail()` (a deliberate bypass, not a third value on the shared `xce_dispatch_fail_class_t` enum — see Components); `w->host.status != 0` keeps its existing `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY` behavior unchanged (now rarely hit for `XCE_WORKER_RECONCILING` specifically, since the scheduler-side filter below keeps it from being selected in the first place).
 - `src/controller/cluster.c` — new `case OPCode_jobStateReport:` in `myreq_handler`'s dispatch switch (runs after `ha_dispatch_gate`); route to the new reconciliation-apply logic.
-- `src/include/xce/types.h` — add `XCE_WORKER_RECONCILING` to the worker status bitmask; add a `resync_state_t` (queried job-ID set, round token, deadline) referenced from `worker_t`.
+- `src/scheduler/src/scheduler.c` — extend the existing candidate-host filter (`scheduler.c:1211`, today `if (IS_WORKER_UNAVAIL(w->status)) { ... }`) to also skip a host with the new reconciling bit set — same shape, one added disjunct, no new call site.
+- `src/include/xce/types.h` — add `XCE_WORKER_RECONCILING` to the worker status bitmask plus an `IS_WORKER_RECONCILING(status)` macro (mirroring `IS_WORKER_UNAVAIL`'s existing shape) for the scheduler-side filter above; add a `resync_state_t` (queried job-ID set, round token, deadline) referenced from `worker_t`.
 - `src/lib/request.c` — `xce_job_state_query_req()`/`xce_job_state_report_req()` encode/decode helpers, following `xce_reply_get`/`xce_push_data_req`'s pagination loop shape.
 
 ### New Files
@@ -194,20 +202,20 @@ sequenceDiagram
         else still running
             JC->>R: clear this jobid from outstanding set (no-op fact)
         else unrecognized
-            JC->>JOBC: record_job_reconcile_pending_fact (NEW)
-            JOBC->>SCHED: notify scheduler of new PEND state (Req 7.2) via the same HOOK_DETAIL_JOB_UPDATE path
+            JC->>JOBC: record_job_reconcile_pending_fact (NEW) -- eager mutation + allocs release, durable append, then notify
+            JOBC->>SCHED: persist_notify_job_updated_already_persisted (Req 7.2), same HOOK_DETAIL_JOB_UPDATE path dispatch_a_job_ already uses
         else unqueried extra job
-            JC->>JOBC: record_job_reconcile_assign_fact (NEW)
+            JC->>JOBC: record_job_reconcile_assign_fact (NEW) -- eager mutation, durable append, then notify
+            JOBC->>SCHED: persist_notify_job_updated_already_persisted, same call dispatch_a_job_'s own success path already makes
         end
-        JOBC->>JOBC: commit-then-apply, existing guards, unchanged
-        JOBC->>SCHED: existing HOOK_DETAIL_JOB event notify path, unchanged
+        Note over JOBC: finished success/failure stay fully deferred (record_job_finish_fact/<br/>record_job_fail_fact, unchanged) -- mutation + notify happen later via<br/>apply_committed_entry once the fact commits, same as any ordinary finish/fail.<br/>The two NEW functions above are eager instead, mirroring dispatch_a_job_'s<br/>established pattern, not this deferred one -- see Components for why.
     end
 
     R->>WC: outstanding set empty? clear XCE_WORKER_RECONCILING
     Note over R: if deadline fires first (delaymgr trigger),<br/>resolve outstanding as answered/timed out (Req 5.3)<br/>rather than withholding dispatch indefinitely
 ```
 
-**Gating conditions**: `dispatch_a_job_` already refuses dispatch whenever `w->host.status != 0` (job.c:1511) — `XCE_WORKER_RECONCILING` reuses this unchanged. A dispatch attempt to a disconnected worker (`w->ep == NULL`) fails immediately and returns the job to pending (Requirement 5.4) via the same new reconcile-to-pending fact path used for an unrecognized job, rather than the existing retry-then-terminal-fail branch (whose retry semantics — retry the dispatch itself — don't fit "the worker isn't even connected").
+**Gating conditions**: Requirement 5.1's primary enforcement is one layer up from `dispatch_a_job_` — the scheduler's own candidate filter (`scheduler.c:1211`) excludes a `RECONCILING` host from consideration before a dispatch decision is ever made (see Components). `dispatch_a_job_`'s existing check (`w->host.status != 0`, job.c:1511) still refuses dispatch as a backstop for the rare decision-predates-the-bit race, and — unchanged — routes through the existing retry-then-terminal-fail classification the same as any other non-UNAVAIL status bit already does today. A dispatch attempt to a disconnected worker (`w->ep == NULL`) is handled separately and differently: it fails immediately and returns the job to pending (Requirement 5.4) via `record_job_reconcile_pending_fact`, checked and resolved *before* the status-bit check reaches `xce_classify_dispatch_fail()` at all — not the existing retry-then-terminal-fail branch (whose retry semantics — retry the dispatch itself — don't fit "the worker isn't even connected," and whose bounded-retry-then-`mark_job_fail()` outcome is the wrong terminal shape for a case Requirement 5.4 wants recoverable-via-reschedule instead). This split is a required code change to `dispatch_a_job_`, not a reuse — see Components.
 
 **Discard-on-disconnect** (Requirement 6): if the worker disconnects before its report is fully processed, `rejoin_worker`'s next invocation (on the next join) finds the prior round's token stale and starts a fresh one; any report that later arrives carrying the stale token is discarded (Requirement 6.3) rather than applied against the wrong round.
 
@@ -224,11 +232,11 @@ sequenceDiagram
 | 3.7 | Bounded, cluster-configurable reporting window | `ClusterConfig.workerResyncWindowMs`, worker retention | config field | Data Models |
 | 3.8 | Worker delays resuming normal post-join operation until its report has finished sending | worker `xcecd_resync.c`, new gate flag | resync-in-progress flag | Sequence: post-report note |
 | 4.1–4.5 | Apply via existing idempotent commit-then-apply; no-op duplicates; never clobber newer/terminal state | `job.c` (existing fns), `apply_job_*_transition` | existing | Sequence: apply per outcome |
-| 4.6 | Unrecognized → return to pending, eligible for scheduling | `record_job_reconcile_pending_fact` (NEW) | new fn | Sequence: unrecognized branch |
-| 4.7, 4.8 | Reconcile jobs outside the query; assign if no existing non-terminal record | `record_job_reconcile_assign_fact` (NEW) | new fn | Sequence: extras branch; Data Models limitation |
-| 5.1, 5.2 | Gate dispatch while reconciliation outstanding; restore eligibility on completion | `XCE_WORKER_RECONCILING` bit, `dispatch_a_job_` (reused) | status bitmask | Architecture: dispatch gating reuse |
+| 4.6 | Unrecognized → return to pending, eligible for scheduling, allocation released | `record_job_reconcile_pending_fact` (NEW, incl. `job->job.allocs` release — bug-1631) | new fn | Sequence: unrecognized branch |
+| 4.7, 4.8 | Reconcile jobs outside the query; assign if no existing non-terminal record | `record_job_reconcile_assign_fact` (NEW, calls `persist_submit_job_dispatch_fact(alloc=NULL)`) | new fn | Sequence: extras branch; Data Models limitation |
+| 5.1, 5.2 | Gate dispatch while reconciliation outstanding; restore eligibility on completion | `scheduler.c:1211` candidate filter (NEW disjunct, primary), `XCE_WORKER_RECONCILING` bit + `dispatch_a_job_` (backstop, reused) | status bitmask | Architecture: dispatch gating |
 | 5.3 | Bounded timeout resolves outstanding reconciliation | `resync.c` delaymgr trigger, `ClusterConfig.workerResyncTimeoutMs` | timeout | Sequence: deadline note |
-| 5.4 | Dispatch to disconnected worker fails fast, returns to pending | new dispatch-fail classification, `record_job_reconcile_pending_fact` (NEW, reused) | — | Error Handling |
+| 5.4 | Dispatch to disconnected worker fails fast, returns to pending | new pre-classifier `w->ep == NULL` branch in `dispatch_a_job_`, `record_job_reconcile_pending_fact` (NEW) | — | Error Handling |
 | 6.1–6.3 | Disconnect mid-round discards and re-triggers; idempotent re-application; stale-round reports discarded | `resync.c` round token | round token | Sequence: discard-on-disconnect note |
 | 7.1–7.3 | Reconciled changes reach the scheduler via the existing notify path | `XCE_CONTROLLER_HOOK_JOB`, `sched.c`, `handler.c` (all reused, unchanged) | `notify()` | Sequence: apply → notify |
 | 8.1–8.3 | Works for any job count; paginated multi-message exchange | `JobStateQuery`/`JobStateReport` pagination | `{pos,count,ntotal}` | Sequence: loop each query/report page |
@@ -240,10 +248,11 @@ sequenceDiagram
 | Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
 |---|---|---|---|---|---|
 | `resync.c` (new) | Controller | Per-worker reconciliation bookkeeping + timeout | 1, 5.3, 6 | `worker.c` (P0), `commit.c` admission pattern (P1) | State, Batch |
-| `job.c` additions | Controller | Two new idempotent fact-recording functions | 4.6, 4.8, 5.4 | existing commit-then-apply seam (P0) | Service |
+| `job.c` additions | Controller | Two new idempotent fact-recording functions + split disconnected-worker dispatch-fail site | 4.6, 4.8, 5.4 | existing commit-then-apply seam (P0), `xce_classify_dispatch_fail()` (P0) | Service |
 | `xcecd_resync.c` (new) | Worker | Answer job-state query from running list + retention | 3, 8 | `xcecd_worker_utils.c` (P0) | Batch, Event |
 | `xcecd_worker_utils.c` additions | Worker | Short-lived finished-job retention | 3.5, 3.7 | none new | State |
 | Wire opcode pair | Protocol | Paginated query/report exchange | 2, 3, 8 | `xce_reply_get`/`xce_push_data_req` pattern (P0) | API/Batch |
+| `scheduler.c` candidate-filter addition | Scheduler | Exclude a RECONCILING host from dispatch candidacy | 5.1, 5.2 | existing `IS_WORKER_UNAVAIL` filter (P0) | State |
 
 ### Controller: `resync.c`
 
@@ -279,13 +288,17 @@ sequenceDiagram
 
 | Field | Detail |
 |-------|--------|
-| Intent | The two reconciliation outcomes with no existing idempotent fact analog |
+| Intent | The two reconciliation outcomes with no existing idempotent fact analog, plus a required correction to a dormant retryable-dispatch-failure gap this feature reactivates |
 | Requirements | 4.6, 4.7, 4.8, 5.4 |
 
 **Responsibilities & Constraints**
-- `record_job_reconcile_pending_fact(cluster_t*, job_t*)` — submits a `WAL_JOB_UPDATE` fact targeting `JOB_STAT_PEND`, through the existing `apply_job_update_transition`/`job_state_is_backward` guard (RUN→PEND is non-terminal→non-terminal, already permitted); fires the existing `HOOK_DETAIL_JOB_UPDATE` path exactly as any other job-state-update fact would. Also used for Requirement 5.4 (dispatch-to-disconnected-worker fail-fast).
-- `record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*, xce_jobstate_report_t*)` — mirrors `dispatch_a_job_`'s own host/state assignment logic (job.c:1686-1734) for a job the controller has no existing non-terminal record of; does **not** attempt to populate `job->allocs` (see Data Models).
-- Neither function introduces a new guard — both submit through the existing `apply_job_*_transition` seam.
+
+Both new functions follow `dispatch_a_job_`'s own established pattern (job.c:1686-1748: eager, synchronous mutation under the write lock, immediately followed by a durable append, immediately followed by an explicit `persist_notify_job_updated_already_persisted(c, job)` call) — **not** `record_job_finish_fact`'s purely-deferred pattern (state mutation withheld entirely until `apply_committed_entry` applies the committed fact). This choice is deliberate, not copied without thought: `persist_notify_job_updated_already_persisted`'s own doc comment (`persist.h`) frames it generically ("e.g. `dispatch_a_job_()`'s own... call" — an example, not an exclusive binding) as the sanctioned way to notify for *any* caller whose "live, synchronous state... mutation" would otherwise leave `apply_committed_entry`'s later replay finding "state already matches" and silently skipping its own notify (bug-1679's exact failure mode). Both reconciliation outcomes share dispatch_a_job_'s justification for going eager rather than deferred: the fact being recorded is something the *worker* already confirmed really happened (or definitely didn't) — ground truth already changed outside this controller — not a same-node admin command whose commit could still be rolled back with nothing external yet depending on it (that second, riskier shape is `job_ctrl`'s `mark_job_update`, explicitly flagged in `roadmap.md`'s backlog as a legacy pattern to migrate *away* from, not a template to extend). Mixing an eager mutation for one field (e.g. `allocs`) with a deferred mutation for another (`state`) on the *same* transition was considered and rejected — it would create a real window where a job reads back as `RUN` with `allocs == NULL`, an inconsistent combination nothing else in this codebase produces; both fields move together, eagerly, under one lock hold.
+
+- `record_job_reconcile_pending_fact(cluster_t*, job_t*)`: under the job's write lock, calls `apply_job_update_transition(job, JOB_STAT_PEND)` directly (not as a "peek" — as the real, live mutation) — this reuses the existing `job_state_is_backward` guard unchanged, returning early as a no-op if it reports `JOB_TRANSITION_REJECTED_BACKWARD` or `JOB_TRANSITION_NOOP_DUPLICATE` (Requirement 4.3/4.4). Only on `JOB_TRANSITION_APPLIED`: releases `job->job.allocs` (`free_JobAlloc_ptr(&job->job.allocs); job->job.allocs = NULL;`) if non-NULL, still under the same lock hold — this is required, not optional: `apply_job_update_transition` only ever touches `job->job.state` (job.c:2808-2834, confirmed by reading it) and never allocs, and bug-1631 (confirmed live, deliberately left unfixed) documents exactly this gap on `dispatch_a_job_`'s own retryable-failure paths, scoped as unreachable "only because nothing in the current system re-admits an already-RUN job back into the scheduler's PEND pool." Requirement 4.6 is precisely that re-admission path. Then, still under the lock: durably appends the fact via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)` (the plain `{jobid, state}` shape is sufficient here — no extra fields, unlike DISPATCH/START/FINISH's richer siblings). After unlocking: calls `persist_notify_job_updated_already_persisted(c, job)`, reaching the scheduler through Requirement 7's existing notify path exactly as `dispatch_a_job_` already does for its own eager mutation. Also used for Requirement 5.4 (dispatch-to-disconnected-worker fail-fast) — see the `job.c` split below; the allocs release is a no-op there since a disconnected-worker dispatch attempt never committed one.
+- `record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*, xce_jobstate_report_t*)`: under the job's write lock, eagerly sets `job->job.state` and the host assignment to match the worker's report (mirroring `dispatch_a_job_`'s own inline mutation at job.c:1686-1717, minus the allocation fields — there are none to set); `job->job.allocs` stays `NULL` (see Data Models — there genuinely is none to write). Then, still under the lock: calls the existing `persist_submit_job_dispatch_fact(&c->persist, jobid, state, ts, alloc)` (job.c:1731-1734, the exact primitive `dispatch_a_job_`'s own success path already calls) directly, with `alloc = NULL` — bug-1678's fix (2026-08-25) already made this primitive's NULL-alloc case an explicit, tested, backward-compatible branch on both apply seams, so this is a thin caller into existing, proven code, not a new WAL fact shape. After unlocking: calls `persist_notify_job_updated_already_persisted(c, job)`, same as above and for the same reason — without it, `apply_committed_entry`'s later replay of this exact `WAL_JOB_DISPATCH` entry finds state already matching and silently skips its own notify, reproducing bug-1679 at this new call site if omitted.
+- Split in `dispatch_a_job_` (job.c:1511-1512): `w->ep == NULL` is checked and resolved **before** the existing `w->host.status != 0` check reaches `xce_classify_dispatch_fail()` at all — calling `record_job_reconcile_pending_fact()` directly, with the job's `dispatch_fail_count` left untouched and `mark_job_fail()` never called, then returning. This is deliberately a bypass, not a third value added to `xce_dispatch_fail_class_t`: that enum's whole contract (per its own precedent, `commit-batch-window`'s requirements.md: "an exhaustive site-identity enum maps to a decision — TERMINAL vs RETRYABLE") is a two-way choice every other call site's `switch` already exhaustively handles; adding a third value would force each of those ~5 unrelated switches to grow a case they can never hit. `w->host.status != 0` (any bit, `XCE_WORKER_RECONCILING` included) is otherwise unchanged — still reaches `xce_classify_dispatch_fail(XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY)` and its existing retryable/terminal path, same as today, on the (now rare, since the scheduler-side filter keeps it from being the common case) chance a dispatch decision predates the bit being observed.
+- No function introduces a new *commit/idempotency* guard — all three reuse existing guarded seams (`apply_job_update_transition`'s `job_state_is_backward`, `persist_submit_job_fact`/`persist_submit_job_dispatch_fact`'s existing WAL append contracts) and the existing, already-proven-safe `persist_notify_job_updated_already_persisted` notify primitive. Nothing here is a new persistence *mechanism* — it is the same eager-mutation-plus-explicit-notify shape `dispatch_a_job_` already established, applied to two new call sites.
 
 **Contracts**: Service [x]
 
@@ -295,9 +308,9 @@ int record_job_reconcile_pending_fact(cluster_t* c, job_t* job);
 int record_job_reconcile_assign_fact(cluster_t* c, job_t* job,
                                      worker_t* w, xce_jobstate_report_t* report);
 ```
-- Preconditions: caller holds `job`'s write lock (matching every other `record_job_*_fact` sibling); `job` is non-terminal (pending case) or has no existing non-terminal assignment (assign case)
-- Postconditions: a WAL fact is durably appended; mutation happens later via `apply_committed_entry`, identical to every other fact type
-- Invariants: never called for a job the controller already knows is terminal or more-final (Requirement 4.4/4.5 — enforced by the existing guard, not by new logic here)
+- Preconditions: caller holds `job`'s write lock on entry (matching every other `record_job_*_fact` sibling); `job` is non-terminal (pending case) or has no existing non-terminal assignment (assign case)
+- Postconditions: unlike `record_job_finish_fact`/`record_job_fail_fact` (deferred — mutation happens later via `apply_committed_entry`), both of these functions mutate `job`'s live in-memory state **eagerly, before returning** — matching `dispatch_a_job_`'s established pattern, not the deferred one — then durably append the fact, then (after the caller releases the lock) notify via `persist_notify_job_updated_already_persisted`. `record_job_reconcile_pending_fact` additionally guarantees `job->job.allocs == NULL` on return whenever it was non-NULL on entry, mutated atomically alongside the state change under the same lock hold (in-memory only, not itself a WAL field — see Data Models).
+- Invariants: never called for a job the controller already knows is terminal or more-final (Requirement 4.4/4.5 — enforced by the existing `job_state_is_backward` guard, not by new logic here); the caller must call `persist_notify_job_updated_already_persisted(c, job)` after unlocking, exactly once, for every actual (non-no-op) mutation — omitting it reproduces bug-1679 at this call site.
 
 ### Worker: `xcecd_resync.c` (new)
 
@@ -344,6 +357,29 @@ int record_job_reconcile_assign_fact(cluster_t* c, job_t* job,
 - Persistence & consistency: **not persisted** — a worker restart loses this retention, which is acceptable: a restarted worker has no running jobs of its own to report on rejoin either way (its child processes died with it), so the retention's only job is to survive a *controller-side* disconnect/reconnect window, not a worker restart
 - Concurrency strategy: same locking discipline already used for `xcecd_worker_t`'s list
 
+### Scheduler: `scheduler.c` candidate-filter addition
+
+| Field | Detail |
+|-------|--------|
+| Intent | Keep a `RECONCILING` host out of dispatch candidacy at the source, so Requirement 5.1 is enforced without depending on the controller-side retry/terminal-fail machinery for the common case |
+| Requirements | 5.1, 5.2 |
+
+**Responsibilities & Constraints**
+- One added disjunct to the existing check at `scheduler.c:1211` — today `if (IS_WORKER_UNAVAIL(w->status)) { ... }`, becomes `if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { ... }` (new macro, `types.h`, same shape as the existing `IS_WORKER_UNAVAIL`/`IS_WORKER_LOCKEDU`/etc. family).
+- Reads `w->status`, a field the scheduler already receives via the existing worker-status mirror (`core_events.c:527`/`:595`) — no new wire content, no new push.
+- Does **not** touch `scheduler.c`/`driver.c`'s outbound decision-*publish* logic or introduce any reconciliation-protocol concept into either file — see Architecture Integration for why this stays clear of `test_no_worker_resync_symbols_in_scheduler_publish_path`'s fence.
+- Requirement 5.2 ("eligible for new job dispatch again" once reconciliation completes) needs no separate scheduler-side action: once the controller clears `XCE_WORKER_RECONCILING` and that status change is next mirrored to the scheduler, this same filter simply stops excluding the host.
+
+**Dependencies**
+- Inbound: the existing worker-status mirror (P0) — no new dependency introduced
+
+**Contracts**: State [x]
+
+##### State Management
+- State model: none new — reads the existing `xce_sched_worker_t.status` mirror field
+- Persistence & consistency: not applicable — a read of already-existing, already-synced state
+- Concurrency strategy: unchanged — same read path as the existing `IS_WORKER_UNAVAIL` check it sits beside
+
 ---
 
 ## Data Models
@@ -382,10 +418,10 @@ Per `design-principles.md`'s Persistence Review requirement and `persistence.md`
 | Finished success | `record_job_finish_fact` (existing, reused as-is) | None — existing complete fact shape (already carries `finish_time`, exit info); the reconciliation caller passes the worker-reported `finishTimeMs` (Requirement 3.5) as that fact's finish time, never the time the controller processed the report (Requirement 4.2) | Already covered by existing `apply_committed_entry`/standby-catch-up handling — unchanged |
 | Finished failure | `record_job_fail_fact` (existing, reused as-is) | None — same Requirement 4.2 worker-reported-time rule applies | Unchanged, existing |
 | Still running | No fact — pure no-op against the outstanding set | N/A | N/A |
-| Unrecognized → pending | `record_job_reconcile_pending_fact` (NEW) | None beyond the existing `WAL_JOB_UPDATE` state field | **Must confirm at implementation time** that both `apply_committed_entry` (primary boot replay) and the standby catch-up apply path decode `WAL_JOB_UPDATE` identically for an arbitrary target state — expected yes, since this is the same fact *type* as any other state update, just a new call site, but this is a mechanical check to perform, not assume |
-| Unqueried job → assign | `record_job_reconcile_assign_fact` (NEW) | jobid, host assignment, state — **explicitly does NOT include `job->allocs`** (see below) | Both seams must decode/adopt jobid+host+state identically — new call site into what is otherwise the same assignment shape `dispatch_a_job_` already produces |
+| Unrecognized → pending | `persist_submit_job_fact(..., WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)` (existing generic primitive, new call site) | None — the plain `{jobid, state}` shape is sufficient, same as every other `WAL_JOB_UPDATE` caller. **`job->job.allocs` release is in-memory only, eager, not itself a persisted field** — mutated live under the same lock as the state change (mirroring `dispatch_a_job_`'s own success path in reverse; see Components), then notified via `persist_notify_job_updated_already_persisted`, not left for `apply_committed_entry` to discover | Both seams already decode plain `WAL_JOB_UPDATE` identically today (it is not a new payload shape) — a replay of this fact re-derives `job->job.state = JOB_STAT_PEND` correctly; it does **not** re-derive the allocs release (that mutation is not wire content), but since `dispatch_a_job_`'s eager pattern already established this exact asymmetry (live mutation + explicit notify now, replay-idempotent state field only, no replay-time side effect needed for a purely in-memory cleanup), this is consistent with existing precedent, not a new gap |
+| Unqueried job → assign | `persist_submit_job_dispatch_fact(..., alloc=NULL)` (existing primitive, new call site, `record_job_reconcile_assign_fact`) | None — reuses `persist_submit_job_dispatch_fact`'s existing `WAL_JOB_DISPATCH` payload shape verbatim with `alloc=NULL`, exercising the same NULL-alloc branch bug-1678 already added and tested. Live mutation is eager (see Components), matching `dispatch_a_job_`'s own pattern for the same fact type | Already covered — both seams (`apply_committed_entry`, `apply_wal_entry`) were updated for this exact NULL-alloc case by bug-1678's fix; no new decode path to verify |
 
-**Documented scope decision — `job->allocs` on a reconciliation-assigned job (Requirement 4.8):** when the controller reconciles a job it never held any prior record of (e.g. it was dispatched by a controller instance before a failover whose dispatch fact was itself lost), it has no way to learn what resource allocation the scheduler originally computed for that job — the worker's report carries the job's execution outcome, not its original resource request. This design deliberately does not attempt to reconstruct `job->allocs` for this case. This is not a new problem: `persistence.md`'s `APPLY_DIVERGENT` section already documents `WAL_JOB_DISPATCH` itself not carrying `job->allocs` through replay as a tracked, separate gap (roadmap.md's `raft-apply-decoupling` item #4); this design does not worsen it. The practical consequence is bounded by `resource-accounting.md`'s existing policy: a job with an unknown/zero reservation is charged `max(reserved=0, actual)` — i.e., its real usage — the moment its next rusage tick arrives, identical to how a legitimately-unreserved job is already handled today. The job's *state* is correct immediately; its resource *accounting* converges within one rusage cycle, not instantly. No fabricated allocation value is ever written.
+**Documented scope decision — `job->allocs` on a reconciliation-assigned job (Requirement 4.8):** when the controller reconciles a job it never held any prior record of (e.g. it was dispatched by a controller instance before a failover whose dispatch fact was itself lost), it has no way to learn what resource allocation the scheduler originally computed for that job — the worker's report carries the job's execution outcome, not its original resource request. This design deliberately does not attempt to reconstruct `job->allocs` for this case. **Correction (2026-09-01, re-validated against current code):** an earlier draft of this section cited `persistence.md`'s `APPLY_DIVERGENT` section / bug-1678 as evidence this is "a tracked, separate gap" `WAL_JOB_DISPATCH` still has — that citation was stale. Bug-1678 was fixed 2026-08-25 (before this design was finalized): `WAL_JOB_DISPATCH` now durably carries `job->allocs` when one exists, on both apply seams, with an explicit, tested `NULL`-alloc backward-compatible case. The real reason this design still writes no allocation for Requirement 4.8 is simpler and unrelated to that now-closed gap: there just isn't one to write — the scheduler, not this controller, is the only party that ever computes a job's allocation, and for a job this controller never had any record of, no dispatch (successful or otherwise) ever ran through this controller to produce one. `record_job_reconcile_assign_fact` reuses the *fixed* `persist_submit_job_dispatch_fact` primitive precisely because its NULL-alloc case is already the correct, tested representation of "state and host known, allocation not." The practical consequence is bounded by `resource-accounting.md`'s existing policy: a job with an unknown/zero reservation is charged `max(reserved=0, actual)` — i.e., its real usage — the moment its next rusage tick arrives, identical to how a legitimately-unreserved job is already handled today. The job's *state* is correct immediately; its resource *accounting* converges within one rusage cycle, not instantly. No fabricated allocation value is ever written.
 
 ### Configuration Model
 
@@ -409,7 +445,9 @@ Every reconciliation outcome resolves definitively — there is no "leave it amb
 - **Worker disconnects mid-round** (Requirement 6.1): the outstanding round is discarded (not applied partially); the next join re-triggers a fresh round via the same Requirement 1 check.
 - **Stale/superseded report arrives** (Requirement 6.3): discarded by round-token mismatch in `resync.c` — never applied against a round that's already been superseded.
 - **Reconciliation report never arrives / no reply within the bound** (Requirement 5.3): a delaymgr-scheduled trigger (non-blocking fire, per the async-I/O thread-role contract) resolves the outstanding round — accepted-as-answered, timed out, or left for the next rejoin to retry — rather than withholding that worker from dispatch indefinitely. This must not be a blocking wait anywhere in the iomgr/delaymgr call chain; if a synchronous wait style is preferred at implementation time instead of a pure trigger-and-resolve design, it must follow `commit.c`'s `COMMIT_WAIT_RESERVED` admission-reservation pattern (bounded blocking under an explicit cap on the vtask pool), never a raw wait.
-- **Dispatch attempted to a disconnected worker** (Requirement 5.4): fails immediately (not queued for later delivery) and returns the job to pending via `record_job_reconcile_pending_fact` — a distinct classification from the existing retry-then-terminal-fail branch in `dispatch_a_job_`, whose retry semantics (retry the dispatch attempt itself against the same worker) don't fit "the worker isn't even connected."
+- **Dispatch attempted to a disconnected worker** (Requirement 5.4): fails immediately (not queued for later delivery) and returns the job to pending via `record_job_reconcile_pending_fact` — checked and resolved directly in `dispatch_a_job_`, ahead of and separate from the `w->host.status != 0`/`xce_classify_dispatch_fail()` path, not a new value bolted onto that shared TERMINAL/RETRYABLE enum: retrying the same dispatch attempt against a worker that isn't even connected doesn't fit either existing outcome, and unlike a TERMINAL resolution this one must NOT call `mark_job_fail()` — it must leave the job schedulable again, not fail it.
+- **Dispatch attempted to a `RECONCILING` worker** (Requirement 5.1): primarily prevented at the source — `scheduler.c`'s candidate filter (extended, see Components) excludes a `RECONCILING` host, so the scheduler does not produce a dispatch decision for it in the first place. If one is produced anyway (a decision-predates-the-bit race), `dispatch_a_job_`'s unchanged `w->host.status != 0` check still refuses it, falling into the existing `WORKER_NOT_READY` retry-then-terminal-fail path — the same, unmodified behavior every other non-UNAVAIL status bit (`LOCKEDU`/`LOCKEDW`/`BUSY`/`CLOSED`) already produces today. This design does not change that behavior for the race case; it only makes the race rare by filtering earlier.
+- **A job reconciled from RUN back to PEND is later re-dispatched and fails retryably** (Requirement 4.6, bug-1631): confirmed as bug-1631's own deliberately-deferred trigger condition — a RUN job with a real `job->job.allocs` returned to PEND and re-admitted to the scheduler's dispatch pool, something nothing in the system did before this feature. `record_job_reconcile_pending_fact` closes this by releasing `job->job.allocs` at the same point it sets the job PEND (see Components), so a subsequent redispatch attempt — successful or retryably-failing — never observes a stale allocation left over from the job's prior dispatch. This must ship together with Requirement 4.6, not be left as a follow-on: shipping 4.6 without it would take bug-1631 from "deliberately deferred, unreachable" to "live, reachable, and unfixed."
 - **New opcode reaches a non-primary controller**: per `api-standards.md`'s "Adding a New Opcode" checklist (bug-1700 precedent) — verify at implementation time whether `OPCode_jobStateQuery`/`OPCode_jobStateReport` need an explicit redirect-reply path. Expected answer: no, since a worker only ever holds a connection to the current primary (connection-lifecycle-tied-to-primary-status, per the HA connection-ownership model), so this opcode pair should never reach a standby in practice — but this must be confirmed, not assumed, given bug-1700's documented history of this exact class of gap recurring across two other opcodes.
 
 ### Monitoring
@@ -423,11 +461,19 @@ Reuses existing job-state-change logging (job.c's existing log call sites for fi
 - Reconciliation triggers iff the host has ≥1 non-terminal job at rejoin (Requirement 1.1/1.2), for all three join circumstances (first join, network reconnect, failover reconnect) without the code needing to distinguish them (Requirement 1.3).
 - Query construction lists exactly the controller's non-terminal jobs for that host by ID (Requirement 2).
 - Each of the five reported outcomes (running, finished-success, finished-failure, unrecognized, unqueried-extra) applies through the correct existing/new fact function exactly once, and a repeated report for the same round is a no-op (Requirement 4, mirroring `test_guard_replay_finish_equals_single_application`'s existing style).
-- `XCE_WORKER_RECONCILING` blocks `dispatch_a_job_` and clears once the round resolves (Requirement 5.1/5.2).
+- `XCE_WORKER_RECONCILING` still blocks `dispatch_a_job_` as a backstop (unchanged existing behavior) and clears once the round resolves (Requirement 5.1/5.2).
+- `record_job_reconcile_pending_fact` releases a non-NULL `job->job.allocs` when reconciling a RUN job back to PEND, and leaves it untouched (already NULL) for a job that never held one — regression coverage for bug-1631's exact deferred trigger condition: dispatch (real allocs committed) → reconcile-to-pending via an `unrecognized` report → assert `job->job.allocs == NULL` and `job->job.state == JOB_STAT_PEND` **immediately after the call returns, before any commit round-trip** (the mutation is eager, not deferred — see Components) → redispatch → a retryable dispatch failure on that redispatch leaves no stale allocation and no incorrect state (Requirement 4.6).
+- Both `record_job_reconcile_pending_fact` and `record_job_reconcile_assign_fact` call `persist_notify_job_updated_already_persisted` exactly once per actual (non-no-op) mutation, reaching the scheduler's/apiserver's notify path immediately — not only after a later `apply_committed_entry` replay finds a no-op and silently skips its own notify (bug-1679's exact failure mode, reactivated at these two new call sites if this assertion is missing). Assert the hook fires synchronously within the call, not merely "eventually."
+- `record_job_reconcile_assign_fact` round-trips through both apply seams (`apply_committed_entry`, `apply_wal_entry`) via `persist_submit_job_dispatch_fact`'s existing NULL-alloc path — assert a real WAL entry is written and a **restart replay** (a separate scenario from the live eager-mutation path above) reconstructs jobid/host/state with `allocs == NULL` and correctly no-ops its own hook fire (state already matches from the eager mutation, or from the fact's own replay if this happens after a crash before the eager mutation's process ever ran again) — not a new decode path (Requirement 4.8).
+- `dispatch_a_job_`'s new pre-classifier `w->ep == NULL` branch: dispatch to a disconnected worker fails immediately, does not increment `dispatch_fail_count`, does not call `mark_job_fail`, never reaches `xce_classify_dispatch_fail()`, and returns the job to PEND via `record_job_reconcile_pending_fact` (Requirement 5.4) — asserted separately from the unchanged `w->host.status != 0`/`WORKER_NOT_READY` case, which keeps its existing retry-then-terminal-fail behavior verbatim.
 - Timeout resolves an outstanding round without a blocking call (Requirement 5.3).
 - Disconnect mid-round discards and re-triggers; a stale-round report is discarded (Requirement 6).
 - A reconciled state change reaches the scheduler via the existing notify path, asserted the same way existing finish/fail notify tests assert it (Requirement 7).
 
+### Unit Tests (Scheduler, existing `scheduler.c` candidate-selection test coverage)
+- A host with `XCE_WORKER_RECONCILING` set is excluded from dispatch candidacy the same way an `IS_WORKER_UNAVAIL` host already is (Requirement 5.1); a host with the bit cleared is eligible again (Requirement 5.2). Added alongside the existing `IS_WORKER_UNAVAIL` candidate-filter test, not a new test file.
+- Confirm the new check introduces no string/symbol named `resync` into `scheduler.c`/`driver.c` — `test_no_worker_resync_symbols_in_scheduler_publish_path` (`test_scheduler_decision_redelivery.c:432`) must stay green unmodified.
+
 ### Unit Tests (Worker, new `src/worker/testsuite/test_xcecd_resync.c`)
 - Query answered correctly from the running list, from retention, and as `UNRECOGNIZED` when in neither (Requirement 3.1–3.3).
 - Real finish time reported, not send time (Requirement 3.5).

.kiro/specs/worker-rejoin-resync/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "worker-rejoin-resync",
   "created_at": "2026-08-10T00:00:00Z",
-  "updated_at": "2026-08-31T23:59:00Z",
+  "updated_at": "2026-09-01T01:05:00Z",
   "language": "en",
-  "phase": "design-generated",
+  "phase": "tasks-generated",
   "description": "Controller-initiated job-state reconciliation on worker rejoin: on reconnect, the controller asks the worker what really happened to jobs it believes are running there, and reconciles the answer idempotently, so facts lost to fire-and-forget notify drop (bug-1741) self-heal instead of leaving jobs stuck in RUN forever.",
   "approvals": {
     "requirements": {
@@ -12,13 +12,15 @@
     },
     "design": {
       "generated": true,
-      "approved": false
+      "approved": true,
+      "validation_note": "kiro-validate-design 2026-09-01: initial pass found 2 critical issues (stale bug-1678 citation blocking reuse of persist_submit_job_dispatch_fact; XCE_WORKER_RECONCILING vs. scheduler-dispatch-commit's dispatch-fail retry/terminal-fail classification). Fixed in design.md, then re-validated during the same pass and found 2 more (bug-1631 reactivation -- job.allocs never released when a RUN job reconciles back to PEND; eager-vs-deferred mutation-pattern inconsistency between the new job.c functions and the bug-1679 explicit-notify precedent). All 4 fixed directly in design.md; a further self-check found no additional issues. Self-approved per explicit user instruction to iterate to GO -- a human skim before /kiro-spec-tasks is still advisable given the scope of the changes."
     },
     "tasks": {
-      "generated": false,
-      "approved": false
+      "generated": true,
+      "approved": true,
+      "validation_note": "kiro-spec-tasks 2026-09-01: 7 major tasks / 19 sub-tasks drafted covering all 33 requirement IDs (1.1-8.3) and every design.md component. Task-plan review gate passed after tightening 4 over/under-annotated _Depends:_ tags. Independent task-graph sanity review (fresh subagent, pass 1): NEEDS_FIXES -- Task 5 was a single-sub-task container (violates 2-level collapse rule), missing the design's scheduler-symbol-fence constraint, and no task verified the HA-redirect requirement for the 2 new opcodes (bug-1700 class). All 3 fixed. Sanity review pass 2 (fresh subagent): NEEDS_FIXES -- one dangling _Depends: 5.1_ left over from flattening task 5, fixed to _Depends: 5_. Auto-approved per -y flag."
     }
   },
-  "ready_for_implementation": false,
+  "ready_for_implementation": true,
   "blocked_on": []
 }

.kiro/specs/worker-rejoin-resync/tasks.md

@@ -0,0 +1,142 @@
+# Implementation Plan
+
+- [ ] 1. Foundation: wire protocol schema and shared reconciliation types
+- [ ] 1.1 Define the wire protocol for the reconciliation exchange
+  - Add the controller→worker job-state query and worker→controller job-state report as new opcodes, each carrying a paginated list of per-job entries (job ID, status, exit code, worker-reported finish time, and whether the job was part of the original query)
+  - Add the two new cluster-wide configuration fields: the recently-finished reporting window and the reconciliation timeout
+  - Observable completion: the schema builds cleanly and the new message/opcode/config-field definitions are visible in the generated bindings
+  - _Requirements: 2.1, 2.2, 3.5, 3.7, 5.3, 8.1, 8.2, 8.3_
+
+- [ ] 1.2 (P) Add a worker reconciling status and round-tracking state
+  - Add a new worker status value meaning "job-state reconciliation outstanding for this host"
+  - Add a per-worker reconciliation-round structure capable of holding a round identifier, the set of queried job IDs, and a deadline
+  - Observable completion: the new status value and round structure compile and are reachable from the worker record, unused by any caller yet
+  - _Requirements: 5.1, 6.3_
+  - _Boundary: shared types_
+
+- [ ] 1.3 (P) Establish worker-side C test infrastructure
+  - Add the first test-runner harness for the worker daemon, mirroring the controller's existing runner conventions
+  - Observable completion: an empty/placeholder worker-side test program builds and runs successfully via the existing test build target
+  - _Requirements: 3.1, 3.2, 3.3_
+  - _Boundary: worker test infrastructure_
+
+- [ ] 1.4 (P) Add wire encode/decode helpers for the reconciliation exchange
+  - Add request/reply encode and decode helpers for the new query and report messages, following the existing paginated-list pattern already used for job/worker/resource listings
+  - Observable completion: a round-trip encode-then-decode of a query and of a report (including an empty page and a multi-page case) reproduces the original data exactly
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Boundary: wire codec_
+  - _Depends: 1.1_
+
+- [ ] 2. Core: controller job-table reconciliation outcomes
+- [ ] 2.1 Return an unrecognized job to pending state, releasing its resource allocation
+  - When a worker reports a queried job as unrecognized, transition that job back to pending using the existing state-transition guard so a terminal or already-more-final job is never overwritten
+  - Release any resource allocation the job was carrying, so a job returned to pending never carries a stale allocation from a prior dispatch into its next dispatch attempt
+  - Durably record the transition and notify the scheduler of the new pending state through the existing notification path, without waiting for a later replay to discover the change
+  - Observable completion: a job previously dispatched (real allocation committed) that reconciles to unrecognized ends the call with pending state, no resource allocation, and the scheduler notified — verified before any commit round-trip completes
+  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 7.1, 7.2, 7.3_
+  - _Boundary: job.c additions_
+
+- [ ] 2.2 Assign an unqueried job to the worker that reported it
+  - When a worker reports a job the controller has no existing non-terminal record of, record that job as assigned to the reporting worker and host, matching the worker's reported status, through the same idempotent path used for a normal dispatch
+  - Do not fabricate a resource allocation for this job — leave the allocation unset, consistent with the controller genuinely not knowing what the scheduler originally computed
+  - Notify the scheduler of the new assignment through the existing notification path
+  - Observable completion: a job the controller never previously tracked, reported running by a worker, ends the call recorded as running on that host with the scheduler notified, and durably survives a restart replay with the same result
+  - _Requirements: 4.1, 4.3, 4.4, 4.7, 4.8, 7.1, 7.2, 7.3_
+  - _Boundary: job.c additions_
+
+- [ ] 2.3 Fail a dispatch to a disconnected worker immediately, without retry or job failure
+  - When the controller attempts to dispatch a job to a worker it is not currently connected to, fail that specific dispatch attempt right away and return the job to pending, reusing the pending-and-release behavior from task 2.1, rather than retrying the attempt or exhausting a retry count toward permanent failure
+  - Leave the existing behavior for every other reason a worker might not be ready (locked, busy, closed, or mid-reconciliation) completely unchanged — confirm those cases still retry and eventually fail exactly as before
+  - Observable completion: dispatching to a disconnected worker returns the job to pending on the first attempt with the job's failure-retry count untouched and no permanent failure recorded, while dispatching to a worker that is connected but otherwise not ready still follows the pre-existing retry-then-fail behavior unchanged
+  - _Requirements: 5.1, 5.4_
+  - _Boundary: job.c additions_
+  - _Depends: 2.1_
+
+- [ ] 3. Core: controller reconciliation-round bookkeeping
+- [ ] 3.1 (P) Track one reconciliation round per worker with a bounded timeout
+  - Given a worker host with one or more non-terminal jobs currently assigned to it, construct the exact list of those jobs' unique IDs as the round's query set
+  - Track a round identifier that distinguishes the current reconciliation attempt from any prior one for the same worker
+  - Schedule a bounded timeout for the round using a non-blocking trigger; when it fires before the round otherwise resolves, resolve the round rather than leaving the worker withheld from dispatch indefinitely
+  - Observable completion: arming a round for a worker with known non-terminal jobs produces a query set matching exactly those job IDs, a fresh round identifier, and an active timeout; letting the timeout elapse resolves the round without any blocking call
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 5.3, 6.1_
+  - _Boundary: resync.c_
+
+- [ ] 3.2 Apply each reported job outcome through the correct idempotent path
+  - For each entry in a worker's job-state report: a still-running job is cleared from the round's outstanding set with no fact recorded; a finished-success or finished-failure job is applied through the existing finish/fail recording used for an ordinary report; an unrecognized queried job is applied through task 2.1's pending-and-release path; a reported job outside the original query is applied through task 2.2's assignment path
+  - Discard a report that arrives for a round already superseded by a newer one for the same worker, rather than applying it
+  - When the round's outstanding set becomes empty, resolve the round so the worker becomes eligible for dispatch again
+  - Applying the same report twice for the same round must produce the same result as applying it once
+  - Observable completion: a full report covering all five outcome kinds leaves the job table in the correct state for each, a duplicate re-application of the same report changes nothing further, and a report carrying a stale round identifier is dropped without touching the job table
+  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 6.1, 6.2, 6.3_
+  - _Boundary: resync.c_
+  - _Depends: 2.1, 2.2, 3.1_
+
+- [ ] 4. Core: worker-side job-state reporting
+- [ ] 4.1 (P) Retain a job's true outcome for a bounded window after it exits
+  - At the moment a job's exit is detected, before its record would otherwise be discarded, retain its exit status, exit code, and true finish time for a configurable bounded window
+  - Observable completion: a job's outcome remains queryable for the configured window after it exits and is no longer available once the window elapses
+  - _Requirements: 3.4, 3.5, 3.7_
+  - _Boundary: worker retention_
+
+- [ ] 4.2 Answer the controller's job-state query, including extra unqueried jobs
+  - For every job ID in the controller's query, report exactly one outcome: still running (from the live running list), finished successfully or with failure and its true finish time (from the new retention), or unrecognized if found in neither — every queried ID gets an answer, none are silently skipped, and an indeterminate case is reported as unrecognized rather than guessed
+  - After answering the query, also report any job the worker is currently running or has recently finished that was not part of the query
+  - Send both the query answers and the extra jobs in paginated reports, correct regardless of how many jobs are involved
+  - Observable completion: a query covering a mix of running, finished, and unknown job IDs produces one correctly-classified entry per ID plus a separate page of extras for jobs outside the query, with no job silently dropped even when the total exceeds one page
+  - _Requirements: 3.1, 3.2, 3.3, 3.6, 8.2, 8.3_
+  - _Boundary: xcecd_resync.c_
+
+- [ ] 4.3 Delay resuming normal post-join operation until the report has fully sent
+  - Set a local gate the moment a job-state query is received; keep it set across every page of the report; clear it only once the full report, including any extra-job pages, has finished sending
+  - Make whatever the worker does next after rejoining (resuming its own periodic background reporting) check this gate first, so that activity never begins or interleaves while the report is still being sent
+  - Observable completion: the worker's post-join activity is observed to start only after the last report page has been sent, never before, across a report spanning multiple pages
+  - _Requirements: 3.8_
+  - _Boundary: xcecd_resync.c_
+
+- [ ] 5. (P) Core: exclude a reconciling host from scheduler dispatch candidacy
+  - Extend the scheduler's existing candidate-host filter so a host currently undergoing job-state reconciliation is excluded from dispatch candidacy the same way an already-unavailable host is
+  - When the reconciling status clears, the same filter must stop excluding that host without any separate action
+  - Introduces no new reconciliation-protocol name or concept into the scheduler's outbound decision-publish code — this is an ordinary status-bit read alongside the existing unavailable-host check, not a new symbol; the standing regression test guarding that publish path against ever referencing reconciliation concepts must stay green, unmodified
+  - Observable completion: a host flagged as reconciling is never selected as a dispatch candidate while the flag is set, and becomes selectable again as soon as it clears, with no change to how any other exclusion reason behaves, and the publish-path symbol-guard test still passes unmodified
+  - _Requirements: 5.1, 5.2_
+  - _Boundary: scheduler candidate filter_
+
+- [ ] 6. Integration: cross-daemon wiring
+- [ ] 6.1 Trigger reconciliation on worker rejoin and send the query
+  - At the moment a worker host becomes connected again (first join, network reconnect, or reconnect after controller failover, treated identically), check whether the controller currently holds non-terminal jobs for that host and, if so, arm a reconciliation round for it
+  - Send the round's query to the worker as a follow-up exchange after the join reply and the existing configuration push have both already been sent, not as an extension of the join call itself
+  - Observable completion: rejoining a worker with known non-terminal jobs results in a reconciliation round being armed and the corresponding query being sent to that worker, in that order, after the join reply; rejoining a worker with none results in no round and no query
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2_
+  - _Depends: 1.4, 3.1_
+
+- [ ] 6.2 Route an incoming job-state report to reconciliation processing
+  - Dispatch an incoming job-state report message to the reconciliation-processing logic, correctly reassembling a report that arrives across multiple pages before it is applied
+  - Confirm whether the two new message kinds need an explicit redirect response when received by a controller that is not currently primary, matching how every other primary-only message already behaves for a standby recipient — do not assume the existing generic rejection path already covers this without checking, given this exact class of gap has recurred for other message kinds before
+  - Observable completion: a report sent across several pages is fully reassembled and applied exactly once, with the same result as an equivalent single-page report; and a query or report sent to a non-primary controller receives the same redirect behavior an equivalent primary-only message already receives, not a bare unrecoverable error
+  - _Requirements: 8.1, 8.3_
+  - _Depends: 1.4, 3.2_
+
+- [ ] 7. Validation: end-to-end and regression coverage
+- [ ] 7.1 Full rejoin-reconciliation round trip
+  - Exercise a real rejoin against a live controller and worker pair: a worker with a mix of still-running, finished-success, finished-failure, and unrecognized jobs reconnects, and every job ends up in the correct final state with the scheduler notified
+  - Observable completion: the round trip produces the correct end state for every job kind in one live test run, matching the per-outcome behavior verified in isolation by tasks 2.1-2.2 and 3.2
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, 3.2, 3.3, 3.6, 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 7.1, 7.2, 7.3_
+  - _Depends: 6.1, 6.2, 4.3, 5_
+
+- [ ] 7.2 Failover-during-job recovery
+  - Reproduce the feature's core motivating scenario: a job's finish report is lost across a controller primary failover, and the new primary's rejoin reconciliation with that worker recovers the job's true outcome
+  - Observable completion: a job whose finish notification never reached any controller instance ends up correctly finished, not stuck running, once the worker rejoins the new primary
+  - _Requirements: 4.1, 4.2, 4.6_
+  - _Depends: 7.1_
+
+- [ ] 7.3 Burst/array recovery unbounded by job count
+  - Reproduce a burst of many near-simultaneous job completions on one worker host where most of the individual finish reports are lost, and confirm reconciliation recovers all of them on the worker's next rejoin regardless of count
+  - Observable completion: every job in the burst is correctly finished after reconciliation, with none left stuck and none duplicated
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Depends: 7.1_
+
+- [ ] 7.4 Large job-count pagination correctness
+  - Drive the reconciliation exchange for a worker running enough jobs to require multiple pages of both the query and the report, confirming the exchange completes correctly rather than truncating or silently dropping jobs past a single page
+  - Observable completion: a job count an order of magnitude larger than a single page holds is fully queried and fully reported, with every job accounted for
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Depends: 7.1_