feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)

Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.

Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.

Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.

Full controller suite: 1239 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 15:24:29 +00:00
parent 442fcf4544
commit eec2596c5d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1600 additions and 7 deletions

View file

@ -29,7 +29,7 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
### This Spec Owns
- The wire protocol for the reconciliation exchange: two new opcodes, `OPCode_jobStateQuery` (controller → worker) and `OPCode_jobStateReport` (worker → controller), both paginated.
- Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign); gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly *after* the worker-join RPC (`xce_join`/`xcecli_join`) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
- Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign) — **Correction (2026-09-01, bug-1747, task 3.2): a third new function, `record_job_finish_fact_at`, was added for the finished-success outcome's own Requirement 4.2 fix; see the Persistence Field-Completeness Review section's own correction note for the full rationale** — gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly *after* the worker-join RPC (`xce_join`/`xcecli_join`) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
- Worker-side: a new short-lived retention of recently-finished job outcomes (jobid, exit status, real finish time), bounded by a configurable window; answering the controller's query from that retention plus the still-running job list; reporting extra jobs outside the query; a new local gate flag that delays the worker's own resumption of normal post-join operation until its report has finished sending (Requirement 3.8).
- Scheduler-side: one narrow addition — the existing candidate-host filter (`scheduler.c:1211`, today `IS_WORKER_UNAVAIL`-only) also excludes a host with `XCE_WORKER_RECONCILING` set, so a RECONCILING host is never proposed as a dispatch target in the first place (Requirement 5.1). No other scheduler-side change; the worker-status mirror this reads from already exists (`core_events.c:527`/`:595`) and needs no new wire content for this bit.
- Two new shared-config fields (the Requirement 3.7 reporting window; the Requirement 5.3 reconciliation timeout).
@ -86,7 +86,7 @@ graph TB
RW[rejoin_worker / apply_join_state / worker.c]
RS[Reconciliation state: worker_t.resync]
RQ[Reconciliation query+timeout / new resync.c]
JOBC[job.c: record_job_finish_fact / record_job_fail_fact<br/>record_job_reconcile_pending_fact NEW<br/>record_job_reconcile_assign_fact NEW]
JOBC[job.c: record_job_finish_fact / record_job_fail_fact<br/>record_job_reconcile_pending_fact NEW<br/>record_job_reconcile_assign_fact NEW<br/>record_job_finish_fact_at NEW - bug-1747]
COMMIT[commit.c / apply_committed_entry — unchanged, reused]
HOOK[XCE_CONTROLLER_HOOK_JOB fan-out]
SCHEDC[sched.c hook callbacks]
@ -120,7 +120,7 @@ graph TB
**Architecture Integration**:
- Selected pattern: additive extension of an existing daemon-to-daemon RPC + existing fact/apply pipeline — no new subsystem, no new process.
- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (two new functions, rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is) / scheduler candidate-filter (one added disjunct to an existing check, see below — not zero, but not new machinery either).
- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (three new functions — `record_job_reconcile_pending_fact`, `record_job_reconcile_assign_fact`, and `record_job_finish_fact_at` (**Correction, 2026-09-01, bug-1747**: added by task 3.2 for Requirement 4.2's finished-success fix; the original count of two predates that fix) — rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is) / scheduler candidate-filter (one added disjunct to an existing check, see below — not zero, but not new machinery either).
- Existing patterns preserved: commit-then-apply idempotency, `{pos,count,n_total}` pagination, hook→notify scheduler path, per-worker write-lock discipline, and — newly leaned on — the scheduler's existing worker-status-bit candidate filter (`IS_WORKER_UNAVAIL` at `scheduler.c:1211`).
- New components rationale: reconciliation bookkeeping and worker retention have no existing analog to extend; everything else is reuse.
- **Scheduler-side touch, and why it doesn't cross `test_no_worker_resync_symbols_in_scheduler_publish_path`'s fence**: that standing test guards `scheduler.c`/`driver.c`'s *outbound decision-publish* direction against ever learning reconciliation *protocol* concepts (round tokens, query/report messages, retention windows). The one line this design adds there — skip a host if `IS_WORKER_RECONCILING(w->status)` — is not that: it's the same *category* of check as the pre-existing `IS_WORKER_UNAVAIL(w->status)` line right next to it, reading an ordinary worker-status bit the scheduler already mirrors for an unrelated reason (`core_events.c:527`/`:595`). No opcode, no reconciliation-round concept, and no symbol named "resync" is introduced into either file. This is called out explicitly, not left for an implementer to discover the fence and guess whether it applies.
@ -144,7 +144,7 @@ graph TB
- `src/schema/controller.capnp` — add `OPCode_jobStateQuery`, `OPCode_jobStateReport` to the `OPCode` enum; add `JobStateQuery{jobids: List(UInt64), pos, count, ntotal}`, `JobStateReport{entries: List(JobStateEntry), pos, count, ntotal}`, `JobStateEntry{jobid, status: JobStateStatus, exitCode, finishTimeMs, queried: Bool}` (`queried=false` marks a Requirement 3.6 extra job); add both as `Request`/`Reply` union arms; add `workerResyncWindowMs`/`workerResyncTimeoutMs` fields to `ClusterConfig`.
- `src/controller/worker.c` — `rejoin_worker`/`apply_join_state`: after clearing `XCE_WORKER_UNAVAIL`, check for non-terminal jobs assigned to this host (Requirement 1) and, if any, set `XCE_WORKER_RECONCILING` and initialize the per-worker reconciliation record before releasing the write lock. No change to admin-bit preservation or attribute/endpoint update behavior. `on_worker_join` (worker.c:623) — extended, not `handle_worker_join`: it already calls `worker_push_config` as its one existing step (the config the worker needs — hosttype/hostmodel/resource variables — is foundational baseline state, and its own stub comment, `/* consider to push jobs on hosts to */`, already anticipates this exact follow-on). The reconciliation query send is added right after it, in that fixed order (config push, then query) — both sent fire-and-forget, no new ack/reply invented for this. This ordering is safe, not just conventional: the worker's incoming-request dispatch (`myreq_handler`, `controller_connector.c:1563`) is a plain synchronous `switch` invoked one message at a time off its single long-lived client connection (`xcecli_ctx`) — confirmed by reading it, not assumed — so the two requests are not just sent in order, they are *handled* to completion in that same order on the worker side; no thread-pool fan-out is in the path (the worker's own `thread_pool_t` is for job-forking work, unrelated to request dispatch). The actual wait in this design belongs at the *end* of the sequence, not in the middle: the controller does not consider the worker's rejoin/reconciliation complete until the job-state *report* arrives (or the Requirement 5.3 timeout fires) — see Error Handling.
- `src/controller/join.c` — **no change**. `handle_worker_join` itself is untouched; the reconciliation query is sent from the (extended) `on_worker_join`, which `handle_worker_join` already calls unconditionally after the join reply.
- `src/controller/job.c` — two new functions, both following `dispatch_a_job_`'s eager-mutation-plus-explicit-notify pattern (not `record_job_finish_fact`'s deferred one — see Components for why): `record_job_reconcile_pending_fact()` (Requirement 4.6, eagerly calls `apply_job_update_transition(job, JOB_STAT_PEND)` under the write lock, releases `job->job.allocs` — `free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;` — on an applied (non-no-op) transition, durably appends via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)`, then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking; this is bug-1631's own documented candidate fix, applied here because Requirement 4.6 is exactly the "RUN job re-admitted to the PEND pool" precondition bug-1631's fix was deferred pending), `record_job_reconcile_assign_fact()` (Requirement 4.8, eagerly sets `job->job.state`/host assignment under the write lock, calls the existing `persist_submit_job_dispatch_fact(c->persist, jobid, state, ts, alloc=NULL)` — reusing bug-1678's already-tested NULL-alloc decode path on both apply seams verbatim, not a new fact shape — then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking, same as above and for the same reason). Existing `record_job_finish_fact`/`record_job_fail_fact` are called as-is for the finished-success/finished-failure cases. Also: split `dispatch_a_job_`'s `w->ep == NULL || w->host.status != 0` check (job.c:1511-1512) into two branches, checked in order — `w->ep == NULL` is resolved directly via `record_job_reconcile_pending_fact()` with **no retry-counter increment and no `mark_job_fail()`**, *before* the code ever reaches `xce_classify_dispatch_fail()` (a deliberate bypass, not a third value on the shared `xce_dispatch_fail_class_t` enum — see Components); `w->host.status != 0` keeps its existing `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY` behavior unchanged (now rarely hit for `XCE_WORKER_RECONCILING` specifically, since the scheduler-side filter below keeps it from being selected in the first place).
- `src/controller/job.c` — **Correction (2026-09-01, bug-1747, task 3.2): three new functions, not two — see below.** `record_job_reconcile_pending_fact()` and `record_job_reconcile_assign_fact()` both follow `dispatch_a_job_`'s eager-mutation-plus-explicit-notify pattern (not `record_job_finish_fact`'s deferred one — see Components for why): `record_job_reconcile_pending_fact()` (Requirement 4.6, eagerly calls `apply_job_update_transition(job, JOB_STAT_PEND)` under the write lock, releases `job->job.allocs` — `free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;` — on an applied (non-no-op) transition, durably appends via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)`, then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking; this is bug-1631's own documented candidate fix, applied here because Requirement 4.6 is exactly the "RUN job re-admitted to the PEND pool" precondition bug-1631's fix was deferred pending), `record_job_reconcile_assign_fact()` (Requirement 4.8, eagerly sets `job->job.state`/host assignment under the write lock, calls the existing `persist_submit_job_dispatch_fact(c->persist, jobid, state, ts, alloc=NULL)` — reusing bug-1678's already-tested NULL-alloc decode path on both apply seams verbatim, not a new fact shape — then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking, same as above and for the same reason). Existing `record_job_finish_fact`/`record_job_fail_fact` were originally scoped to be called as-is for the finished-success/finished-failure cases; **found live during task 3.2 implementation that `record_job_finish_fact`'s specific call site has no way to carry the worker-reported finish time (Requirement 4.2) — its own primitive, `persist_submit_job_finish_fact`, accepts one, but `record_job_finish_fact` always derives `time(NULL)` instead.** Fixed with a THIRD new function, `record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms)` — `record_job_finish_fact`'s own worker-reported-finish-time sibling, following `record_job_finish_fact`'s own DEFERRED pattern (durable append only, not eager like the other two new functions above), differing only in passing the caller-supplied (ms→s converted) time to `persist_submit_job_finish_fact` instead of `time(NULL)`; `record_job_finish_fact` itself stays unmodified, still used as-is for `on_job_finish`'s ordinary (non-reconciliation) receipt path. `record_job_fail_fact` is still called as-is for the finished-failure case — Requirement 4.2 remains genuinely unsatisfiable there via any existing path (see the Persistence Field-Completeness Review section's correction note; tracked as the FAIL half of bug-1747, deliberately deferred). Also: split `dispatch_a_job_`'s `w->ep == NULL || w->host.status != 0` check (job.c:1511-1512) into two branches, checked in order — `w->ep == NULL` is resolved directly via `record_job_reconcile_pending_fact()` with **no retry-counter increment and no `mark_job_fail()`**, *before* the code ever reaches `xce_classify_dispatch_fail()` (a deliberate bypass, not a third value on the shared `xce_dispatch_fail_class_t` enum — see Components); `w->host.status != 0` keeps its existing `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY` behavior unchanged (now rarely hit for `XCE_WORKER_RECONCILING` specifically, since the scheduler-side filter below keeps it from being selected in the first place).
- `src/controller/cluster.c` — new `case OPCode_jobStateReport:` in `myreq_handler`'s dispatch switch (runs after `ha_dispatch_gate`); route to the new reconciliation-apply logic.
- `src/scheduler/src/scheduler.c` — extend the existing candidate-host filter (`scheduler.c:1211`, today `if (IS_WORKER_UNAVAIL(w->status)) { ... }`) to also skip a host with the new reconciling bit set — same shape, one added disjunct, no new call site.
- `src/include/xce/types.h` — add `XCE_WORKER_RECONCILING` to the worker status bitmask plus an `IS_WORKER_RECONCILING(status)` macro (mirroring `IS_WORKER_UNAVAIL`'s existing shape) for the scheduler-side filter above; add a `resync_state_t` (queried job-ID set, round token, deadline) referenced from `worker_t`.
@ -416,12 +416,14 @@ Per `design-principles.md`'s Persistence Review requirement and `persistence.md`
| Outcome | Fact path | New fields? | Both apply seams? |
|---|---|---|---|
| Finished success | `record_job_finish_fact` (existing, reused as-is) | None — existing complete fact shape (already carries `finish_time`, exit info); the reconciliation caller passes the worker-reported `finishTimeMs` (Requirement 3.5) as that fact's finish time, never the time the controller processed the report (Requirement 4.2) | Already covered by existing `apply_committed_entry`/standby-catch-up handling — unchanged |
| Finished success | `record_job_finish_fact_at` (NEW, task 3.2/bug-1747) | None — reuses `persist_submit_job_finish_fact`'s existing, already-complete `WAL_JOB_FINISH` payload shape verbatim (`finish_time`, exit info); the reconciliation caller passes the worker-reported `finishTimeMs` (Requirement 3.5, converted ms→s) as that fact's finish time, never the time the controller processed the report (Requirement 4.2) | Already covered by existing `apply_committed_entry`/standby-catch-up handling — unchanged, since the wire payload shape is identical to `record_job_finish_fact`'s own |
| Finished failure | `record_job_fail_fact` (existing, reused as-is) | None — same Requirement 4.2 worker-reported-time rule applies | Unchanged, existing |
| Still running | No fact — pure no-op against the outstanding set | N/A | N/A |
| Unrecognized → pending | `persist_submit_job_fact(..., WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)` (existing generic primitive, new call site) | None — the plain `{jobid, state}` shape is sufficient, same as every other `WAL_JOB_UPDATE` caller. **`job->job.allocs` release is in-memory only, eager, not itself a persisted field** — mutated live under the same lock as the state change (mirroring `dispatch_a_job_`'s own success path in reverse; see Components), then notified via `persist_notify_job_updated_already_persisted`, not left for `apply_committed_entry` to discover | Both seams already decode plain `WAL_JOB_UPDATE` identically today (it is not a new payload shape) — a replay of this fact re-derives `job->job.state = JOB_STAT_PEND` correctly; it does **not** re-derive the allocs release (that mutation is not wire content), but since `dispatch_a_job_`'s eager pattern already established this exact asymmetry (live mutation + explicit notify now, replay-idempotent state field only, no replay-time side effect needed for a purely in-memory cleanup), this is consistent with existing precedent, not a new gap |
| Unqueried job → assign | `persist_submit_job_dispatch_fact(..., alloc=NULL)` (existing primitive, new call site, `record_job_reconcile_assign_fact`) | None — reuses `persist_submit_job_dispatch_fact`'s existing `WAL_JOB_DISPATCH` payload shape verbatim with `alloc=NULL`, exercising the same NULL-alloc decode branch bug-1678 already added. Live mutation is eager (see Components), matching `dispatch_a_job_`'s own pattern for the same fact type | Decode side already covered by bug-1678 on both seams. **Encode side required a fix** (bug-1746, task 2.2): `capnp_encode_joballoc` unconditionally dereferenced a NULL `alloc` — this was this codebase's first genuinely-NULL caller. Fixed with a minimal zero-length-buffer guard, matching the decode side's `alloc_len > 0` contract exactly; no new decode path needed once the encode side actually produces the zero-length payload the decode side already expected |
**Correction (2026-09-01, found live during task 3.2 implementation, bug-1747) — a THIRD new function, not two:** this section's original "Finished success" row asserted `record_job_finish_fact` could be reused "as-is," citing an "existing complete fact shape (already carries `finish_time`)" — accurate about `persist_submit_job_finish_fact` (the underlying primitive, which does accept an explicit `finish_time` parameter) but not about `record_job_finish_fact`'s own specific call site, which never exposed that parameter to a caller and always derives `time(NULL)` at receipt time instead. Calling `record_job_finish_fact` unmodified therefore could NOT satisfy Requirement 4.2 for the finished-success case — confirmed live when task 3.2 wired the reconciliation caller and found no way to pass the worker-reported time through. Fixed (coordinator-directed, mirroring bug-1746's own precedent for an in-boundary companion fix) by adding `record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms)` (job.c) — `record_job_finish_fact`'s own worker-reported-finish-time sibling: an identical guard/hook-fire/durable-append body, differing only in passing the caller-supplied (ms→s converted) time to `persist_submit_job_finish_fact` instead of `time(NULL)`. `record_job_finish_fact` itself is left UNMODIFIED — its own caller (`on_job_finish`, an ordinary near-instantaneous worker report) is correctly served by receipt-time there. Unlike `record_job_reconcile_pending_fact`/`record_job_reconcile_assign_fact` (both EAGER, `dispatch_a_job_`-style), `record_job_finish_fact_at` follows `record_job_finish_fact`'s own DEFERRED pattern (durable append only; the in-memory mutation and scheduler notify happen later via `apply_committed_entry`'s replay, exactly as an ordinary finish already works) — so this section's job.c fact-recording count is **three new functions**, not two (see also the Architecture Pattern & Boundary Map bullet and the Modified Files entry for `job.c`, both correspondingly updated). The sibling "Finished failure" outcome remains genuinely unfixable via any existing path: `WAL_JOB_FAIL`'s wire payload carries no finish-time slot at any layer, for any caller in this codebase, and `apply_job_fail_transition` unconditionally stamps `time(NULL)` at apply time — a pre-existing, systemic gap that predates this feature entirely (bug-1747, re-scoped 2026-09-01), deliberately left deferred rather than patched narrowly inside this task.
**Documented scope decision — `job->allocs` on a reconciliation-assigned job (Requirement 4.8):** when the controller reconciles a job it never held any prior record of (e.g. it was dispatched by a controller instance before a failover whose dispatch fact was itself lost), it has no way to learn what resource allocation the scheduler originally computed for that job — the worker's report carries the job's execution outcome, not its original resource request. This design deliberately does not attempt to reconstruct `job->allocs` for this case. **Correction (2026-09-01, re-validated against current code):** an earlier draft of this section cited `persistence.md`'s `APPLY_DIVERGENT` section / bug-1678 as evidence this is "a tracked, separate gap" `WAL_JOB_DISPATCH` still has — that citation was stale. Bug-1678 was fixed 2026-08-25 (before this design was finalized): `WAL_JOB_DISPATCH` now durably carries `job->allocs` when one exists, on both apply/decode seams. The real reason this design still writes no allocation for Requirement 4.8 is simpler and unrelated to that gap: there just isn't one to write — the scheduler, not this controller, is the only party that ever computes a job's allocation, and for a job this controller never had any record of, no dispatch (successful or otherwise) ever ran through this controller to produce one. `record_job_reconcile_assign_fact` reuses the *fixed* `persist_submit_job_dispatch_fact` primitive because its NULL-alloc *decode* case is already the correct representation of "state and host known, allocation not" — though the *encode* side of that same NULL case had never actually been exercised before this feature (bug-1746: `capnp_encode_joballoc` dereferenced a NULL `alloc` unconditionally, since `dispatch_a_job_`, bug-1678's only prior caller, always passes a real allocation). Task 2.2 fixed the encode side with a minimal zero-length-buffer guard as part of making this call safe to actually make — a real, if narrow, gap in the "already-tested" characterization this design originally gave the primitive, closed as part of implementation rather than left latent. The practical consequence is bounded by `resource-accounting.md`'s existing policy: a job with an unknown/zero reservation is charged `max(reserved=0, actual)` — i.e., its real usage — the moment its next rusage tick arrives, identical to how a legitimately-unreserved job is already handled today. The job's *state* is correct immediately; its resource *accounting* converges within one rusage cycle, not instantly. No fabricated allocation value is ever written.
### Configuration Model

View file

@ -61,7 +61,7 @@
- _Boundary: job.c additions_
- _Depends: 2.1_
- [ ] 3. Core: controller reconciliation-round bookkeeping
- [x] 3. Core: controller reconciliation-round bookkeeping
- [x] 3.1 (P) Track one reconciliation round per worker with a bounded timeout
- Given a worker host with one or more non-terminal jobs currently assigned to it, construct the exact list of those jobs' unique IDs as the round's query set
- Track a round identifier that distinguishes the current reconciliation attempt from any prior one for the same worker
@ -70,7 +70,7 @@
- _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 5.3, 6.1_
- _Boundary: resync.c_
- [ ] 3.2 Apply each reported job outcome through the correct idempotent path
- [x] 3.2 Apply each reported job outcome through the correct idempotent path
- For each entry in a worker's job-state report: a still-running job is cleared from the round's outstanding set with no fact recorded; a finished-success or finished-failure job is applied through the existing finish/fail recording used for an ordinary report; an unrecognized queried job is applied through task 2.1's pending-and-release path; a reported job outside the original query is applied through task 2.2's assignment path
- Discard a report that arrives for a round already superseded by a newer one for the same worker, rather than applying it
- When the round's outstanding set becomes empty, resolve the round so the worker becomes eligible for dispatch again
@ -157,3 +157,4 @@
- **Task 2.2 found and fixed bug-1746**, a real pre-existing defect in `persist.c` (part of bug-1678's original fix, predates this feature): `capnp_encode_joballoc()` unconditionally dereferenced a NULL `alloc` via `encode_JobAlloc()` instead of the already-existing NULL-safe `encode_JobAlloc_ptr()` wrapper — never triggered before since `dispatch_a_job_` (the only prior caller) always passes non-NULL. This directly contradicted design.md's claim that the NULL-alloc case was "tested, backward-compatible" on both apply seams — true for *decode*, never actually true for *encode* until this task exercised it. Fixed with a minimal guard producing a true zero-length buffer (not a capnp-null-pointer encoding — verified against `apply_committed_entry`'s `alloc_len > 0` decode gate, which a non-zero-length "null" encoding would have wrongly satisfied). design.md corrected in 4 places. **Any future task calling `persist_submit_job_dispatch_fact` with a NULL alloc can now trust this path — it wasn't safe before task 2.2, it is now.**
- Task 2.2 (self-correction): a coordinator-applied one-line comment fix to `job.c` accidentally closed the enclosing `/* */` comment block early, leaving the comment's remaining lines as bare (uncompilable) code — caught immediately by re-running the build before marking the task complete, not left for review to find. Reinforces why fresh-evidence re-verification applies even to "trivial, obviously-safe" doc-only edits made outside the TDD subagent loop.
- Task 3.1: `resync.c`'s `resync_timeout_ms()` applies its own private default-if-unset fallback (15000ms) for `workerResyncTimeoutMs` rather than following `myconf.c`'s established `SET_DEFAULT_IF_UNSET`-style canonicalization convention used for the sibling `heartbeat_interval` field (`myconf.c:74`) — `myconf.c` is outside task 3.1's boundary, so this was correctly left as-is, but if a SECOND consumer of `workerResyncTimeoutMs` (or `workerResyncWindowMs`) is ever added, it would have to either duplicate this private fallback or the two could silently disagree. Worth canonicalizing in `myconf.c` if/when that happens — not needed for this feature's own single consumer. delaymgr timer cancellation: `delay.c`/`delay.h` expose no dequeue/cancel primitive for an already-armed one-shot task; `resync.c` handles this correctly via a by-value round-token check inside the fired callback (verified safe against both the logical "resolves the wrong round" bug and use-after-free, since the callback's closure never holds a pointer into the freed `resync_state_t`) — any future delaymgr one-shot user in this codebase facing the same "need to cancel" problem should use this same by-value-token-check pattern rather than assuming a cancel API exists.
- **Task 3.2 needed 3 review rounds — a genuine cautionary tale about a boolean short-circuit hiding a second, adjacent bug.** Round 1 caught the unqueried-extra-"running" branch gating on terminal-vs-non-terminal instead of Req 4.8's real trigger (assignment status), silently leaving a job stuck PEND when its `WAL_JOB_DISPATCH` fact never reached durable commit before a failover — the feature's own core motivating scenario. The FIRST fix (`!terminal && assigned_to_w's_host`) solved that but introduced a NEW, distinct Req 4.4 violation via the same `&&`'s short-circuit: a TERMINAL job (DONE/EXIT) falls into the same `else`/assign branch and gets clobbered back to RUN — caught by round 2, and confirmed the implementer's own new test had been asserting the bug as correct behavior (a green suite is not proof of correctness when the test itself encodes the wrong expectation). The SECOND fix (a genuine three-way `if`/`else if`/`else`: terminal→noop, non-terminal-and-already-assigned-anywhere→noop, non-terminal-and-unassigned→assign) passed round 3's from-scratch re-derivation plus an explicit hunt for job==NULL reachability. **Lesson for future tasks in this codebase**: when a guard combines two orthogonal conditions with `&&`/`||` and only ONE of the two is corrected after a review finding, re-derive the FULL truth table before re-submitting — a targeted patch to the specific counterexample a reviewer names can leave an adjacent, differently-shaped counterexample in the same boolean untouched. bug-1747 (finish-time gap, split fixed) and bug-1748 (this guard defect) both logged.