feat(worker-rejoin-resync): reconciliation round lifecycle (task 3.1)
New src/controller/resync.c/resync.h: resync_arm_round(cluster_t*, worker_t*) enumerates every non-terminal job assigned to a host, populates worker_t.resync (round_token, query job-ID set, deadline), sets XCE_WORKER_RECONCILING, and schedules a non-blocking delaymgr one-shot timeout. resync_resolve_round() clears the round; a fired timeout resolves it too, via resync_round_is_current()'s by-value round-token check -- delay.c/delay.h expose no true cancel/dequeue primitive, so a stale timer from an already-superseded round becomes a safe no-op instead of touching a newer round's live state (verified safe against both misresolving the wrong round and use-after-free: the timeout closure never holds a pointer into resync_state_t itself). 5 new tests: query-set correctness, fresh-token-per-arm, the RECONCILING flag's set/clear lifecycle, and a direct (non-blocking) timeout-fire proof via an XCE_TESTING-gated seam confirmed compiled out of the production binary. Full suite: 1232 tests, 0 failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
ea0658ecf4
commit
442fcf4544
5 changed files with 927 additions and 1 deletions
|
|
@ -62,7 +62,7 @@
|
|||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 3. Core: controller reconciliation-round bookkeeping
|
||||
- [ ] 3.1 (P) Track one reconciliation round per worker with a bounded timeout
|
||||
- [x] 3.1 (P) Track one reconciliation round per worker with a bounded timeout
|
||||
- Given a worker host with one or more non-terminal jobs currently assigned to it, construct the exact list of those jobs' unique IDs as the round's query set
|
||||
- Track a round identifier that distinguishes the current reconciliation attempt from any prior one for the same worker
|
||||
- Schedule a bounded timeout for the round using a non-blocking trigger; when it fires before the round otherwise resolves, resolve the round rather than leaving the worker withheld from dispatch indefinitely
|
||||
|
|
@ -156,3 +156,4 @@
|
|||
- Task 1.5: `xceadmin config dump`'s `_format_cluster_config` (`build.py`) does not print `workerResyncWindowMs`/`workerResyncTimeoutMs` — an operator can set them via `config build` but can't verify the built value via `config dump`. Confirmed genuinely out of task 1.5's boundary (config-*build*, not config-*dump*) and not required by Requirements 3.7/5.3 as written, but a real, minor observability gap worth a small follow-up task if this becomes a live operator pain point. Not blocking; not added to this plan.
|
||||
- **Task 2.2 found and fixed bug-1746**, a real pre-existing defect in `persist.c` (part of bug-1678's original fix, predates this feature): `capnp_encode_joballoc()` unconditionally dereferenced a NULL `alloc` via `encode_JobAlloc()` instead of the already-existing NULL-safe `encode_JobAlloc_ptr()` wrapper — never triggered before since `dispatch_a_job_` (the only prior caller) always passes non-NULL. This directly contradicted design.md's claim that the NULL-alloc case was "tested, backward-compatible" on both apply seams — true for *decode*, never actually true for *encode* until this task exercised it. Fixed with a minimal guard producing a true zero-length buffer (not a capnp-null-pointer encoding — verified against `apply_committed_entry`'s `alloc_len > 0` decode gate, which a non-zero-length "null" encoding would have wrongly satisfied). design.md corrected in 4 places. **Any future task calling `persist_submit_job_dispatch_fact` with a NULL alloc can now trust this path — it wasn't safe before task 2.2, it is now.**
|
||||
- Task 2.2 (self-correction): a coordinator-applied one-line comment fix to `job.c` accidentally closed the enclosing `/* */` comment block early, leaving the comment's remaining lines as bare (uncompilable) code — caught immediately by re-running the build before marking the task complete, not left for review to find. Reinforces why fresh-evidence re-verification applies even to "trivial, obviously-safe" doc-only edits made outside the TDD subagent loop.
|
||||
- Task 3.1: `resync.c`'s `resync_timeout_ms()` applies its own private default-if-unset fallback (15000ms) for `workerResyncTimeoutMs` rather than following `myconf.c`'s established `SET_DEFAULT_IF_UNSET`-style canonicalization convention used for the sibling `heartbeat_interval` field (`myconf.c:74`) — `myconf.c` is outside task 3.1's boundary, so this was correctly left as-is, but if a SECOND consumer of `workerResyncTimeoutMs` (or `workerResyncWindowMs`) is ever added, it would have to either duplicate this private fallback or the two could silently disagree. Worth canonicalizing in `myconf.c` if/when that happens — not needed for this feature's own single consumer. delaymgr timer cancellation: `delay.c`/`delay.h` expose no dequeue/cancel primitive for an already-armed one-shot task; `resync.c` handles this correctly via a by-value round-token check inside the fired callback (verified safe against both the logical "resolves the wrong round" bug and use-after-free, since the callback's closure never holds a pointer into the freed `resync_state_t`) — any future delaymgr one-shot user in this codebase facing the same "need to cancel" problem should use this same by-value-token-check pattern rather than assuming a cancel API exists.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue