feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)
Add resync_apply_report()/resync_apply_running_entry()/ resync_apply_finished_entry()/resync_apply_unrecognized_entry() to resync.c: routes each entry in a worker's job-state report to the correct existing idempotent path (still-running -> clear from outstanding set, no fact; finished-success/failure -> record_job_finish_fact_at/ on_job_fail; unrecognized -> record_job_reconcile_pending_fact; unqueried extra -> record_job_reconcile_assign_fact or no-op), discards reports for superseded rounds (Req 6.1/6.3), and resolves the round once its outstanding set empties. Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(), additive alongside the unmodified record_job_finish_fact, so a finished-success reconciliation entry records the WORKER-reported finish time instead of reconciliation-processing time -- proven via a real WAL-write + persist_recover() restart-replay test. The FAIL-side half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at all, for any caller) is confirmed pre-existing and systemic, correctly left out of this task's scope. Three review rounds surfaced and fixed two real, distinct bugs in the unqueried-extra-job routing guard (see tasks.md's Implementation Notes for the full account and the general lesson about re-deriving a full truth table after a partial boolean-guard fix): round 1 caught the wrong axis (terminal-vs-non-terminal instead of assignment-status, Req 4.8); round 2 caught that the first fix's short-circuit clobbered an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including confirming the implementer's own test had certified the bug as correct; round 3 independently re-derived the final three-way guard from scratch and found no further defect. bug-1748 logged for the guard fix. Full controller suite: 1239 tests, 0 failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
442fcf4544
commit
eec2596c5d
8 changed files with 1600 additions and 7 deletions
|
|
@ -29,7 +29,7 @@ On every worker (re)join — a fresh network reconnect after a disconnect, or a
|
|||
|
||||
### This Spec Owns
|
||||
- The wire protocol for the reconciliation exchange: two new opcodes, `OPCode_jobStateQuery` (controller → worker) and `OPCode_jobStateReport` (worker → controller), both paginated.
|
||||
- Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign); gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly *after* the worker-join RPC (`xce_join`/`xcecli_join`) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
|
||||
- Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign) — **Correction (2026-09-01, bug-1747, task 3.2): a third new function, `record_job_finish_fact_at`, was added for the finished-success outcome's own Requirement 4.2 fix; see the Persistence Field-Completeness Review section's own correction note for the full rationale** — gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly *after* the worker-join RPC (`xce_join`/`xcecli_join`) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
|
||||
- Worker-side: a new short-lived retention of recently-finished job outcomes (jobid, exit status, real finish time), bounded by a configurable window; answering the controller's query from that retention plus the still-running job list; reporting extra jobs outside the query; a new local gate flag that delays the worker's own resumption of normal post-join operation until its report has finished sending (Requirement 3.8).
|
||||
- Scheduler-side: one narrow addition — the existing candidate-host filter (`scheduler.c:1211`, today `IS_WORKER_UNAVAIL`-only) also excludes a host with `XCE_WORKER_RECONCILING` set, so a RECONCILING host is never proposed as a dispatch target in the first place (Requirement 5.1). No other scheduler-side change; the worker-status mirror this reads from already exists (`core_events.c:527`/`:595`) and needs no new wire content for this bit.
|
||||
- Two new shared-config fields (the Requirement 3.7 reporting window; the Requirement 5.3 reconciliation timeout).
|
||||
|
|
@ -86,7 +86,7 @@ graph TB
|
|||
RW[rejoin_worker / apply_join_state / worker.c]
|
||||
RS[Reconciliation state: worker_t.resync]
|
||||
RQ[Reconciliation query+timeout / new resync.c]
|
||||
JOBC[job.c: record_job_finish_fact / record_job_fail_fact<br/>record_job_reconcile_pending_fact NEW<br/>record_job_reconcile_assign_fact NEW]
|
||||
JOBC[job.c: record_job_finish_fact / record_job_fail_fact<br/>record_job_reconcile_pending_fact NEW<br/>record_job_reconcile_assign_fact NEW<br/>record_job_finish_fact_at NEW - bug-1747]
|
||||
COMMIT[commit.c / apply_committed_entry — unchanged, reused]
|
||||
HOOK[XCE_CONTROLLER_HOOK_JOB fan-out]
|
||||
SCHEDC[sched.c hook callbacks]
|
||||
|
|
@ -120,7 +120,7 @@ graph TB
|
|||
|
||||
**Architecture Integration**:
|
||||
- Selected pattern: additive extension of an existing daemon-to-daemon RPC + existing fact/apply pipeline — no new subsystem, no new process.
|
||||
- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (two new functions, rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is) / scheduler candidate-filter (one added disjunct to an existing check, see below — not zero, but not new machinery either).
|
||||
- Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (three new functions — `record_job_reconcile_pending_fact`, `record_job_reconcile_assign_fact`, and `record_job_finish_fact_at` (**Correction, 2026-09-01, bug-1747**: added by task 3.2 for Requirement 4.2's finished-success fix; the original count of two predates that fix) — rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is) / scheduler candidate-filter (one added disjunct to an existing check, see below — not zero, but not new machinery either).
|
||||
- Existing patterns preserved: commit-then-apply idempotency, `{pos,count,n_total}` pagination, hook→notify scheduler path, per-worker write-lock discipline, and — newly leaned on — the scheduler's existing worker-status-bit candidate filter (`IS_WORKER_UNAVAIL` at `scheduler.c:1211`).
|
||||
- New components rationale: reconciliation bookkeeping and worker retention have no existing analog to extend; everything else is reuse.
|
||||
- **Scheduler-side touch, and why it doesn't cross `test_no_worker_resync_symbols_in_scheduler_publish_path`'s fence**: that standing test guards `scheduler.c`/`driver.c`'s *outbound decision-publish* direction against ever learning reconciliation *protocol* concepts (round tokens, query/report messages, retention windows). The one line this design adds there — skip a host if `IS_WORKER_RECONCILING(w->status)` — is not that: it's the same *category* of check as the pre-existing `IS_WORKER_UNAVAIL(w->status)` line right next to it, reading an ordinary worker-status bit the scheduler already mirrors for an unrelated reason (`core_events.c:527`/`:595`). No opcode, no reconciliation-round concept, and no symbol named "resync" is introduced into either file. This is called out explicitly, not left for an implementer to discover the fence and guess whether it applies.
|
||||
|
|
@ -144,7 +144,7 @@ graph TB
|
|||
- `src/schema/controller.capnp` — add `OPCode_jobStateQuery`, `OPCode_jobStateReport` to the `OPCode` enum; add `JobStateQuery{jobids: List(UInt64), pos, count, ntotal}`, `JobStateReport{entries: List(JobStateEntry), pos, count, ntotal}`, `JobStateEntry{jobid, status: JobStateStatus, exitCode, finishTimeMs, queried: Bool}` (`queried=false` marks a Requirement 3.6 extra job); add both as `Request`/`Reply` union arms; add `workerResyncWindowMs`/`workerResyncTimeoutMs` fields to `ClusterConfig`.
|
||||
- `src/controller/worker.c` — `rejoin_worker`/`apply_join_state`: after clearing `XCE_WORKER_UNAVAIL`, check for non-terminal jobs assigned to this host (Requirement 1) and, if any, set `XCE_WORKER_RECONCILING` and initialize the per-worker reconciliation record before releasing the write lock. No change to admin-bit preservation or attribute/endpoint update behavior. `on_worker_join` (worker.c:623) — extended, not `handle_worker_join`: it already calls `worker_push_config` as its one existing step (the config the worker needs — hosttype/hostmodel/resource variables — is foundational baseline state, and its own stub comment, `/* consider to push jobs on hosts to */`, already anticipates this exact follow-on). The reconciliation query send is added right after it, in that fixed order (config push, then query) — both sent fire-and-forget, no new ack/reply invented for this. This ordering is safe, not just conventional: the worker's incoming-request dispatch (`myreq_handler`, `controller_connector.c:1563`) is a plain synchronous `switch` invoked one message at a time off its single long-lived client connection (`xcecli_ctx`) — confirmed by reading it, not assumed — so the two requests are not just sent in order, they are *handled* to completion in that same order on the worker side; no thread-pool fan-out is in the path (the worker's own `thread_pool_t` is for job-forking work, unrelated to request dispatch). The actual wait in this design belongs at the *end* of the sequence, not in the middle: the controller does not consider the worker's rejoin/reconciliation complete until the job-state *report* arrives (or the Requirement 5.3 timeout fires) — see Error Handling.
|
||||
- `src/controller/join.c` — **no change**. `handle_worker_join` itself is untouched; the reconciliation query is sent from the (extended) `on_worker_join`, which `handle_worker_join` already calls unconditionally after the join reply.
|
||||
- `src/controller/job.c` — two new functions, both following `dispatch_a_job_`'s eager-mutation-plus-explicit-notify pattern (not `record_job_finish_fact`'s deferred one — see Components for why): `record_job_reconcile_pending_fact()` (Requirement 4.6, eagerly calls `apply_job_update_transition(job, JOB_STAT_PEND)` under the write lock, releases `job->job.allocs` — `free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;` — on an applied (non-no-op) transition, durably appends via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)`, then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking; this is bug-1631's own documented candidate fix, applied here because Requirement 4.6 is exactly the "RUN job re-admitted to the PEND pool" precondition bug-1631's fix was deferred pending), `record_job_reconcile_assign_fact()` (Requirement 4.8, eagerly sets `job->job.state`/host assignment under the write lock, calls the existing `persist_submit_job_dispatch_fact(c->persist, jobid, state, ts, alloc=NULL)` — reusing bug-1678's already-tested NULL-alloc decode path on both apply seams verbatim, not a new fact shape — then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking, same as above and for the same reason). Existing `record_job_finish_fact`/`record_job_fail_fact` are called as-is for the finished-success/finished-failure cases. Also: split `dispatch_a_job_`'s `w->ep == NULL || w->host.status != 0` check (job.c:1511-1512) into two branches, checked in order — `w->ep == NULL` is resolved directly via `record_job_reconcile_pending_fact()` with **no retry-counter increment and no `mark_job_fail()`**, *before* the code ever reaches `xce_classify_dispatch_fail()` (a deliberate bypass, not a third value on the shared `xce_dispatch_fail_class_t` enum — see Components); `w->host.status != 0` keeps its existing `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY` behavior unchanged (now rarely hit for `XCE_WORKER_RECONCILING` specifically, since the scheduler-side filter below keeps it from being selected in the first place).
|
||||
- `src/controller/job.c` — **Correction (2026-09-01, bug-1747, task 3.2): three new functions, not two — see below.** `record_job_reconcile_pending_fact()` and `record_job_reconcile_assign_fact()` both follow `dispatch_a_job_`'s eager-mutation-plus-explicit-notify pattern (not `record_job_finish_fact`'s deferred one — see Components for why): `record_job_reconcile_pending_fact()` (Requirement 4.6, eagerly calls `apply_job_update_transition(job, JOB_STAT_PEND)` under the write lock, releases `job->job.allocs` — `free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;` — on an applied (non-no-op) transition, durably appends via the existing generic `persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)`, then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking; this is bug-1631's own documented candidate fix, applied here because Requirement 4.6 is exactly the "RUN job re-admitted to the PEND pool" precondition bug-1631's fix was deferred pending), `record_job_reconcile_assign_fact()` (Requirement 4.8, eagerly sets `job->job.state`/host assignment under the write lock, calls the existing `persist_submit_job_dispatch_fact(c->persist, jobid, state, ts, alloc=NULL)` — reusing bug-1678's already-tested NULL-alloc decode path on both apply seams verbatim, not a new fact shape — then calls `persist_notify_job_updated_already_persisted(c, job)` after unlocking, same as above and for the same reason). Existing `record_job_finish_fact`/`record_job_fail_fact` were originally scoped to be called as-is for the finished-success/finished-failure cases; **found live during task 3.2 implementation that `record_job_finish_fact`'s specific call site has no way to carry the worker-reported finish time (Requirement 4.2) — its own primitive, `persist_submit_job_finish_fact`, accepts one, but `record_job_finish_fact` always derives `time(NULL)` instead.** Fixed with a THIRD new function, `record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms)` — `record_job_finish_fact`'s own worker-reported-finish-time sibling, following `record_job_finish_fact`'s own DEFERRED pattern (durable append only, not eager like the other two new functions above), differing only in passing the caller-supplied (ms→s converted) time to `persist_submit_job_finish_fact` instead of `time(NULL)`; `record_job_finish_fact` itself stays unmodified, still used as-is for `on_job_finish`'s ordinary (non-reconciliation) receipt path. `record_job_fail_fact` is still called as-is for the finished-failure case — Requirement 4.2 remains genuinely unsatisfiable there via any existing path (see the Persistence Field-Completeness Review section's correction note; tracked as the FAIL half of bug-1747, deliberately deferred). Also: split `dispatch_a_job_`'s `w->ep == NULL || w->host.status != 0` check (job.c:1511-1512) into two branches, checked in order — `w->ep == NULL` is resolved directly via `record_job_reconcile_pending_fact()` with **no retry-counter increment and no `mark_job_fail()`**, *before* the code ever reaches `xce_classify_dispatch_fail()` (a deliberate bypass, not a third value on the shared `xce_dispatch_fail_class_t` enum — see Components); `w->host.status != 0` keeps its existing `XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY` behavior unchanged (now rarely hit for `XCE_WORKER_RECONCILING` specifically, since the scheduler-side filter below keeps it from being selected in the first place).
|
||||
- `src/controller/cluster.c` — new `case OPCode_jobStateReport:` in `myreq_handler`'s dispatch switch (runs after `ha_dispatch_gate`); route to the new reconciliation-apply logic.
|
||||
- `src/scheduler/src/scheduler.c` — extend the existing candidate-host filter (`scheduler.c:1211`, today `if (IS_WORKER_UNAVAIL(w->status)) { ... }`) to also skip a host with the new reconciling bit set — same shape, one added disjunct, no new call site.
|
||||
- `src/include/xce/types.h` — add `XCE_WORKER_RECONCILING` to the worker status bitmask plus an `IS_WORKER_RECONCILING(status)` macro (mirroring `IS_WORKER_UNAVAIL`'s existing shape) for the scheduler-side filter above; add a `resync_state_t` (queried job-ID set, round token, deadline) referenced from `worker_t`.
|
||||
|
|
@ -416,12 +416,14 @@ Per `design-principles.md`'s Persistence Review requirement and `persistence.md`
|
|||
|
||||
| Outcome | Fact path | New fields? | Both apply seams? |
|
||||
|---|---|---|---|
|
||||
| Finished success | `record_job_finish_fact` (existing, reused as-is) | None — existing complete fact shape (already carries `finish_time`, exit info); the reconciliation caller passes the worker-reported `finishTimeMs` (Requirement 3.5) as that fact's finish time, never the time the controller processed the report (Requirement 4.2) | Already covered by existing `apply_committed_entry`/standby-catch-up handling — unchanged |
|
||||
| Finished success | `record_job_finish_fact_at` (NEW, task 3.2/bug-1747) | None — reuses `persist_submit_job_finish_fact`'s existing, already-complete `WAL_JOB_FINISH` payload shape verbatim (`finish_time`, exit info); the reconciliation caller passes the worker-reported `finishTimeMs` (Requirement 3.5, converted ms→s) as that fact's finish time, never the time the controller processed the report (Requirement 4.2) | Already covered by existing `apply_committed_entry`/standby-catch-up handling — unchanged, since the wire payload shape is identical to `record_job_finish_fact`'s own |
|
||||
| Finished failure | `record_job_fail_fact` (existing, reused as-is) | None — same Requirement 4.2 worker-reported-time rule applies | Unchanged, existing |
|
||||
| Still running | No fact — pure no-op against the outstanding set | N/A | N/A |
|
||||
| Unrecognized → pending | `persist_submit_job_fact(..., WAL_JOB_UPDATE, jobid, JOB_STAT_PEND)` (existing generic primitive, new call site) | None — the plain `{jobid, state}` shape is sufficient, same as every other `WAL_JOB_UPDATE` caller. **`job->job.allocs` release is in-memory only, eager, not itself a persisted field** — mutated live under the same lock as the state change (mirroring `dispatch_a_job_`'s own success path in reverse; see Components), then notified via `persist_notify_job_updated_already_persisted`, not left for `apply_committed_entry` to discover | Both seams already decode plain `WAL_JOB_UPDATE` identically today (it is not a new payload shape) — a replay of this fact re-derives `job->job.state = JOB_STAT_PEND` correctly; it does **not** re-derive the allocs release (that mutation is not wire content), but since `dispatch_a_job_`'s eager pattern already established this exact asymmetry (live mutation + explicit notify now, replay-idempotent state field only, no replay-time side effect needed for a purely in-memory cleanup), this is consistent with existing precedent, not a new gap |
|
||||
| Unqueried job → assign | `persist_submit_job_dispatch_fact(..., alloc=NULL)` (existing primitive, new call site, `record_job_reconcile_assign_fact`) | None — reuses `persist_submit_job_dispatch_fact`'s existing `WAL_JOB_DISPATCH` payload shape verbatim with `alloc=NULL`, exercising the same NULL-alloc decode branch bug-1678 already added. Live mutation is eager (see Components), matching `dispatch_a_job_`'s own pattern for the same fact type | Decode side already covered by bug-1678 on both seams. **Encode side required a fix** (bug-1746, task 2.2): `capnp_encode_joballoc` unconditionally dereferenced a NULL `alloc` — this was this codebase's first genuinely-NULL caller. Fixed with a minimal zero-length-buffer guard, matching the decode side's `alloc_len > 0` contract exactly; no new decode path needed once the encode side actually produces the zero-length payload the decode side already expected |
|
||||
|
||||
**Correction (2026-09-01, found live during task 3.2 implementation, bug-1747) — a THIRD new function, not two:** this section's original "Finished success" row asserted `record_job_finish_fact` could be reused "as-is," citing an "existing complete fact shape (already carries `finish_time`)" — accurate about `persist_submit_job_finish_fact` (the underlying primitive, which does accept an explicit `finish_time` parameter) but not about `record_job_finish_fact`'s own specific call site, which never exposed that parameter to a caller and always derives `time(NULL)` at receipt time instead. Calling `record_job_finish_fact` unmodified therefore could NOT satisfy Requirement 4.2 for the finished-success case — confirmed live when task 3.2 wired the reconciliation caller and found no way to pass the worker-reported time through. Fixed (coordinator-directed, mirroring bug-1746's own precedent for an in-boundary companion fix) by adding `record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms)` (job.c) — `record_job_finish_fact`'s own worker-reported-finish-time sibling: an identical guard/hook-fire/durable-append body, differing only in passing the caller-supplied (ms→s converted) time to `persist_submit_job_finish_fact` instead of `time(NULL)`. `record_job_finish_fact` itself is left UNMODIFIED — its own caller (`on_job_finish`, an ordinary near-instantaneous worker report) is correctly served by receipt-time there. Unlike `record_job_reconcile_pending_fact`/`record_job_reconcile_assign_fact` (both EAGER, `dispatch_a_job_`-style), `record_job_finish_fact_at` follows `record_job_finish_fact`'s own DEFERRED pattern (durable append only; the in-memory mutation and scheduler notify happen later via `apply_committed_entry`'s replay, exactly as an ordinary finish already works) — so this section's job.c fact-recording count is **three new functions**, not two (see also the Architecture Pattern & Boundary Map bullet and the Modified Files entry for `job.c`, both correspondingly updated). The sibling "Finished failure" outcome remains genuinely unfixable via any existing path: `WAL_JOB_FAIL`'s wire payload carries no finish-time slot at any layer, for any caller in this codebase, and `apply_job_fail_transition` unconditionally stamps `time(NULL)` at apply time — a pre-existing, systemic gap that predates this feature entirely (bug-1747, re-scoped 2026-09-01), deliberately left deferred rather than patched narrowly inside this task.
|
||||
|
||||
**Documented scope decision — `job->allocs` on a reconciliation-assigned job (Requirement 4.8):** when the controller reconciles a job it never held any prior record of (e.g. it was dispatched by a controller instance before a failover whose dispatch fact was itself lost), it has no way to learn what resource allocation the scheduler originally computed for that job — the worker's report carries the job's execution outcome, not its original resource request. This design deliberately does not attempt to reconstruct `job->allocs` for this case. **Correction (2026-09-01, re-validated against current code):** an earlier draft of this section cited `persistence.md`'s `APPLY_DIVERGENT` section / bug-1678 as evidence this is "a tracked, separate gap" `WAL_JOB_DISPATCH` still has — that citation was stale. Bug-1678 was fixed 2026-08-25 (before this design was finalized): `WAL_JOB_DISPATCH` now durably carries `job->allocs` when one exists, on both apply/decode seams. The real reason this design still writes no allocation for Requirement 4.8 is simpler and unrelated to that gap: there just isn't one to write — the scheduler, not this controller, is the only party that ever computes a job's allocation, and for a job this controller never had any record of, no dispatch (successful or otherwise) ever ran through this controller to produce one. `record_job_reconcile_assign_fact` reuses the *fixed* `persist_submit_job_dispatch_fact` primitive because its NULL-alloc *decode* case is already the correct representation of "state and host known, allocation not" — though the *encode* side of that same NULL case had never actually been exercised before this feature (bug-1746: `capnp_encode_joballoc` dereferenced a NULL `alloc` unconditionally, since `dispatch_a_job_`, bug-1678's only prior caller, always passes a real allocation). Task 2.2 fixed the encode side with a minimal zero-length-buffer guard as part of making this call safe to actually make — a real, if narrow, gap in the "already-tested" characterization this design originally gave the primitive, closed as part of implementation rather than left latent. The practical consequence is bounded by `resource-accounting.md`'s existing policy: a job with an unknown/zero reservation is charged `max(reserved=0, actual)` — i.e., its real usage — the moment its next rusage tick arrives, identical to how a legitimately-unreserved job is already handled today. The job's *state* is correct immediately; its resource *accounting* converges within one rusage cycle, not instantly. No fabricated allocation value is ever written.
|
||||
|
||||
### Configuration Model
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@
|
|||
- _Boundary: job.c additions_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 3. Core: controller reconciliation-round bookkeeping
|
||||
- [x] 3. Core: controller reconciliation-round bookkeeping
|
||||
- [x] 3.1 (P) Track one reconciliation round per worker with a bounded timeout
|
||||
- Given a worker host with one or more non-terminal jobs currently assigned to it, construct the exact list of those jobs' unique IDs as the round's query set
|
||||
- Track a round identifier that distinguishes the current reconciliation attempt from any prior one for the same worker
|
||||
|
|
@ -70,7 +70,7 @@
|
|||
- _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 5.3, 6.1_
|
||||
- _Boundary: resync.c_
|
||||
|
||||
- [ ] 3.2 Apply each reported job outcome through the correct idempotent path
|
||||
- [x] 3.2 Apply each reported job outcome through the correct idempotent path
|
||||
- For each entry in a worker's job-state report: a still-running job is cleared from the round's outstanding set with no fact recorded; a finished-success or finished-failure job is applied through the existing finish/fail recording used for an ordinary report; an unrecognized queried job is applied through task 2.1's pending-and-release path; a reported job outside the original query is applied through task 2.2's assignment path
|
||||
- Discard a report that arrives for a round already superseded by a newer one for the same worker, rather than applying it
|
||||
- When the round's outstanding set becomes empty, resolve the round so the worker becomes eligible for dispatch again
|
||||
|
|
@ -157,3 +157,4 @@
|
|||
- **Task 2.2 found and fixed bug-1746**, a real pre-existing defect in `persist.c` (part of bug-1678's original fix, predates this feature): `capnp_encode_joballoc()` unconditionally dereferenced a NULL `alloc` via `encode_JobAlloc()` instead of the already-existing NULL-safe `encode_JobAlloc_ptr()` wrapper — never triggered before since `dispatch_a_job_` (the only prior caller) always passes non-NULL. This directly contradicted design.md's claim that the NULL-alloc case was "tested, backward-compatible" on both apply seams — true for *decode*, never actually true for *encode* until this task exercised it. Fixed with a minimal guard producing a true zero-length buffer (not a capnp-null-pointer encoding — verified against `apply_committed_entry`'s `alloc_len > 0` decode gate, which a non-zero-length "null" encoding would have wrongly satisfied). design.md corrected in 4 places. **Any future task calling `persist_submit_job_dispatch_fact` with a NULL alloc can now trust this path — it wasn't safe before task 2.2, it is now.**
|
||||
- Task 2.2 (self-correction): a coordinator-applied one-line comment fix to `job.c` accidentally closed the enclosing `/* */` comment block early, leaving the comment's remaining lines as bare (uncompilable) code — caught immediately by re-running the build before marking the task complete, not left for review to find. Reinforces why fresh-evidence re-verification applies even to "trivial, obviously-safe" doc-only edits made outside the TDD subagent loop.
|
||||
- Task 3.1: `resync.c`'s `resync_timeout_ms()` applies its own private default-if-unset fallback (15000ms) for `workerResyncTimeoutMs` rather than following `myconf.c`'s established `SET_DEFAULT_IF_UNSET`-style canonicalization convention used for the sibling `heartbeat_interval` field (`myconf.c:74`) — `myconf.c` is outside task 3.1's boundary, so this was correctly left as-is, but if a SECOND consumer of `workerResyncTimeoutMs` (or `workerResyncWindowMs`) is ever added, it would have to either duplicate this private fallback or the two could silently disagree. Worth canonicalizing in `myconf.c` if/when that happens — not needed for this feature's own single consumer. delaymgr timer cancellation: `delay.c`/`delay.h` expose no dequeue/cancel primitive for an already-armed one-shot task; `resync.c` handles this correctly via a by-value round-token check inside the fired callback (verified safe against both the logical "resolves the wrong round" bug and use-after-free, since the callback's closure never holds a pointer into the freed `resync_state_t`) — any future delaymgr one-shot user in this codebase facing the same "need to cancel" problem should use this same by-value-token-check pattern rather than assuming a cancel API exists.
|
||||
- **Task 3.2 needed 3 review rounds — a genuine cautionary tale about a boolean short-circuit hiding a second, adjacent bug.** Round 1 caught the unqueried-extra-"running" branch gating on terminal-vs-non-terminal instead of Req 4.8's real trigger (assignment status), silently leaving a job stuck PEND when its `WAL_JOB_DISPATCH` fact never reached durable commit before a failover — the feature's own core motivating scenario. The FIRST fix (`!terminal && assigned_to_w's_host`) solved that but introduced a NEW, distinct Req 4.4 violation via the same `&&`'s short-circuit: a TERMINAL job (DONE/EXIT) falls into the same `else`/assign branch and gets clobbered back to RUN — caught by round 2, and confirmed the implementer's own new test had been asserting the bug as correct behavior (a green suite is not proof of correctness when the test itself encodes the wrong expectation). The SECOND fix (a genuine three-way `if`/`else if`/`else`: terminal→noop, non-terminal-and-already-assigned-anywhere→noop, non-terminal-and-unassigned→assign) passed round 3's from-scratch re-derivation plus an explicit hunt for job==NULL reachability. **Lesson for future tasks in this codebase**: when a guard combines two orthogonal conditions with `&&`/`||` and only ONE of the two is corrected after a review finding, re-derive the FULL truth table before re-submitting — a targeted patch to the specific counterexample a reviewer names can leave an adjacent, differently-shaped counterexample in the same boolean untouched. bug-1747 (finish-time gap, split fixed) and bug-1748 (this guard defect) both logged.
|
||||
|
|
|
|||
|
|
@ -20371,6 +20371,68 @@
|
|||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-09-01T01:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1747",
|
||||
"timestamp": "2026-09-01T00:00:00Z",
|
||||
"error_message": "Pre-existing, systemic gap predating worker-rejoin-resync entirely, discovered while implementing task 3.2 (resync_apply_report, resync.c): WAL_JOB_FAIL carries no fail-time/finish-time slot in its wire payload at ANY layer, and apply_job_fail_transition() (job.c) unconditionally stamps job.finish_time = time(NULL) at APPLY time (not even receipt time) for EVERY fail-reporting caller in this codebase -- not just reconciliation. This means Requirement 4.2 ('the Controller shall record the finish time the Worker reported ... rather than the time the Controller processed the reconciliation') is currently unsatisfiable for a reconciled finished-FAILURE outcome via any existing path. UPDATE (2026-09-01, coordinator-directed re-scope): the sibling FINISH-success half of this gap is FIXED -- see 'fix' field below; only the FAIL half remains open.",
|
||||
"file": "src/controller/job.c",
|
||||
"root_cause": "job_failinfo_t (xce/types.h) is {jobid, stage, code, detail} -- no finish_time field, and WAL_JOB_FAIL's durable payload (record_job_fail_fact(), job.c: persist_submit_job_fact_ex(&c->persist, WAL_JOB_FAIL, jobid, 0, WAL_JOB_FAIL_STAGE_FLAG|stage)) carries no finish-time slot at all -- confirmed by direct inspection of the payload construction. apply_job_fail_transition() (job.c, the guarded core BOTH the live job_ctrl path (mark_job_fail) and the deferred apply seam (apply_committed_entry's WAL_JOB_FAIL case) invoke) unconditionally executes `job->job.finish_time = time(NULL);` with no external value ever consulted, at APPLY time -- meaning even a caller willing to plumb a real value through record_job_fail_fact's WAL payload would still need a apply_job_fail_transition() signature change to have it actually land in job.job.finish_time. This is a pre-existing, already-documented residual: job.c's own comment near apply_job_postdone_transition (task 4.1) already states 'finish_time is not part of the wal_jobstate_t wire format ... has no apply-time equivalent' for a sibling fact type, confirming this is systemic to the wal_jobstate_t-based fact family, not something worker-rejoin-resync introduced. Fixing it for real would require extending WAL_JOB_FAIL's wire payload (wal_types.h, mirroring bug-1689's own WAL_JOB_FINISH trailing-fields precedent) PLUS an apply_job_fail_transition() signature/behavior change reaching every fail-reporting call site in the system (job_ctrl's mark_job_fail, on_job_fail's ordinary worker-fail receipt path, and this feature's own reconciliation caller) -- real, broad-blast-radius work, not a narrow patch.",
|
||||
"fix": "SPLIT (2026-09-01, coordinator-directed): the FINISH-success half of this gap is FIXED as part of worker-rejoin-resync task 3.2 -- added record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms) (job.c), record_job_finish_fact()'s own worker-reported-finish-time sibling (identical guard/hook-fire/durable-append body, differing only in passing the caller-supplied finish_time -- converted ms->s -- to persist_submit_job_finish_fact() instead of time(NULL)); record_job_finish_fact() itself is UNMODIFIED, its own caller (on_job_finish, an ordinary near-instantaneous worker report) correctly keeps using receipt-time. resync_apply_report()'s finishedOk branch now calls record_job_finish_fact_at(c, job, &finish, entry->finish_time_ms). Verified via a real restart-replay test (test_apply_report_finished_success_uses_worker_reported_finish_time, test_worker_resync.c): a finished-success entry with finish_time_ms set to 'now minus 1 day' recovers job.finish_time equal to that worker-reported value after persist_recover(), not reconciliation-processing wall-clock time. This was possible narrowly for FINISH because persist_submit_job_finish_fact() already accepted an explicit finish_time parameter at the primitive level -- record_job_finish_fact()'s specific call site just never exposed it. The FAIL half (see root_cause) is NOT YET FIXED and is NOT task 3.2's (or any single narrow task's) job to fix -- it needs the broader WAL_JOB_FAIL payload extension + apply_job_fail_transition() signature change described above, deliberately deferred pending a dedicated task, matching the same 'deliberately deferred, documented, not blocking' treatment bug-1631's own sibling gap received earlier in this same feature. The reconciled job's STATE (EXIT) is always correct for a finished-failure outcome either way -- only its recorded finish_time reflects apply-processing time rather than the worker's true report.",
|
||||
"tags": [
|
||||
"worker-rejoin-resync",
|
||||
"task-3.2",
|
||||
"resync-c",
|
||||
"job-c",
|
||||
"requirement-4.2",
|
||||
"finish-time",
|
||||
"wal-job-fail",
|
||||
"wal-job-finish",
|
||||
"apply_job_fail_transition",
|
||||
"pre-existing",
|
||||
"predates-feature",
|
||||
"systemic",
|
||||
"partially-fixed",
|
||||
"resync_apply_report",
|
||||
"record_job_finish_fact_at"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1689",
|
||||
"bug-1678",
|
||||
"bug-1746",
|
||||
"bug-1631"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-01T01:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1748",
|
||||
"timestamp": "2026-09-01T02:00:00Z",
|
||||
"error_message": "worker-rejoin-resync task 3.2 (resync_apply_running_entry, resync.c), round-2 independent review: the round-1 fix for the PEND/unassigned-job regression (bug-1747-adjacent, not itself in bug-1747) introduced a NEW Requirement 4.4 violation. The round-1 guard `if (!resync_job_is_terminal(job->job.state) && resync_job_assigned_to_host(job, w->host.name)) { no-op } else { record_job_reconcile_assign_fact(...) }` short-circuits the `&&` to false whenever job->job.state IS terminal, falling into the `else` branch and calling record_job_reconcile_assign_fact() on an ALREADY-TERMINAL (DONE/EXIT) job -- which has no backward-transition guard of its own and unconditionally sets job->job.state = JOB_STAT_RUN, clobbering a terminal job back to RUN. Confirmed live and, worse, CERTIFIED AS CORRECT by the round-1 implementation's own test: test_apply_report_all_five_outcomes_updates_job_table seeded j_extra at JOB_STAT_DONE with a comment claiming this 'satisfies Req 4.8's trigger,' then asserted j_extra->job.state == JOB_STAT_RUN afterward and the suite reported PASS -- the test baked the bug in as an expected outcome instead of catching it.",
|
||||
"file": "src/controller/resync.c",
|
||||
"root_cause": "Requirement 4.8's actual trigger condition is 'no existing non-terminal record of that job assigned elsewhere' (requirements.md AC 4.8; record_job_reconcile_assign_fact's own doc comment and design.md's Service Interface Preconditions: 'job is non-terminal ... or has no existing non-terminal assignment'). Terminal-vs-non-terminal is NOT the axis that decides assign-eligibility on its own -- Requirement 4.4 ('shall not overwrite that state with a less final or older report') independently and unconditionally forbids ever assign-routing a terminal job, regardless of host. The round-1 guard conflated two orthogonal conditions (terminal-ness, and host-assignment) into a single boolean expression via `&&`, which is only correct when BOTH must jointly hold for the no-op case -- but Req 4.4's terminal exclusion needs to apply UNCONDITIONALLY, independent of the host check, not only when the host check also happens to be true. A two-way boolean cannot express three independent outcomes (terminal-always-noop; non-terminal-but-already-assigned-noop; non-terminal-and-unassigned-assign) without this exact class of short-circuit bug.",
|
||||
"fix": "FIXED (2026-09-01, round-2 review, task 3.2). resync_apply_running_entry() rewritten as a genuine three-way `if`/`else if`/`else` split, not a boolean guard: (a) resync_job_is_terminal(job->job.state) -> ALWAYS a no-op, checked FIRST and independently, regardless of host (closes the Req 4.4 violation). (b) non-terminal AND job->job.allocs != NULL (already has SOME live non-terminal host assignment -- to the reporting worker's own host, or to a different one, it does not matter which) -> a no-op; also tightened from round 1's own host-specific check (resync_job_assigned_to_host(job, w->host.name)) to a host-agnostic allocs!=NULL check, since round-1's version would have ALSO wrongly routed a job assigned to a DIFFERENT host into the assign branch, resetting its start_time and re-appending a spurious WAL_JOB_DISPATCH fact for a job that was never actually lost. (c) non-terminal AND allocs == NULL (genuinely unassigned) -> record_job_reconcile_assign_fact(), Req 4.8's real trigger. Verified: (1) test_apply_report_all_five_outcomes_updates_job_table's j_extra is now seeded TERMINAL and asserted COMPLETELY UNTOUCHED (state stays DONE, no WAL append) -- the exact inversion of the round-1 assertion that had certified the bug. (2) test_apply_report_unqueried_extra_already_tracked_is_noop (branch (b), assigned to a DIFFERENT host) strengthened with a start_time-unchanged assertion (record_job_reconcile_assign_fact unconditionally stamps time(NULL), bug-1697 -- invisible to a bare state/allocs check since the job was already RUN either way) and an LSN-unchanged assertion (no spurious WAL_JOB_DISPATCH append) -- round 1's version of this test could not have caught either defect. (3) test_apply_report_unqueried_extra_unassigned_pend_job_gets_assigned (branch (c), genuinely unassigned/PEND) already covers the real assign-success case with full restart-replay rigor, added in the same round-1 fix pass. Full controller test suite (meson test -C src/xce.build 'controller test'): all green.",
|
||||
"tags": [
|
||||
"worker-rejoin-resync",
|
||||
"task-3.2",
|
||||
"resync-c",
|
||||
"requirement-4.4",
|
||||
"requirement-4.8",
|
||||
"resync_apply_running_entry",
|
||||
"record_job_reconcile_assign_fact",
|
||||
"guard-logic",
|
||||
"short-circuit",
|
||||
"review-caught",
|
||||
"test-baked-in-bug",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1747",
|
||||
"bug-1697",
|
||||
"bug-1631"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-01T02:00:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -241,6 +241,28 @@ extern int record_job_reconcile_pending_fact(cluster_t*, job_t*);
|
|||
extern int record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
|
||||
xce_job_state_entry_t*);
|
||||
|
||||
/* worker-rejoin-resync task 3.2 (bug-1747 fix, Req 4.2): record_job_finish_
|
||||
* fact()'s worker-reported-finish-time sibling, for the reconciliation
|
||||
* finished-success path only. Same body/guard/hook-fire/deferred-append
|
||||
* pattern as record_job_finish_fact() (job.c, unmodified) -- differs only
|
||||
* in passing `finish_time_ms` (converted ms -> s) to persist_submit_job_
|
||||
* finish_fact() instead of time(NULL). See job.c's own doc comment on this
|
||||
* function for the full rationale, including the persist_submit_job_
|
||||
* finish_fact() unit confirmation.
|
||||
*
|
||||
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
|
||||
* holding it until this call returns -- this function does not lock/unlock
|
||||
* `job` itself, matching record_job_finish_fact's own convention.
|
||||
*
|
||||
* DEFERRED pattern (unlike record_job_reconcile_pending_fact/assign_fact
|
||||
* above): this function only durably appends the fact; it does NOT mutate
|
||||
* job's in-memory state and the caller must NOT call persist_notify_job_
|
||||
* updated_already_persisted() afterward -- exactly like record_job_finish_
|
||||
* fact(), the mutation and notify happen later, via apply_committed_entry's
|
||||
* replay of the committed WAL_JOB_FINISH entry. */
|
||||
extern int record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*,
|
||||
uint64_t);
|
||||
|
||||
extern int on_job_start(cluster_t*, job_pidinfo_t*);
|
||||
extern int on_job_finish(cluster_t*, xce_jobfinish_t*);
|
||||
extern int on_job_fail(cluster_t*, job_failinfo_t*);
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@
|
|||
|
||||
#include <stdint.h>
|
||||
|
||||
#include "typemap.h" /* xce_job_state_entry_t/xce_job_state_report_t --
|
||||
* worker-rejoin-resync task 3.2 */
|
||||
#include "xce/cluster_def.h"
|
||||
#include "xce/types.h"
|
||||
|
||||
|
|
@ -75,6 +77,103 @@ extern void resync_resolve_round(cluster_t* c, worker_t* w);
|
|||
* Precondition: caller holds at least `w`'s read lock. */
|
||||
extern int resync_round_is_current(worker_t* w, uint64_t token);
|
||||
|
||||
/* worker-rejoin-resync task 3.2 (Req 4.1-4.5, 6.1-6.3): applies a worker's
|
||||
* already-decoded job-state report to `c`'s job table -- the glue that maps
|
||||
* each `xce_job_state_entry_t` outcome in `report` to the correct existing
|
||||
* (record_job_fail_fact via on_job_fail, job.c) or new (record_job_finish_
|
||||
* fact_at, task 3.2/bug-1747; record_job_reconcile_pending_fact/record_job_
|
||||
* reconcile_assign_fact, task 2.1/2.2, job.c) idempotent fact-recording
|
||||
* call, per design.md's "System Flows" sequence diagram (`loop each
|
||||
* reported entry`).
|
||||
*
|
||||
* Routing per `xce_job_state_entry_t.status`:
|
||||
* - JobStateStatus_running, entry->queried true: cleared from the round's
|
||||
* outstanding set; no fact recorded.
|
||||
* - JobStateStatus_running, entry->queried false (Req 3.6 extra): a
|
||||
* genuine THREE-WAY split (review round 2 fix -- a two-way boolean
|
||||
* guard is WRONG and introduces a Req 4.4 violation, see resync.c's
|
||||
* own doc comment on resync_apply_running_entry for the full incident):
|
||||
* (a) TERMINAL -> ALWAYS a no-op, regardless of host (Req 4.4 -- a
|
||||
* terminal job is never assign-eligible; record_job_reconcile_assign_
|
||||
* fact() has no backward-transition guard of its own and would
|
||||
* unconditionally clobber DONE/EXIT back to RUN if called here).
|
||||
* (b) non-terminal AND already has SOME live non-terminal host
|
||||
* assignment (`job->job.allocs != NULL` -- to THIS reporting worker's
|
||||
* own host, or to a DIFFERENT one, it does not matter which) -> a
|
||||
* no-op (already validly assigned; calling assign here would still
|
||||
* wrongly reset start_time and re-append a WAL_JOB_DISPATCH fact for a
|
||||
* job that was never actually lost).
|
||||
* (c) non-terminal AND genuinely UNASSIGNED (`allocs == NULL`) -> Req
|
||||
* 4.8's real trigger ("no existing non-terminal record... assigned
|
||||
* elsewhere" -- there IS a non-terminal record, but it carries no
|
||||
* assignment at all, e.g. still PEND because its WAL_JOB_DISPATCH fact
|
||||
* never reached durable majority commit before a controller failover,
|
||||
* even though the worker really is running it) -> reconciles via
|
||||
* record_job_reconcile_assign_fact() (Req 4.8), confirmed against that
|
||||
* function's own doc comment/assert (job.c), which documents itself as
|
||||
* the running-only case. A jobid with NO record at all in the job
|
||||
* table cannot be reconciled this way (that function mutates an
|
||||
* existing job_t, it does not construct one) and is logged and skipped
|
||||
* -- out of this feature's scope; every job a worker can legitimately
|
||||
* run was submitted through this controller at some point, so this is
|
||||
* expected to be unreachable in practice.
|
||||
* - JobStateStatus_finishedOk: routed through record_job_finish_fact_at()
|
||||
* (task 3.2/bug-1747 fix, job.c) -- record_job_finish_fact()'s own
|
||||
* worker-reported-finish-time sibling, so the recorded finish_time is
|
||||
* entry->finish_time_ms (Req 3.5), not reconciliation-processing time
|
||||
* (Req 4.2). record_job_finish_fact() itself is unmodified; its own
|
||||
* caller (on_job_finish, an ordinary worker report) still uses
|
||||
* receipt-time, correctly, for that near-instantaneous case.
|
||||
* - JobStateStatus_finishedFail: routed through on_job_fail() (job.c),
|
||||
* unchanged -- Requirement 4.2 remains unsatisfiable for this outcome
|
||||
* via any existing path in this codebase (WAL_JOB_FAIL carries no
|
||||
* finish-time slot at any layer, for any caller -- a pre-existing,
|
||||
* systemic gap that predates this feature; see buglog.json bug-1747).
|
||||
* - Both finished outcomes apply identically whether entry->queried is
|
||||
* true or false (Req 4.7: an unqueried finished extra reconciles
|
||||
* through the same path as a queried one; there is no "assign" case for
|
||||
* a finished outcome). Cleared from the outstanding set when
|
||||
* entry->queried is true.
|
||||
* - JobStateStatus_unrecognized: routed through record_job_reconcile_
|
||||
* pending_fact() (Req 4.6, task 2.1); cleared from the outstanding set.
|
||||
*
|
||||
* Req 6.1/6.3: if `round_token` is not `w`'s CURRENTLY outstanding round
|
||||
* (resync_round_is_current), the entire report is discarded before
|
||||
* anything else is touched -- no fact is applied, no outstanding-set
|
||||
* mutation occurs. This also makes a duplicate re-application of an
|
||||
* already-fully-resolved report (Req 6.2) a pure no-op by the same
|
||||
* mechanism: once every queried job has a reconciled outcome,
|
||||
* resync_resolve_round() clears w->resync, so `round_token` is no longer
|
||||
* current on a later identical call -- no facts are re-applied, no
|
||||
* additional WAL entries appended (a duplicate applied WHILE the round is
|
||||
* still open instead re-applies each fact through job.c's own idempotent
|
||||
* guards, which are themselves safe no-ops on a repeat).
|
||||
*
|
||||
* Req 5.1/5.2: once the round's outstanding set becomes empty (every
|
||||
* queried jobid has been cleared), the round is resolved
|
||||
* (resync_resolve_round()), clearing XCE_WORKER_RECONCILING so `w` becomes
|
||||
* eligible for dispatch again.
|
||||
*
|
||||
* Locking: unlike resync_arm_round/resync_resolve_round/resync_round_is_
|
||||
* current above, this function manages its OWN `w` write-locking
|
||||
* internally and narrowly (never nested with a job's own write lock) --
|
||||
* mirroring dispatch_a_job_'s established discipline (job.c: every branch
|
||||
* releases `w`'s lock BEFORE acquiring a job's) rather than this file's
|
||||
* other three functions' "caller holds w's lock for the whole call"
|
||||
* convention, which fits their simple, single-object, single-call-site
|
||||
* mutations but not this function's fan-out into per-job job.c calls, each
|
||||
* requiring its own job-level lock. The caller must NOT pre-hold `w`'s
|
||||
* lock when calling this function.
|
||||
*
|
||||
* `report` is treated as a single, already-fully-assembled list of
|
||||
* `report->count` entries (`report->entries[0..count)`) -- multi-page wire
|
||||
* reassembly is a later task's concern (6.2), out of this function's scope.
|
||||
*
|
||||
* Returns 0 on success (whether the report was applied or discarded as
|
||||
* stale), or a negative errno for invalid arguments (NULL c/w/report). */
|
||||
extern int resync_apply_report(cluster_t* c, worker_t* w, uint64_t round_token,
|
||||
xce_job_state_report_t* report);
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/* worker-rejoin-resync task 3.1 test seam: directly invokes the SAME
|
||||
* internal timeout-fire logic resync_arm_round's own delaymgr callback runs
|
||||
|
|
|
|||
|
|
@ -2729,6 +2729,64 @@ static int record_job_finish_fact(cluster_t* c, job_t* job,
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* worker-rejoin-resync task 3.2 (bug-1747 fix, Req 4.2): WAL_JOB_FINISH's
|
||||
* own worker-reported-finish-time sibling of record_job_finish_fact() above
|
||||
* -- identical guard/hook-fire/durable-append body, except it passes a
|
||||
* CALLER-supplied finish time (converted from the worker's reported
|
||||
* finish_time_ms, Req 3.5) to persist_submit_job_finish_fact() instead of
|
||||
* always deriving time(NULL) at receipt time. record_job_finish_fact()
|
||||
* itself is deliberately left UNMODIFIED: its own callers (on_job_finish's
|
||||
* ordinary, near-instantaneous fire-and-forget worker report) are correctly
|
||||
* served by receipt-time ~= true finish-time; only a RECONCILED finish
|
||||
* (received long after the fact, on rejoin) needs the worker's own reported
|
||||
* time instead -- mirroring task 2.1/2.2's own "new function alongside an
|
||||
* unmodified existing one" pattern (record_job_reconcile_pending_fact/
|
||||
* record_job_reconcile_assign_fact), not a new precedent.
|
||||
*
|
||||
* persist_submit_job_finish_fact()'s own `finish_time` parameter is SECONDS
|
||||
* (time_t semantics) -- confirmed by its only other call site
|
||||
* (record_job_finish_fact() above: `time_t finish_time = time(NULL);
|
||||
* ...(uint64_t)finish_time`) and by the WAL_JOB_FINISH decode side
|
||||
* (apply_committed_entry, persist.c: `j->job.finish_time =
|
||||
* (time_t)wire_finish_time;`, a direct, unconverted assignment). The
|
||||
* worker-reported wire field (xce_job_state_entry_t.finish_time_ms,
|
||||
* typemap.h) is MILLISECONDS ("finishTimeMs" on the wire, Req 3.5) -- this
|
||||
* function converts ms -> s (`finish_time_ms / 1000`) before calling
|
||||
* persist_submit_job_finish_fact(), so the caller (resync.c) can pass the
|
||||
* raw worker-reported value straight through without needing to know this
|
||||
* unit detail itself.
|
||||
*
|
||||
* Same locking convention as record_job_finish_fact(): caller holds `job`
|
||||
* under RTTI_LOCK_WRITE on entry and keeps holding it until this call
|
||||
* returns; this function does not lock/unlock `job` itself. Same DEFERRED
|
||||
* pattern too: durable append only here, no mutation -- the in-memory state
|
||||
* change happens later via apply_committed_entry()'s replay, exactly as an
|
||||
* ordinary finish already works (unlike record_job_reconcile_pending_fact/
|
||||
* assign_fact's own EAGER pattern) -- so the caller must NOT call
|
||||
* persist_notify_job_updated_already_persisted() after this returns; that
|
||||
* notify happens later, at apply time, exactly as it already does for
|
||||
* record_job_finish_fact(). */
|
||||
int record_job_finish_fact_at(cluster_t* c, job_t* job,
|
||||
xce_jobfinish_t* finish, uint64_t finish_time_ms)
|
||||
{
|
||||
if (job_state_is_terminal(job->job.state)) {
|
||||
apply_job_finish_transition(job, finish); /* logs + no-op, unchanged */
|
||||
return 0;
|
||||
}
|
||||
|
||||
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
|
||||
HOOK_DETAIL_JOB_EXIT, c, finish, job);
|
||||
|
||||
(void)persist_submit_job_finish_fact(&c->persist, job->job.jobid,
|
||||
(finish->exit_code == 0)
|
||||
? JOB_STAT_DONE
|
||||
: JOB_STAT_EXIT,
|
||||
finish_time_ms / 1000,
|
||||
finish->exit_code);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int on_job_finish(cluster_t* c, xce_jobfinish_t* finish)
|
||||
{
|
||||
job_t* job = NULL;
|
||||
|
|
|
|||
|
|
@ -23,7 +23,11 @@
|
|||
#include <time.h>
|
||||
|
||||
#include "cluster.h"
|
||||
#include "controller.capnp.h" /* JobStateStatus_* -- worker-rejoin-resync
|
||||
* task 3.2 */
|
||||
#include "delay.h"
|
||||
#include "persist.h" /* persist_notify_job_updated_already_persisted --
|
||||
* worker-rejoin-resync task 3.2 */
|
||||
#include "resync.h"
|
||||
#include "utils.h"
|
||||
#include "xce/types.h"
|
||||
|
|
@ -412,6 +416,343 @@ int resync_round_is_current(worker_t* w, uint64_t token)
|
|||
return w->resync->round_token == token;
|
||||
}
|
||||
|
||||
/* ── Report application (task 3.2, Req 4.1-4.5, 6.1-6.3) ─────────────────
|
||||
*
|
||||
* Maps each JobStateEntry in a worker's job-state report to the correct
|
||||
* idempotent job.c fact-recording call, then resolves the round once every
|
||||
* queried job has a reconciled outcome. See resync.h's doc comment on
|
||||
* resync_apply_report() for the full per-outcome routing contract, and
|
||||
* design.md's "System Flows" sequence diagram (the `loop each reported
|
||||
* entry` block) for the design this implements.
|
||||
*/
|
||||
|
||||
/* Req 6.2: removes `jobid` from w->resync's outstanding query set, if
|
||||
* present -- swap-with-last-and-shrink (the array is small, bounded by one
|
||||
* host's non-terminal job count per resync_collect_query_jobids' own
|
||||
* reasoning, so O(n) removal is not a concern). A no-op when w->resync is
|
||||
* NULL or `jobid` is not present -- safe to call unconditionally and
|
||||
* repeatedly (an already-cleared id, or one that was never queried, e.g. an
|
||||
* unqueried extra). Caller holds w's write lock. */
|
||||
static void resync_clear_outstanding(worker_t* w, uint64_t jobid)
|
||||
{
|
||||
resync_state_t* rs = w->resync;
|
||||
int i;
|
||||
|
||||
if (rs == NULL)
|
||||
return;
|
||||
|
||||
for (i = 0; i < rs->n_queried_jobids; i++) {
|
||||
if (rs->queried_jobids[i] == jobid) {
|
||||
rs->queried_jobids[i]
|
||||
= rs->queried_jobids[rs->n_queried_jobids - 1];
|
||||
rs->n_queried_jobids--;
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Req 5.1/5.2: resolves w's round if its outstanding set has just become
|
||||
* empty. Caller holds w's write lock and must have already confirmed the
|
||||
* round is still current for the token in question (see
|
||||
* resync_settle_entry below) -- this function itself only re-checks
|
||||
* w->resync != NULL (resync_resolve_round's own precondition), not token
|
||||
* identity, since the caller's own check under the SAME uninterrupted lock
|
||||
* hold already establishes that. */
|
||||
static void resync_maybe_resolve(cluster_t* c, worker_t* w)
|
||||
{
|
||||
if (w->resync != NULL && w->resync->n_queried_jobids == 0)
|
||||
resync_resolve_round(c, w);
|
||||
}
|
||||
|
||||
/* Req 4.1, 5.1, 5.2, 6.1-6.3: clears `jobid` from w's outstanding set for
|
||||
* round `token` and, if the set has just become empty, resolves the round
|
||||
* -- both performed under a SINGLE w write-lock acquisition, both gated on
|
||||
* `token` still being w's CURRENT round. The gate matters: a concurrent Req
|
||||
* 5.3 timeout fire, or a fresh resync_arm_round superseding this round on
|
||||
* another thread, can happen between this report-apply loop's per-entry
|
||||
* fact application (which holds no lock on `w`) and this call -- without
|
||||
* re-validating here, a stale round's late bookkeeping could otherwise
|
||||
* mutate a DIFFERENT (newer) round's resync_state_t. This is also the exact
|
||||
* mechanism that makes re-applying an already-fully-resolved report a safe
|
||||
* no-op (Req 6.2): once resync_resolve_round() has run, w->resync is NULL,
|
||||
* so `token` no longer reports current and this becomes a pure no-op. */
|
||||
static void resync_settle_entry(cluster_t* c, worker_t* w, uint64_t token,
|
||||
uint64_t jobid)
|
||||
{
|
||||
lock(w, RTTI_LOCK_WRITE);
|
||||
if (resync_round_is_current(w, token)) {
|
||||
resync_clear_outstanding(w, jobid);
|
||||
resync_maybe_resolve(c, w);
|
||||
}
|
||||
unlock(w);
|
||||
}
|
||||
|
||||
/* Looks up `jobid` in c->jobmgr, reference()-ing it on success -- the same
|
||||
* key_find+reference idiom on_job_start/on_job_finish/on_job_fail/
|
||||
* dispatch_a_job_ (job.c) already use throughout this codebase. Returns
|
||||
* NULL if not found -- never treated as an error by this file's own callers
|
||||
* (see resync_apply_running_entry's doc comment for why a genuinely-
|
||||
* untracked jobid is a legitimate, if narrow, possibility). Caller must
|
||||
* dereference() the result when done, if non-NULL. */
|
||||
static job_t* resync_lookup_job(cluster_t* c, uint64_t jobid)
|
||||
{
|
||||
job_t* j = NULL;
|
||||
|
||||
lock(c->jobmgr, RTTI_LOCK_READ);
|
||||
key_find(c->jobmgr, (void*)jobid, (void**)&j);
|
||||
if (j != NULL)
|
||||
reference(j);
|
||||
unlock(c->jobmgr);
|
||||
|
||||
return j;
|
||||
}
|
||||
|
||||
/* Req 4.1, 4.4, 4.7, 4.8: routes a "running" entry. A QUERIED running job's
|
||||
* own state already correctly reflects "running" -- no fact is recorded
|
||||
* (design.md's sequence diagram: "else still running -> clear this jobid
|
||||
* from outstanding set (no-op fact)"); the caller clears the outstanding
|
||||
* set for every queried entry uniformly (resync_apply_report below), so
|
||||
* this function has nothing further to do for that case.
|
||||
*
|
||||
* An UNQUERIED extra (Req 3.6) running job is a genuine THREE-WAY split, not
|
||||
* a two-way boolean (review round 2 regression fix, following round 1's own
|
||||
* review-caught fix -- the round-1 guard introduced a NEW Req 4.4 violation:
|
||||
* `!terminal && assigned-to-w` short-circuits to `false` whenever the job IS
|
||||
* terminal, falling through to the assign branch and unconditionally
|
||||
* clobbering a DONE/EXIT job back to RUN -- confirmed live: round 1's own
|
||||
* test seeded a TERMINAL j_extra and asserted it became RUN, certifying the
|
||||
* bug as "correct" instead of catching it):
|
||||
*
|
||||
* (a) TERMINAL (resync_job_is_terminal()) -> ALWAYS a no-op, regardless of
|
||||
* host. Requirement 4.4 ("shall not overwrite that state with a less
|
||||
* final or older report") is unconditional -- a terminal job is never
|
||||
* assign-eligible, full stop. record_job_reconcile_assign_fact() has NO
|
||||
* backward-transition guard of its own (its own doc comment: "no
|
||||
* reject/no-op case" is a documented CALLER precondition, never
|
||||
* re-checked internally) and unconditionally sets job->job.state =
|
||||
* JOB_STAT_RUN -- calling it here would clobber DONE/EXIT back to RUN.
|
||||
* (b) Non-terminal AND job->job.allocs != NULL (already has SOME live
|
||||
* non-terminal host assignment -- to THIS reporting worker's own host,
|
||||
* or to a DIFFERENT one, it does not matter which) -> a no-op. It is
|
||||
* already validly assigned; record_job_reconcile_assign_fact() would
|
||||
* still be wrong to call even though its net STATE effect looks
|
||||
* harmless (RUN -> RUN) -- it unconditionally resets job->job.start_time
|
||||
* and re-appends a fresh WAL_JOB_DISPATCH fact, neither of which is
|
||||
* warranted for a job that was never actually lost/unassigned.
|
||||
* (c) Non-terminal AND job->job.allocs == NULL (genuinely UNASSIGNED) ->
|
||||
* Req 4.8's real trigger condition ("no existing non-terminal record...
|
||||
* assigned elsewhere" -- there IS a non-terminal record, but it carries
|
||||
* no assignment at all) -- reconciled via record_job_reconcile_assign_
|
||||
* fact() (task 2.2), documented (and hard-asserted, job.c) as the
|
||||
* running-only case.
|
||||
*
|
||||
* Concrete motivating scenario for (c) (the exact case design.md's own
|
||||
* "Documented scope decision" paragraph describes as this function's
|
||||
* target, and record_job_reconcile_assign_fact()'s core justification for
|
||||
* existing at all): a job the controller currently shows PEND (non-
|
||||
* terminal, allocs == NULL, unassigned to any host) because its
|
||||
* WAL_JOB_DISPATCH fact never reached durable majority commit before a
|
||||
* controller failover -- even though dispatch_a_job_() already sent
|
||||
* xce_startjob_req() to the worker and the worker really is running it (the
|
||||
* RPC fires BEFORE dispatch_a_job_()'s own eager mutation + WAL append, so
|
||||
* the two can diverge exactly at a failover boundary). Such a job correctly
|
||||
* reconciles via record_job_reconcile_assign_fact() (host now matches the
|
||||
* reporting worker, allocs stays NULL per Req 4.8's own documented scope
|
||||
* decision).
|
||||
*
|
||||
* A jobid with NO record AT ALL in the job table cannot be reconciled via
|
||||
* record_job_reconcile_assign_fact() either -- that function mutates an
|
||||
* EXISTING job_t, it does not construct one -- so this is logged and
|
||||
* skipped rather than fabricating a full job record from a bare entry;
|
||||
* every job a worker can legitimately run was, by this system's own
|
||||
* architecture, submitted through this controller at some point, so this
|
||||
* branch is expected to be unreachable in practice, not a routine path. */
|
||||
static void resync_apply_running_entry(cluster_t* c, worker_t* w,
|
||||
xce_job_state_entry_t* entry)
|
||||
{
|
||||
job_t* job;
|
||||
|
||||
if (entry->queried)
|
||||
return;
|
||||
|
||||
job = resync_lookup_job(c, entry->jobid);
|
||||
if (job == NULL) {
|
||||
WARN("resync: unqueried extra job %lu reported running by worker "
|
||||
"%s has no record at all in the job table -- cannot reconcile "
|
||||
"(no existing job_t to assign onto); skipped",
|
||||
(unsigned long)entry->jobid,
|
||||
(w->host.name != NULL) ? w->host.name : "?");
|
||||
return;
|
||||
}
|
||||
|
||||
lock(job, RTTI_LOCK_WRITE);
|
||||
if (resync_job_is_terminal(job->job.state)) {
|
||||
/* (a) Terminal is always final -- Req 4.4. Never assign-eligible,
|
||||
* regardless of host. */
|
||||
unlock(job);
|
||||
} else if (job->job.allocs != NULL) {
|
||||
/* (b) Non-terminal AND already has SOME live non-terminal host
|
||||
* assignment -- a genuine no-op; do not reset start_time or
|
||||
* re-append a WAL_JOB_DISPATCH fact for a job that was never
|
||||
* actually lost/unassigned. */
|
||||
unlock(job);
|
||||
} else {
|
||||
/* (c) Non-terminal AND genuinely UNASSIGNED -- Req 4.8 fires. */
|
||||
int rc = record_job_reconcile_assign_fact(c, job, w, entry);
|
||||
unlock(job);
|
||||
if (rc == (int)JOB_TRANSITION_APPLIED)
|
||||
persist_notify_job_updated_already_persisted(c, job);
|
||||
}
|
||||
|
||||
dereference(job);
|
||||
}
|
||||
|
||||
/* Req 4.1-4.5, 4.7: routes a "finished" entry (success or failure) through
|
||||
* the SAME existing, deferred-pattern fact-recording used for an ordinary
|
||||
* worker finish/fail report. Applies identically whether entry->queried is
|
||||
* true or false (Req 4.7: an unqueried finished extra reconciles through
|
||||
* "the same idempotent commit-then-apply path... used for queried jobs" --
|
||||
* there is no separate "assign" case for a finished outcome; record_job_
|
||||
* reconcile_assign_fact() is documented as running-only, see job.c's own
|
||||
* doc comment on it).
|
||||
*
|
||||
* finishedOk: bug-1747 fix (Req 4.2) -- routed through the NEW record_job_
|
||||
* finish_fact_at() (job.c), not on_job_finish()/record_job_finish_fact(),
|
||||
* so the job's recorded finish_time is the WORKER-reported entry->
|
||||
* finish_time_ms (Req 3.5), not reconciliation-processing wall-clock time.
|
||||
* record_job_finish_fact_at() expects the caller to already hold job's
|
||||
* write lock (like record_job_reconcile_pending_fact/assign_fact above),
|
||||
* unlike on_job_finish() which locks internally -- this function performs
|
||||
* that lookup/lock/unlock/dereference sequence itself, mirroring on_job_
|
||||
* finish()'s own internal sequence exactly (job.c). A jobid not found in
|
||||
* the job table is a silent no-op, mirroring on_job_finish()'s own
|
||||
* "can not find job" behavior for an unknown jobid.
|
||||
*
|
||||
* finishedFail: still routed through the existing, UNMODIFIED on_job_fail()
|
||||
* /record_job_fail_fact() -- Requirement 4.2 remains unsatisfiable for this
|
||||
* outcome via any existing path in this codebase, not a reconciliation-
|
||||
* specific gap: WAL_JOB_FAIL's payload (persist_submit_job_fact_ex(...,
|
||||
* WAL_JOB_FAIL, jobid, 0, WAL_JOB_FAIL_STAGE_FLAG|stage)) carries no
|
||||
* finish-time slot at any layer, and apply_job_fail_transition() (job.c)
|
||||
* unconditionally sets job->job.finish_time = time(NULL) at APPLY time
|
||||
* (not even receipt time) for EVERY fail-reporting caller in this system,
|
||||
* confirmed by direct inspection -- a pre-existing, systemic gap that
|
||||
* predates this feature and is out of this task's boundary to fix (see
|
||||
* buglog.json bug-1747, re-scoped). The reconciled job's STATE (EXIT) is
|
||||
* always correct either way; only its recorded finish_time reflects
|
||||
* apply-processing time rather than the worker's true report. */
|
||||
static void resync_apply_finished_entry(cluster_t* c,
|
||||
xce_job_state_entry_t* entry)
|
||||
{
|
||||
if (entry->status == JobStateStatus_finishedOk) {
|
||||
job_t* job = resync_lookup_job(c, entry->jobid);
|
||||
|
||||
if (job != NULL) {
|
||||
xce_jobfinish_t finish
|
||||
= { .jobid = entry->jobid, .exit_code = 0 };
|
||||
|
||||
lock(job, RTTI_LOCK_WRITE);
|
||||
record_job_finish_fact_at(c, job, &finish,
|
||||
entry->finish_time_ms);
|
||||
unlock(job);
|
||||
|
||||
dereference(job);
|
||||
}
|
||||
} else {
|
||||
job_failinfo_t fail
|
||||
= { .jobid = entry->jobid,
|
||||
.stage = JOB_EXEC_STAGE_MAIN,
|
||||
.code = entry->exit_code,
|
||||
.detail = (char*)"worker-rejoin-resync reconciliation" };
|
||||
on_job_fail(c, &fail);
|
||||
}
|
||||
}
|
||||
|
||||
/* Req 4.1, 4.5, 4.6: routes an "unrecognized" entry through record_job_
|
||||
* reconcile_pending_fact() (task 2.1). Per Req 3.1/3.3, a worker only ever
|
||||
* reports "unrecognized" for a jobid it was explicitly asked about (an
|
||||
* unqueried extra is, by construction of Req 3.6's own "currently running
|
||||
* or recently finished" trigger, never reported unrecognized) -- so this
|
||||
* path is queried-only in practice; the caller (resync_apply_report below)
|
||||
* still clears the outstanding set uniformly for every queried entry
|
||||
* regardless of which branch handled it. */
|
||||
static void resync_apply_unrecognized_entry(cluster_t* c,
|
||||
xce_job_state_entry_t* entry)
|
||||
{
|
||||
job_t* job = resync_lookup_job(c, entry->jobid);
|
||||
int rc;
|
||||
|
||||
if (job == NULL)
|
||||
return;
|
||||
|
||||
lock(job, RTTI_LOCK_WRITE);
|
||||
rc = record_job_reconcile_pending_fact(c, job);
|
||||
unlock(job);
|
||||
|
||||
if (rc == (int)JOB_TRANSITION_APPLIED)
|
||||
persist_notify_job_updated_already_persisted(c, job);
|
||||
|
||||
dereference(job);
|
||||
}
|
||||
|
||||
int resync_apply_report(cluster_t* c, worker_t* w, uint64_t round_token,
|
||||
xce_job_state_report_t* report)
|
||||
{
|
||||
int i;
|
||||
|
||||
if (c == NULL || w == NULL || report == NULL)
|
||||
return -EINVAL;
|
||||
if (report->count > 0 && report->entries == NULL)
|
||||
return -EINVAL;
|
||||
|
||||
/* Req 6.1, 6.3: a report for a round already superseded/discarded is
|
||||
* dropped in its entirety, before anything else is touched -- checked
|
||||
* under w's own write lock (matching every other w->resync mutation
|
||||
* this function goes on to make, via resync_settle_entry, per entry
|
||||
* below). */
|
||||
lock(w, RTTI_LOCK_WRITE);
|
||||
if (!resync_round_is_current(w, round_token)) {
|
||||
unlock(w);
|
||||
return 0;
|
||||
}
|
||||
unlock(w);
|
||||
|
||||
/* Per-entry routing never holds w's write lock while also locking a
|
||||
* job's own write lock -- mirrors dispatch_a_job_'s established
|
||||
* discipline (job.c: every branch releases w's lock BEFORE acquiring a
|
||||
* job's) rather than this file's other three functions' "caller holds
|
||||
* w's lock for the whole call" convention, which fits their simple,
|
||||
* single-object, single-call-site mutations but not this function's
|
||||
* fan-out into per-job job.c calls, each requiring its own job-level
|
||||
* lock. w's lock is only (re)acquired, narrowly, inside
|
||||
* resync_settle_entry's own outstanding-set/maybe-resolve mutation. */
|
||||
for (i = 0; i < report->count; i++) {
|
||||
xce_job_state_entry_t* entry = report->entries[i];
|
||||
|
||||
if (entry == NULL)
|
||||
continue;
|
||||
|
||||
switch (entry->status) {
|
||||
case JobStateStatus_running:
|
||||
resync_apply_running_entry(c, w, entry);
|
||||
break;
|
||||
case JobStateStatus_finishedOk:
|
||||
case JobStateStatus_finishedFail:
|
||||
resync_apply_finished_entry(c, entry);
|
||||
break;
|
||||
case JobStateStatus_unrecognized:
|
||||
default:
|
||||
resync_apply_unrecognized_entry(c, entry);
|
||||
break;
|
||||
}
|
||||
|
||||
if (entry->queried)
|
||||
resync_settle_entry(c, w, round_token, entry->jobid);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/* worker-rejoin-resync task 3.1 test seam (resync.h): builds a
|
||||
* resync_timeout_ctx_t exactly as resync_arm_round's own delay_task_ms call
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue