lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)

Browse source 
Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.

Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.

Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.

Full controller suite: 1239 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 15:24:29 +00:00
parent 442fcf4544
commit eec2596c5d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1600 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

62
.wolf/buglog.json
View file

@ -20371,6 +20371,68 @@
],
"occurrences": 2,
"last_seen": "2026-09-01T01:00:00Z"
},
{
"id": "bug-1747",
"timestamp": "2026-09-01T00:00:00Z",
"error_message": "Pre-existing, systemic gap predating worker-rejoin-resync entirely, discovered while implementing task 3.2 (resync_apply_report, resync.c): WAL_JOB_FAIL carries no fail-time/finish-time slot in its wire payload at ANY layer, and apply_job_fail_transition() (job.c) unconditionally stamps job.finish_time = time(NULL) at APPLY time (not even receipt time) for EVERY fail-reporting caller in this codebase -- not just reconciliation. This means Requirement 4.2 ('the Controller shall record the finish time the Worker reported ... rather than the time the Controller processed the reconciliation') is currently unsatisfiable for a reconciled finished-FAILURE outcome via any existing path. UPDATE (2026-09-01, coordinator-directed re-scope): the sibling FINISH-success half of this gap is FIXED -- see 'fix' field below; only the FAIL half remains open.",
"file": "src/controller/job.c",
"root_cause": "job_failinfo_t (xce/types.h) is {jobid, stage, code, detail} -- no finish_time field, and WAL_JOB_FAIL's durable payload (record_job_fail_fact(), job.c: persist_submit_job_fact_ex(&c->persist, WAL_JOB_FAIL, jobid, 0, WAL_JOB_FAIL_STAGE_FLAG|stage)) carries no finish-time slot at all -- confirmed by direct inspection of the payload construction. apply_job_fail_transition() (job.c, the guarded core BOTH the live job_ctrl path (mark_job_fail) and the deferred apply seam (apply_committed_entry's WAL_JOB_FAIL case) invoke) unconditionally executes `job->job.finish_time = time(NULL);` with no external value ever consulted, at APPLY time -- meaning even a caller willing to plumb a real value through record_job_fail_fact's WAL payload would still need a apply_job_fail_transition() signature change to have it actually land in job.job.finish_time. This is a pre-existing, already-documented residual: job.c's own comment near apply_job_postdone_transition (task 4.1) already states 'finish_time is not part of the wal_jobstate_t wire format ... has no apply-time equivalent' for a sibling fact type, confirming this is systemic to the wal_jobstate_t-based fact family, not something worker-rejoin-resync introduced. Fixing it for real would require extending WAL_JOB_FAIL's wire payload (wal_types.h, mirroring bug-1689's own WAL_JOB_FINISH trailing-fields precedent) PLUS an apply_job_fail_transition() signature/behavior change reaching every fail-reporting call site in the system (job_ctrl's mark_job_fail, on_job_fail's ordinary worker-fail receipt path, and this feature's own reconciliation caller) -- real, broad-blast-radius work, not a narrow patch.",
"fix": "SPLIT (2026-09-01, coordinator-directed): the FINISH-success half of this gap is FIXED as part of worker-rejoin-resync task 3.2 -- added record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*, uint64_t finish_time_ms) (job.c), record_job_finish_fact()'s own worker-reported-finish-time sibling (identical guard/hook-fire/durable-append body, differing only in passing the caller-supplied finish_time -- converted ms->s -- to persist_submit_job_finish_fact() instead of time(NULL)); record_job_finish_fact() itself is UNMODIFIED, its own caller (on_job_finish, an ordinary near-instantaneous worker report) correctly keeps using receipt-time. resync_apply_report()'s finishedOk branch now calls record_job_finish_fact_at(c, job, &finish, entry->finish_time_ms). Verified via a real restart-replay test (test_apply_report_finished_success_uses_worker_reported_finish_time, test_worker_resync.c): a finished-success entry with finish_time_ms set to 'now minus 1 day' recovers job.finish_time equal to that worker-reported value after persist_recover(), not reconciliation-processing wall-clock time. This was possible narrowly for FINISH because persist_submit_job_finish_fact() already accepted an explicit finish_time parameter at the primitive level -- record_job_finish_fact()'s specific call site just never exposed it. The FAIL half (see root_cause) is NOT YET FIXED and is NOT task 3.2's (or any single narrow task's) job to fix -- it needs the broader WAL_JOB_FAIL payload extension + apply_job_fail_transition() signature change described above, deliberately deferred pending a dedicated task, matching the same 'deliberately deferred, documented, not blocking' treatment bug-1631's own sibling gap received earlier in this same feature. The reconciled job's STATE (EXIT) is always correct for a finished-failure outcome either way -- only its recorded finish_time reflects apply-processing time rather than the worker's true report.",
"tags": [
"worker-rejoin-resync",
"task-3.2",
"resync-c",
"job-c",
"requirement-4.2",
"finish-time",
"wal-job-fail",
"wal-job-finish",
"apply_job_fail_transition",
"pre-existing",
"predates-feature",
"systemic",
"partially-fixed",
"resync_apply_report",
"record_job_finish_fact_at"
],
"related_bugs": [
"bug-1689",
"bug-1678",
"bug-1746",
"bug-1631"
],
"occurrences": 1,
"last_seen": "2026-09-01T01:30:00Z"
},
{
"id": "bug-1748",
"timestamp": "2026-09-01T02:00:00Z",
"error_message": "worker-rejoin-resync task 3.2 (resync_apply_running_entry, resync.c), round-2 independent review: the round-1 fix for the PEND/unassigned-job regression (bug-1747-adjacent, not itself in bug-1747) introduced a NEW Requirement 4.4 violation. The round-1 guard `if (!resync_job_is_terminal(job->job.state) && resync_job_assigned_to_host(job, w->host.name)) { no-op } else { record_job_reconcile_assign_fact(...) }` short-circuits the `&&` to false whenever job->job.state IS terminal, falling into the `else` branch and calling record_job_reconcile_assign_fact() on an ALREADY-TERMINAL (DONE/EXIT) job -- which has no backward-transition guard of its own and unconditionally sets job->job.state = JOB_STAT_RUN, clobbering a terminal job back to RUN. Confirmed live and, worse, CERTIFIED AS CORRECT by the round-1 implementation's own test: test_apply_report_all_five_outcomes_updates_job_table seeded j_extra at JOB_STAT_DONE with a comment claiming this 'satisfies Req 4.8's trigger,' then asserted j_extra->job.state == JOB_STAT_RUN afterward and the suite reported PASS -- the test baked the bug in as an expected outcome instead of catching it.",
"file": "src/controller/resync.c",
"root_cause": "Requirement 4.8's actual trigger condition is 'no existing non-terminal record of that job assigned elsewhere' (requirements.md AC 4.8; record_job_reconcile_assign_fact's own doc comment and design.md's Service Interface Preconditions: 'job is non-terminal ... or has no existing non-terminal assignment'). Terminal-vs-non-terminal is NOT the axis that decides assign-eligibility on its own -- Requirement 4.4 ('shall not overwrite that state with a less final or older report') independently and unconditionally forbids ever assign-routing a terminal job, regardless of host. The round-1 guard conflated two orthogonal conditions (terminal-ness, and host-assignment) into a single boolean expression via `&&`, which is only correct when BOTH must jointly hold for the no-op case -- but Req 4.4's terminal exclusion needs to apply UNCONDITIONALLY, independent of the host check, not only when the host check also happens to be true. A two-way boolean cannot express three independent outcomes (terminal-always-noop; non-terminal-but-already-assigned-noop; non-terminal-and-unassigned-assign) without this exact class of short-circuit bug.",
"fix": "FIXED (2026-09-01, round-2 review, task 3.2). resync_apply_running_entry() rewritten as a genuine three-way `if`/`else if`/`else` split, not a boolean guard: (a) resync_job_is_terminal(job->job.state) -> ALWAYS a no-op, checked FIRST and independently, regardless of host (closes the Req 4.4 violation). (b) non-terminal AND job->job.allocs != NULL (already has SOME live non-terminal host assignment -- to the reporting worker's own host, or to a different one, it does not matter which) -> a no-op; also tightened from round 1's own host-specific check (resync_job_assigned_to_host(job, w->host.name)) to a host-agnostic allocs!=NULL check, since round-1's version would have ALSO wrongly routed a job assigned to a DIFFERENT host into the assign branch, resetting its start_time and re-appending a spurious WAL_JOB_DISPATCH fact for a job that was never actually lost. (c) non-terminal AND allocs == NULL (genuinely unassigned) -> record_job_reconcile_assign_fact(), Req 4.8's real trigger. Verified: (1) test_apply_report_all_five_outcomes_updates_job_table's j_extra is now seeded TERMINAL and asserted COMPLETELY UNTOUCHED (state stays DONE, no WAL append) -- the exact inversion of the round-1 assertion that had certified the bug. (2) test_apply_report_unqueried_extra_already_tracked_is_noop (branch (b), assigned to a DIFFERENT host) strengthened with a start_time-unchanged assertion (record_job_reconcile_assign_fact unconditionally stamps time(NULL), bug-1697 -- invisible to a bare state/allocs check since the job was already RUN either way) and an LSN-unchanged assertion (no spurious WAL_JOB_DISPATCH append) -- round 1's version of this test could not have caught either defect. (3) test_apply_report_unqueried_extra_unassigned_pend_job_gets_assigned (branch (c), genuinely unassigned/PEND) already covers the real assign-success case with full restart-replay rigor, added in the same round-1 fix pass. Full controller test suite (meson test -C src/xce.build 'controller test'): all green.",
"tags": [
"worker-rejoin-resync",
"task-3.2",
"resync-c",
"requirement-4.4",
"requirement-4.8",
"resync_apply_running_entry",
"record_job_reconcile_assign_fact",
"guard-logic",
"short-circuit",
"review-caught",
"test-baked-in-bug",
"fixed"
],
"related_bugs": [
"bug-1747",
"bug-1697",
"bug-1631"
],
"occurrences": 1,
"last_seen": "2026-09-01T02:00:00Z"
}
]
}