lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)

Browse source 
Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.

Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.

Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.

Full controller suite: 1239 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 15:24:29 +00:00
parent 442fcf4544
commit eec2596c5d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1600 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

58
src/controller/job.c
View file

@ -2729,6 +2729,64 @@ static int record_job_finish_fact(cluster_t* c, job_t* job,
return 0;
}
/* worker-rejoin-resync task 3.2 (bug-1747 fix, Req 4.2): WAL_JOB_FINISH's
* own worker-reported-finish-time sibling of record_job_finish_fact() above
* -- identical guard/hook-fire/durable-append body, except it passes a
* CALLER-supplied finish time (converted from the worker's reported
* finish_time_ms, Req 3.5) to persist_submit_job_finish_fact() instead of
* always deriving time(NULL) at receipt time. record_job_finish_fact()
* itself is deliberately left UNMODIFIED: its own callers (on_job_finish's
* ordinary, near-instantaneous fire-and-forget worker report) are correctly
* served by receipt-time ~= true finish-time; only a RECONCILED finish
* (received long after the fact, on rejoin) needs the worker's own reported
* time instead -- mirroring task 2.1/2.2's own "new function alongside an
* unmodified existing one" pattern (record_job_reconcile_pending_fact/
* record_job_reconcile_assign_fact), not a new precedent.
*
* persist_submit_job_finish_fact()'s own `finish_time` parameter is SECONDS
* (time_t semantics) -- confirmed by its only other call site
* (record_job_finish_fact() above: `time_t finish_time = time(NULL);
* ...(uint64_t)finish_time`) and by the WAL_JOB_FINISH decode side
* (apply_committed_entry, persist.c: `j->job.finish_time =
* (time_t)wire_finish_time;`, a direct, unconverted assignment). The
* worker-reported wire field (xce_job_state_entry_t.finish_time_ms,
* typemap.h) is MILLISECONDS ("finishTimeMs" on the wire, Req 3.5) -- this
* function converts ms -> s (`finish_time_ms / 1000`) before calling
* persist_submit_job_finish_fact(), so the caller (resync.c) can pass the
* raw worker-reported value straight through without needing to know this
* unit detail itself.
*
* Same locking convention as record_job_finish_fact(): caller holds `job`
* under RTTI_LOCK_WRITE on entry and keeps holding it until this call
* returns; this function does not lock/unlock `job` itself. Same DEFERRED
* pattern too: durable append only here, no mutation -- the in-memory state
* change happens later via apply_committed_entry()'s replay, exactly as an
* ordinary finish already works (unlike record_job_reconcile_pending_fact/
* assign_fact's own EAGER pattern) -- so the caller must NOT call
* persist_notify_job_updated_already_persisted() after this returns; that
* notify happens later, at apply time, exactly as it already does for
* record_job_finish_fact(). */
int record_job_finish_fact_at(cluster_t* c, job_t* job,
xce_jobfinish_t* finish, uint64_t finish_time_ms)
{
if (job_state_is_terminal(job->job.state)) {
apply_job_finish_transition(job, finish); /* logs + no-op, unchanged */
return 0;
}
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_EXIT, c, finish, job);
(void)persist_submit_job_finish_fact(&c->persist, job->job.jobid,
(finish->exit_code == 0)
? JOB_STAT_DONE
: JOB_STAT_EXIT,
finish_time_ms / 1000,
finish->exit_code);
return 0;
}
int on_job_finish(cluster_t* c, xce_jobfinish_t* finish)
{
job_t* job = NULL;