lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): ordered live apply pump activates apply_committed_entry (task 6.1)

Browse source 
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.

Fixes the two hazards flagged since task 4.1 as required before
going live:

1. Double-append: a thread-local call-provenance guard
   (t_apply_seam_notifying) prevents persist_job_hook_fun/
   persist_queue_hook_fun/persist_worker_hook_fun from re-appending
   entries the apply pump itself just applied, while preserving
   job_ctrl's legitimate live command-path appends (a naive
   detail-based filter would have broken SIGCONT/SIGSTOP). Also
   closes an undocumented instance of the same hazard in the
   queue/worker hook functions.

2. Merge-staleness: record_job_fail_fact/postdone's fact recording
   previously baked a receipt-time-computed final job state into the
   WAL entry, which goes stale under deferred apply -- a FINISH
   followed shortly by an ordinary post-exec FAIL could silently
   clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
   guard catching it. Fixed by repurposing wal_jobstate_t's unused
   _pad field to carry just enough data (FAIL's stage) to recompute
   against the job's actual current state at apply time via the real
   apply_job_fail_transition()/new apply_job_postdone_transition()
   guard cores, instead of overwriting with a stale absolute value.
   Backward compatible: _pad==0 (all prior data) routes to the same
   legacy overwrite behavior.

Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).

Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
This commit is contained in:
Rongsong Shen 2026-07-04 19:00:43 +00:00
parent d3da3ca8bd
commit 2fa955b92e
11 changed files with 1687 additions and 135 deletions
Download patch file Download diff file Expand all files Collapse all files

150
src/controller/job.c
View file

@ -17,7 +17,6 @@
static int record_job_start_fact(cluster_t*, job_t*, job_pidinfo_t*);
static int record_job_finish_fact(cluster_t*, job_t*, xce_jobfinish_t*);
static int record_job_fail_fact(cluster_t*, job_t*, job_failinfo_t*);
static int record_job_update_fact(cluster_t*, job_t*, int);
static int mark_job_fail(cluster_t*, job_t*, job_failinfo_t*);
static int mark_job_update(cluster_t*, job_t*, int);
@ -1157,29 +1156,37 @@ static int mark_job_fail(cluster_t* c, job_t* job, job_failinfo_t* failinfo)
* peek/rejected branch here — BEFORE_EVENT fires unconditionally, exactly as
* mark_job_fail's always does today.
*
* The resultant state depends on the CURRENT job state (PERR merges onto an
* existing DONE/EXIT terminal state rather than resetting it — job.c's
* pre-existing apply_job_fail_transition switch). That computation is
* reproduced here on a throwaway shallow copy of `job` — apply_job_fail_
* transition only ever touches `job->job.state`/`job->job.finish_time`
* (scalar fields, never pidinfo/exit_info), so a shallow copy is safe to
* discard afterward with no cleanup — rather than duplicating the switch,
* so this can never drift from what apply time will eventually compute. */
* task 6.1 round 2 fix (merge-staleness corruption, Req 11.3): the resultant
* state depends on the CURRENT job state (PERR merges onto an existing
* DONE/EXIT terminal state rather than resetting it — apply_job_fail_
* transition's switch below). Earlier this was precomputed HERE, at receipt
* time, on a throwaway shallow copy of `job`, and the precomputed absolute
* result was baked into the WAL entry. That is unsafe under deferred apply:
* another fact for the same job (e.g. a FINISH) can still be pending apply
* when this FAIL is received, so `job->job.state` at receipt time may not
* be the state this FAIL's merge should actually be computed against —
* baking a stale snapshot silently clobbers whatever the other fact applies
* first (bug found in task 6.1 round 1 review). Fixed by carrying ONLY
* `failinfo->stage` (via _pad's WAL_JOB_FAIL_STAGE_FLAG — see wal_types.h's
* doc comment on wal_jobstate_t._pad) and deferring the actual merge
* computation to apply time, against the job's real current state then
* (apply_committed_entry's WAL_JOB_FAIL case, persist.c). */
static int record_job_fail_fact(cluster_t* c, job_t* job,
job_failinfo_t* failinfo)
{
job_t scratch;
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_FAIL, c, failinfo, job);
scratch = *job;
apply_job_fail_transition(&scratch, failinfo);
/* Durable append only — no mutation of the real `job`, no AFTER_EVENT
* firing here (Req 5.1, 5.3). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_FAIL, job->job.jobid,
(int)scratch.job.state);
* firing here (Req 5.1, 5.3). `state` is unused for this apply-time-
* recomputed shape (0 is written for forward-compatibility/diagnostics
* only); the WAL_JOB_FAIL_STAGE_FLAG bit in `_pad` tells
* apply_committed_entry() to route it to the real
* apply_job_fail_transition() core instead of an absolute overwrite. */
(void)persist_submit_job_fact_ex(&c->persist, WAL_JOB_FAIL,
job->job.jobid, 0,
WAL_JOB_FAIL_STAGE_FLAG |
failinfo->stage);
return 0;
}
@ -1221,10 +1228,10 @@ job_transition_result_t apply_job_update_transition(job_t* job, int state)
/* Used only by job_ctrl's SIGCONT/SIGSTOP suspend-resume branches (a client
* command path, do_jobcontrol_op — out of raft-apply-decoupling task 4.1's
* boundary) — mutate-then-notify, unchanged. record_job_update_fact below is
* the separate, durable-append-only sibling used by the worker-fact receipt
* path (on_job_postdone); the two must stay independent since job_ctrl's
* synchronous command semantics are not this task's concern. */
* boundary) — mutate-then-notify, unchanged. record_job_postdone_fact below
* is the separate, durable-append-only sibling used by the worker-fact
* receipt path (on_job_postdone); the two must stay independent since
* job_ctrl's synchronous command semantics are not this task's concern. */
static int mark_job_update(cluster_t* c, job_t* job, int state)
{
int cur = job->job.state;
@ -1252,33 +1259,68 @@ static int mark_job_update(cluster_t* c, job_t* job, int state)
return 0;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the worker-fact receipt sibling
* of mark_job_update above — durable-append-only, used only by
* on_job_postdone. mark_job_update itself is untouched (job_ctrl still needs
* its mutate-then-notify semantics; that command path is out of this task's
* boundary). `state` here is the caller's already-computed target state
* (on_job_postdone computes it from the job's current state before calling
* this, exactly as it always has — that computation is untouched by this
* task; only what happens with the computed value changes). */
static int record_job_update_fact(cluster_t* c, job_t* job, int state)
/* Guarded state-transition core for a postdone fact, extracted from
* on_job_postdone's old inline mutation (raft-apply-decoupling task 6.1
* round 2, Req 3.3/7.1-7.3). Unlike a generic UPDATE, postdone's formula
* (conditionally backstop EXIT, then OR in JOB_STAT_PDONE) is a pure
* function of the job's CURRENT state — so this core is meant to be invoked
* AT APPLY TIME, against whatever state actually holds then, never against
* a receipt-time snapshot (a receipt-time snapshot is exactly the merge-
* staleness hazard this round fixes: another fact for the same job, e.g. a
* FINISH, may still be in flight and apply AFTER receipt but BEFORE this
* fact's own apply, under deferred apply). No-op when PDONE is already set
* and no backstop is needed (idempotent re-apply, Req 3.4). Fires NO hook —
* same convention as the other apply_job_*_transition cores; hook-firing
* stays the caller's (apply_committed_entry's) responsibility. Caller must
* hold `job` under RTTI_LOCK_WRITE.
*
* Known, previously-accepted residual (task 4.1, unchanged by this round):
* finish_time is not part of the wal_jobstate_t wire format, so the old
* defensive `job->job.finish_time = time(NULL)` backstop write has no
* apply-time equivalent — a postdone arriving with no prior exit status
* recorded still gets no defensive finish_time set anywhere. */
job_transition_result_t apply_job_postdone_transition(job_t* job)
{
int cur = job->job.state;
int next = job->job.state;
/* Peek the identical guards apply_job_update_transition() re-checks
* internally — same no-observable-behavior-change rationale as
* mark_job_update's peek above. */
if (job_state_is_backward(cur, state) || (state == cur)) {
apply_job_update_transition(job, state); /* logs + no-op, unchanged */
return 0;
if ((next & (JOB_STAT_EXIT | JOB_STAT_DONE)) == 0) {
WARN("job %ld got post exec fail notification but job's exit "
"status has not been set",
job->job.jobid);
next |= JOB_STAT_EXIT;
}
next |= JOB_STAT_PDONE;
if (next == job->job.state) {
INFO("job %ld: duplicate postdone — no-op", job->job.jobid);
return JOB_TRANSITION_NOOP_DUPLICATE;
}
job->job.state = next;
return JOB_TRANSITION_APPLIED;
}
/* raft-apply-decoupling task 4.1/6.1 (Req 5.1, 3.3, 7.1-7.3): durable-
* append-only sibling of the old inline postdone mutation, used only by
* on_job_postdone. Replaces record_job_update_fact (removed in task 6.1
* round 2 — this was its only caller): postdone's mutation is a pure
* function of current state (see apply_job_postdone_transition above), so
* nothing needs to be computed or peeked at receipt time — the WAL entry
* carries no state value at all, just a marker (_pad ==
* WAL_JOB_UPDATE_POSTDONE_MARK) telling apply_committed_entry() to re-
* derive the merge fresh, against whatever the job's real current state is
* when this entry actually applies (fixes the same class of merge-
* staleness corruption record_job_fail_fact's fix addresses above). */
static int record_job_postdone_fact(cluster_t* c, job_t* job)
{
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_UPDATE, c, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, job->job.jobid,
state);
(void)persist_submit_job_fact_ex(&c->persist, WAL_JOB_UPDATE,
job->job.jobid, 0,
WAL_JOB_UPDATE_POSTDONE_MARK);
return 0;
}
@ -1383,7 +1425,6 @@ int on_job_resume(cluster_t* c, uint64_t jobid)
int on_job_postdone(cluster_t* c, endpoint_t* ep, uint64_t jobid)
{
job_t* job = NULL;
int state;
TRACE("%s job post done jobid %ld", __func__, jobid);
@ -1399,29 +1440,14 @@ int on_job_postdone(cluster_t* c, endpoint_t* ep, uint64_t jobid)
return -errno;
}
/* raft-apply-decoupling task 6.1 round 2 (Req 5.1, 11.3): the backstop-
* EXIT-if-unset + OR-in-PDONE mutation formerly computed HERE, at
* receipt time, is now deferred entirely to apply_job_postdone_
* transition() (called from apply_committed_entry() once this fact
* commits) — see that function's doc comment for why baking it here
* against a receipt-time snapshot was unsafe under deferred apply. */
lock(job, RTTI_LOCK_WRITE);
state = job->job.state;
if ((state & (JOB_STAT_EXIT | JOB_STAT_DONE)) == 0) {
WARN("job %ld got post exec fail notification but job's exit status "
"has not been set",
job->job.jobid);
state |= JOB_STAT_EXIT;
/* raft-apply-decoupling task 4.1 (Req 5.1): the backstop
* `job->job.finish_time = time(NULL)` write that used to live here
* is removed — no state mutation may happen at receipt time.
* finish_time is not part of the wal_jobstate_t wire format (only
* {jobid, state} travels — a pre-existing limitation, see
* apply_committed_entry's doc comment in persist.h), so this
* defensive backstop has no equivalent at apply time either; a
* post-done arriving with no prior exit status recorded no longer
* gets a defensive finish_time set anywhere. Low-severity residual,
* flagged in this task's status report — not fixed here (would
* require wire-format enrichment, explicitly out of this task's
* scope per persist.h's Apply Seam doc comment). */
}
state |= JOB_STAT_PDONE;
record_job_update_fact(c, job, state);
record_job_postdone_fact(c, job);
unlock(job);
dereference(job);