lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): ordered live apply pump activates apply_committed_entry (task 6.1)

Browse source 
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.

Fixes the two hazards flagged since task 4.1 as required before
going live:

1. Double-append: a thread-local call-provenance guard
   (t_apply_seam_notifying) prevents persist_job_hook_fun/
   persist_queue_hook_fun/persist_worker_hook_fun from re-appending
   entries the apply pump itself just applied, while preserving
   job_ctrl's legitimate live command-path appends (a naive
   detail-based filter would have broken SIGCONT/SIGSTOP). Also
   closes an undocumented instance of the same hazard in the
   queue/worker hook functions.

2. Merge-staleness: record_job_fail_fact/postdone's fact recording
   previously baked a receipt-time-computed final job state into the
   WAL entry, which goes stale under deferred apply -- a FINISH
   followed shortly by an ordinary post-exec FAIL could silently
   clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
   guard catching it. Fixed by repurposing wal_jobstate_t's unused
   _pad field to carry just enough data (FAIL's stage) to recompute
   against the job's actual current state at apply time via the real
   apply_job_fail_transition()/new apply_job_postdone_transition()
   guard cores, instead of overwriting with a stale absolute value.
   Backward compatible: _pad==0 (all prior data) routes to the same
   legacy overwrite behavior.

Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).

Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
This commit is contained in:
Rongsong Shen 2026-07-04 19:00:43 +00:00
parent d3da3ca8bd
commit 2fa955b92e
11 changed files with 1687 additions and 135 deletions
Download patch file Download diff file Expand all files Collapse all files

164
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -1860,9 +1860,14 @@ static void test_update_op_rusage_receipt_reply_facts_retained(void)
* entry (payload "p", 1 byte) and any non-jobstate entries are skipped by
* the payload-size check. Single segment is guaranteed here: rotation
* needs 64 MB (WAL_SEGMENT_SIZE_BYTES). Returns the match count;
* *state_out (optional) receives the LAST matching entry's state. */
static int cbrp_count_jobstate_entries(persist_t *p, uint32_t wal_type,
uint64_t jobid, uint32_t *state_out)
* state_out/pad_out (both optional pointers) receive the LAST matching
* entry's state/_pad. raft-apply-decoupling task 6.1 round 2: WAL_JOB_FAIL and a
* postdone WAL_JOB_UPDATE now carry their real information in `_pad` (see
* wal_types.h's doc comment on wal_jobstate_t._pad) rather than a
* precomputed `state` — pad_out lets a test observe that. */
static int cbrp_count_jobstate_entries_ex(persist_t *p, uint32_t wal_type,
uint64_t jobid, uint32_t *state_out,
uint32_t *pad_out)
{
char seg_path[sizeof(p->seg_path)];
uint64_t seg_end, off = 0;
@ -1895,6 +1900,8 @@ static int cbrp_count_jobstate_entries(persist_t *p, uint32_t wal_type,
count++;
if (state_out != NULL)
*state_out = js.state;
if (pad_out != NULL)
*pad_out = js._pad;
}
}
off += sizeof(hdr) + hdr.payload_len;
@ -1903,6 +1910,14 @@ static int cbrp_count_jobstate_entries(persist_t *p, uint32_t wal_type,
return count;
}
/* Thin wrapper preserving every pre-existing caller's signature/contract
* unchanged (pad = 0, "legacy absolute state" — see wal_types.h). */
static int cbrp_count_jobstate_entries(persist_t *p, uint32_t wal_type,
uint64_t jobid, uint32_t *state_out)
{
return cbrp_count_jobstate_entries_ex(p, wal_type, jobid, state_out, NULL);
}
/* Drive one fact call: it must succeed (rc 0 — in particular NEVER
* -ETIMEDOUT) and return latency-bounded. sync_timeout_ms is 5000 in this
* test, so a fact path that wrongly ran a WAIT barrier with ACKs
@ -2372,24 +2387,32 @@ static void test_guard_replay_finish_equals_single_application(void)
/* 5.2 boundary pin — a FORWARD update on a terminal job must NOT be
* over-rejected by the guard. on_job_postdone is the reachable
* record_job_update_fact caller: on a DONE job it OR's JOB_STAT_PDONE
* record_job_postdone_fact caller: on a DONE job it OR's JOB_STAT_PDONE
* (terminal preserved, strictly forward). This must still durably append
* (an entry appends, carrying the merged state) — the guard rejects
* backward, never a forward merge. Preservation pin: green before and after
* task 6.2.
* (an entry appends, marked for the correct apply-time merge) — the guard
* rejects backward, never a forward merge. Preservation pin: green before
* and after task 6.2.
*
* raft-apply-decoupling task 4.1: on_job_postdone is now durable-append-only
* — the oracle for "did the forward merge compute correctly" moves from live
* state to the WAL-recorded fact (cbrp_count_jobstate_entries), mirroring
* test_fact_retention_across_quorum_loss. The live-state-unchanged
* assertion is this task's own new pin (Req 5.1): live state must still show
* only DONE (from cbrp_drive_to_done's direct apply), never PDONE, since
* nothing applies this fact live yet. */
* raft-apply-decoupling task 4.1: on_job_postdone is now durable-append-only.
* task 6.1 round 2 (merge-staleness fix, Req 11.3): the WAL-recorded fact no
* longer carries a precomputed `state` at all — postdone's merge depends on
* whatever state holds at APPLY time (which may differ from receipt time
* once apply is deferred), so it is recomputed then by
* apply_job_postdone_transition(), never baked into the WAL entry. The
* oracle for "was this fact durably recorded correctly" is therefore the
* WAL entry's `_pad` marker (cbrp_count_jobstate_entries_ex); the oracle for
* "does the forward merge itself compute correctly" is a direct call to
* apply_job_postdone_transition() below, mirroring how
* test_guard_fail_after_finish_preserves_perr_merge pins apply_job_fail_
* transition(). The live-state-unchanged assertion is task 4.1's own pin
* (Req 5.1): live state must still show only DONE (from cbrp_drive_to_done's
* direct apply), never PDONE, since nothing applies this fact live yet. */
static void test_guard_forward_update_on_terminal_preserved(void)
{
uint64_t lsn_after_done, lsn_after_pdone;
uint32_t st_disk = 0;
uint32_t pad_disk = 0;
int st_live;
job_t scratch;
cbrp_setup(3000);
cbrp_make_job(73, &g_job_submit);
@ -2404,18 +2427,33 @@ static void test_guard_forward_update_on_terminal_preserved(void)
lsn_after_pdone = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
/* Assert: the update appended, carrying the correctly forward-merged
* state (terminal DONE preserved, PDONE merged in — not rejected). */
/* Assert: the update appended, marked for a correct apply-time forward
* merge (terminal DONE preserved, PDONE merged in — not rejected). */
TEST_ASSERT_TRUE_MESSAGE(lsn_after_pdone > lsn_after_done,
"a forward update on a terminal job must still append its WAL entry "
"— the guard rejects backward transitions, never forward merges");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 73,
&st_disk),
cbrp_count_jobstate_entries_ex(&g_hc.persist, WAL_JOB_UPDATE, 73,
NULL, &pad_disk),
"exactly one UPDATE fact must be durably recorded (Req 5.1)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_DONE) != 0,
TEST_ASSERT_EQUAL_UINT32_MESSAGE(WAL_JOB_UPDATE_POSTDONE_MARK, pad_disk,
"the recorded UPDATE fact must be marked for apply-time postdone "
"recomputation (task 6.1 round 2) — not carry a state precomputed "
"at receipt time, which would go stale under deferred apply");
/* Assert: the actual forward-merge guarantee this test exists to pin —
* DONE preserved, PDONE merged in, not rejected — verified directly
* against the guard core that performs it at apply time, on an
* independent scratch job seeded at the same terminal state job 73 was
* driven to above. */
memset(&scratch, 0, sizeof(scratch));
scratch.job.jobid = 73;
scratch.job.state = JOB_STAT_DONE;
TEST_ASSERT_EQUAL_INT(JOB_TRANSITION_APPLIED,
apply_job_postdone_transition(&scratch));
TEST_ASSERT_TRUE_MESSAGE((scratch.job.state & JOB_STAT_DONE) != 0,
"the recorded UPDATE fact must preserve the terminal bit");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_PDONE) != 0,
TEST_ASSERT_TRUE_MESSAGE((scratch.job.state & JOB_STAT_PDONE) != 0,
"the recorded UPDATE fact must merge JOB_STAT_PDONE (not be "
"rejected)");
@ -2439,18 +2477,27 @@ static void test_guard_forward_update_on_terminal_preserved(void)
* Preservation pin: green before and after task 6.2.
*
* raft-apply-decoupling task 4.1: on_job_fail (via the new
* record_job_fail_fact) is now durable-append-only — it computes the
* PERR-merged target state on a throwaway copy of the job and durably
* appends it, without touching the real job's live state. The oracle for
* "did the merge compute correctly" moves from live state to the
* WAL-recorded fact (cbrp_count_jobstate_entries); the live-state-unchanged
* assertion is this task's own new pin (Req 5.1). */
* record_job_fail_fact) is now durable-append-only. task 6.1 round 2
* (merge-staleness fix, Req 11.3): the WAL-recorded fact no longer carries a
* precomputed target state at all — the PERR merge genuinely depends on
* whatever state holds at APPLY time (a receipt-time snapshot goes stale
* the moment another fact for the same job, e.g. a FINISH, is still pending
* apply — the exact bug this round fixed), so it is recomputed then by the
* REAL apply_job_fail_transition(), never baked into the WAL entry at
* receipt time. The oracle for "was this fact durably recorded correctly"
* is therefore the WAL entry's `_pad`-carried stage
* (cbrp_count_jobstate_entries_ex); the oracle for "does the PERR merge
* itself compute correctly" is a direct call to apply_job_fail_transition()
* below (its own logic is unchanged by this round — only WHEN it runs
* changed). The live-state-unchanged assertion is task 4.1's own pin
* (Req 5.1). */
static void test_guard_fail_after_finish_preserves_perr_merge(void)
{
job_failinfo_t failinfo;
uint64_t lsn_after_done, lsn_after_fail;
uint32_t st_disk = 0;
uint32_t pad_disk = 0;
int st_live;
job_t scratch;
cbrp_setup(3000);
cbrp_make_job(74, &g_job_submit);
@ -2470,19 +2517,36 @@ static void test_guard_fail_after_finish_preserves_perr_merge(void)
lsn_after_fail = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
/* Assert: the FAIL fact appended, carrying the correctly PERR-merged
* state — DONE preserved, PERR merged in, not reset to bare EXIT. */
/* Assert: the FAIL fact appended, marked for a correct apply-time
* PERR-merge — DONE preserved, PERR merged in, not reset to bare EXIT. */
TEST_ASSERT_TRUE_MESSAGE(lsn_after_fail > lsn_after_done,
"the PERR-merge fail must still append its WAL entry (forward "
"terminal→terminal merge — not a guarded-out no-op)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 74,
&st_disk),
cbrp_count_jobstate_entries_ex(&g_hc.persist, WAL_JOB_FAIL, 74,
NULL, &pad_disk),
"exactly one FAIL fact must be durably recorded (Req 5.1)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_DONE) != 0,
TEST_ASSERT_EQUAL_UINT32_MESSAGE(
WAL_JOB_FAIL_STAGE_FLAG | (uint32_t)JOB_EXEC_STAGE_POST_EXEC,
pad_disk,
"the recorded FAIL fact must carry its stage marked for apply-time "
"recomputation (task 6.1 round 2) — not a state precomputed at "
"receipt time, which would go stale under deferred apply");
/* Assert: the actual PERR-merge guarantee this test exists to pin —
* DONE preserved, PERR merged in, not reset to bare EXIT — verified
* directly against the guard core that performs it at apply time, on an
* independent scratch job seeded at the same terminal state job 74 was
* driven to above. */
memset(&scratch, 0, sizeof(scratch));
scratch.job.jobid = 74;
scratch.job.state = JOB_STAT_DONE;
TEST_ASSERT_EQUAL_INT(JOB_TRANSITION_APPLIED,
apply_job_fail_transition(&scratch, &failinfo));
TEST_ASSERT_TRUE_MESSAGE((scratch.job.state & JOB_STAT_DONE) != 0,
"the recorded FAIL fact must keep the DONE terminal bit "
"(PERR-merge behavior)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_PERR) != 0,
TEST_ASSERT_TRUE_MESSAGE((scratch.job.state & JOB_STAT_PERR) != 0,
"the recorded FAIL fact must OR JOB_STAT_PERR onto the existing "
"terminal state (PERR-merge behavior)");
@ -2684,7 +2748,7 @@ static void test_fact_state_unchanged_throughout_quorum_loss_outage(void)
uint64_t lsn_before, lsn_after;
uint64_t committed_at_poll;
int initial_state, poll_state, final_state;
uint32_t st_start = 0, st_fin = 0, st_fail = 0, st_upd = 0;
uint32_t st_start = 0, st_fin = 0, pad_fail = 0, pad_upd = 0;
int cnt_start, cnt_fin, cnt_fail, cnt_upd;
int i;
enum { OUTAGE_POLL_COUNT = 5, OUTAGE_POLL_INTERVAL_MS = 20 };
@ -2790,10 +2854,10 @@ static void test_fact_state_unchanged_throughout_quorum_loss_outage(void)
&st_start);
cnt_fin = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 85,
&st_fin);
cnt_fail = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 85,
&st_fail);
cnt_upd = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 85,
&st_upd);
cnt_fail = cbrp_count_jobstate_entries_ex(&g_hc.persist, WAL_JOB_FAIL, 85,
NULL, &pad_fail);
cnt_upd = cbrp_count_jobstate_entries_ex(&g_hc.persist, WAL_JOB_UPDATE,
85, NULL, &pad_upd);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_start,
"job 85 START fact must remain durably retained exactly once "
@ -2809,14 +2873,20 @@ static void test_fact_state_unchanged_throughout_quorum_loss_outage(void)
"throughout the whole outage window (Req 6.1)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_start);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_fin);
TEST_ASSERT_TRUE_MESSAGE((st_upd & JOB_STAT_PDONE) != 0,
"the recorded UPDATE fact must carry JOB_STAT_PDONE");
TEST_ASSERT_TRUE_MESSAGE((st_upd & JOB_STAT_EXIT) != 0,
"the recorded UPDATE fact must carry the backstop JOB_STAT_EXIT "
"bit postdone computes when no prior exit status was set");
(void)st_fail; /* FAIL's merge result depends on current state at the
* (never-reached) apply step; only its durable
* retention count is this test's concern. */
/* task 6.1 round 2 (Req 11.3): FAIL's and postdone's UPDATE's merge
* results depend on current state at the (never-reached, in this
* outage-only test) apply step — they are no longer precomputed at
* receipt time at all (the merge-staleness fix), so only their durable
* retention count and their apply-time-recomputation marker are this
* test's concern. */
TEST_ASSERT_EQUAL_UINT32_MESSAGE(
WAL_JOB_FAIL_STAGE_FLAG | (uint32_t)JOB_EXEC_STAGE_POST_EXEC,
pad_fail,
"the recorded FAIL fact must carry its stage marked for apply-time "
"recomputation");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(WAL_JOB_UPDATE_POSTDONE_MARK, pad_upd,
"the recorded UPDATE fact must be marked for apply-time postdone "
"recomputation");
/* Final check: after the ENTIRE outage window, job state is STILL
* unchanged and no reaction has fired — sealing the "for the duration