lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard

Browse source 
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.

Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.

Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.

Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
This commit is contained in:
Rongsong Shen 2026-07-08 12:48:44 +00:00
parent f28ce57993
commit 11244dd654
7 changed files with 197 additions and 232 deletions
Download patch file Download diff file Expand all files Collapse all files

42
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -1381,47 +1381,6 @@ static void test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job
" PASS\n");
}
/* bug-1419 layer 3: a jobid claimed by something OTHER than this node's own
* handle_jobsubmit call -- simulating the apply pump processing a
* WAL_JOB_SUBMIT entry received via replication/replay, never locally
* submitted -- must still block a same-jobid submit arriving through
* handle_jobsubmit. Before this fix, job_wal_commit_claim did not exist and
* apply_committed_entry() never registered an in-flight claim for an
* externally-sourced entry, so jobid_range_exists() (c->jobmgr, not yet
* applied) and the in-flight registry (only ever populated by
* handle_jobsubmit itself) both missed it -- exactly the cross-primary
* retry gap a live chaotic soak reproduced (waldump showed two committed
* WAL_JOB_SUBMIT entries for one jobid, same term, no fresh idgen collision
* -- the same generated jobid resent by the client after a connection drop,
* landing on a newly elected primary that never saw the original). */
static void test_jobsubmit_rejected_when_jobid_claimed_by_foreign_commit(void)
{
const uint64_t jobid = 9160;
int rc;
cbrp_setup(2000);
cbrp_arrange_jobsubmit(jobid, 1);
/* Simulate: the apply pump is mid-dispatch on a WAL_JOB_SUBMIT entry for
* this jobid that this node never itself submitted (e.g. replicated
* from a prior primary that committed it before dying). */
TEST_ASSERT_TRUE_MESSAGE(job_wal_commit_claim(jobid),
"fixture sanity: claiming a fresh jobid must succeed");
rc = handle_jobsubmit(&g_hc, &g_req_ep, &g_jobsub_req);
job_wal_commit_release(jobid);
cbrp_teardown();
TEST_ASSERT_EQUAL_INT_MESSAGE(-EEXIST, rc,
"bug-1419 layer 3: a submit for a jobid claimed by a foreign "
"(non-self-originated) commit must be rejected -EEXIST, closing "
"the cross-primary retry gap");
printf("test_jobsubmit_rejected_when_jobid_claimed_by_foreign_commit:"
" PASS\n");
}
/* The explicit-FAILED (non-timeout) sibling of the repeated-timeout test
* above: repeats the truncation-forced FAILED path
* CBRP_JOBSUBMIT_REPEAT_N times in the same fixture, proving the
@ -3388,7 +3347,6 @@ void test_commit_barrier_reply(void)
RUN_TEST(test_jobsubmit_repeated_timeout_failures_no_phantom_jobs);
RUN_TEST(test_jobsubmit_resubmit_before_resolution_rejected_no_dup_wal_entry);
RUN_TEST(test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job);
RUN_TEST(test_jobsubmit_rejected_when_jobid_claimed_by_foreign_commit);
RUN_TEST(test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs);
RUN_TEST(test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements);
/* handle_jobsubmit success path via the ordered apply pump