lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard

Browse source 
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.

Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.

Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.

Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
This commit is contained in:
Rongsong Shen 2026-07-08 12:48:44 +00:00
parent f28ce57993
commit 11244dd654
7 changed files with 197 additions and 232 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/controller/testsuite/test_apply_reapply_idempotency.c
View file

@ -203,9 +203,10 @@ static void test_tari_job_submit_reapplied_thrice_is_idempotent(void)
tari_spy_hook, NULL));
/* First application: real mutation + one notification. */
TEST_ASSERT_EQUAL_INT(0, apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT,
100, payload,
(uint32_t)plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT, 100, payload,
(uint32_t)plen),
"a genuine new job creation must return 1 (bug-1419)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_tari_hook_fires,
"the first application must fire exactly one notification");
@ -218,9 +219,10 @@ static void test_tari_job_submit_reapplied_thrice_is_idempotent(void)
* observable state must not change again, and the notification must not
* fire again, on EITHER repeat. */
for (int i = 0; i < 2; i++) {
TEST_ASSERT_EQUAL_INT(0,
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT, 100, payload,
(uint32_t)plen));
(uint32_t)plen),
"an idempotent duplicate must return 0, not 1 (bug-1419)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_tari_hook_fires,
"re-applying an identical committed SUBMIT must never fire a "
"second notification, no matter how many times it repeats");
@ -313,9 +315,10 @@ test_tari_finish_then_stale_start_preserves_terminal_state_via_dispatch(void)
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tari_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0, apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT,
300, payload,
(uint32_t)plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT, 300, payload,
(uint32_t)plen),
"a genuine new job creation must return 1 (bug-1419)");
free(payload);
job_t* j = tari_find_job(703);