forked from lei.chen/myxce
fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak): a client's commit-timeout retry during a genuine quorum-loss window appends a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never retracted, so if it later achieves majority commit anyway both entries end up committed. This is a legitimate, harmless duplicate — restore_job()'s own -EEXIST correctly no-ops the second one — but replay_wal() tallied every raw WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate made new_job_submits disagree with jobmgr's real growth, firing a false APPLY_DIVERGENT and deadlocking promotion. Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release, claim_committed_job_submits, and their wiring): three attempts at preventing the duplicate at submit/dispatch/commit time all failed to close the gap, because the duplicate isn't a bug to prevent — a WAL entry can't be retracted once appended, so any submit-time guard is structurally the wrong layer. Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected apply-seam/reapply-idempotency tests for the new return contract, and reverses two apply-correctness tests whose premise (duplicate-jobid replay must diverge) is now understood to be the actual bug rather than the expected behavior — they now force divergence directly via the same production fencing seam instead of relying on that trigger. Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job loss, no election stall (previously reproduced within ~5 rounds).
This commit is contained in:
parent
f28ce57993
commit
11244dd654
7 changed files with 197 additions and 232 deletions
|
|
@ -155,41 +155,6 @@ static void inflight_range_remove(uint64_t base_jobid, uint64_t array_size)
|
|||
pthread_mutex_unlock(&s_inflight_submits_lock);
|
||||
}
|
||||
|
||||
/* bug-1419 layer 3 (cross-primary retry gap): apply_committed_entry()
|
||||
* (persist.c) calls this around its WAL_JOB_SUBMIT dispatch so the jobid is
|
||||
* visible in s_inflight_submits for the whole duration of apply, regardless
|
||||
* of whether this node originated the submission (handle_jobsubmit already
|
||||
* holds an overlapping claim in that case -- see the false-return note
|
||||
* below) or received it via replication/replay (no local claim exists yet
|
||||
* without this call). A resend of an already-committed-but-not-yet-applied
|
||||
* jobid landing on this node -- e.g. a client retry reaching a newly
|
||||
* elected primary after the old one committed the entry but died before
|
||||
* acking -- is rejected by inflight_range_try_add here exactly as it would
|
||||
* be for a same-node in-flight submission, closing the window
|
||||
* jobid_range_exists()'s c->jobmgr-only check cannot see.
|
||||
*
|
||||
* Single-element claim (array_size=1): a WAL_JOB_SUBMIT entry always names
|
||||
* one individual jobid (build_job_submit_payloads emits one entry per array
|
||||
* element), never a range. Returns true iff this call added the entry --
|
||||
* false means an overlapping claim already exists (typically this node's
|
||||
* own handle_jobsubmit-originated range claim covering the same jobid), in
|
||||
* which case the caller must NOT release it; that claim's own submitter
|
||||
* releases it via jobsubmit_async_ctx_release as usual. */
|
||||
bool job_wal_commit_claim(uint64_t jobid)
|
||||
{
|
||||
return inflight_range_try_add(jobid, 1);
|
||||
}
|
||||
|
||||
/* Release a claim job_wal_commit_claim() added. Only call when that call
|
||||
* returned true -- inflight_range_remove only matches an EXACT
|
||||
* base_jobid+array_size pair, so calling it for a claim this function never
|
||||
* added is always a safe no-op, but the caller-side guard avoids a wasted
|
||||
* lock+scan on the common already-claimed-elsewhere path. */
|
||||
void job_wal_commit_release(uint64_t jobid)
|
||||
{
|
||||
inflight_range_remove(jobid, 1);
|
||||
}
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/* Test-only seam: s_inflight_submits is process-lifetime, not tied to any
|
||||
* cluster_t fixture. In production that's fine -- a restart wipes it along
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue