forked from lei.chen/myxce
fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak): a client's commit-timeout retry during a genuine quorum-loss window appends a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never retracted, so if it later achieves majority commit anyway both entries end up committed. This is a legitimate, harmless duplicate — restore_job()'s own -EEXIST correctly no-ops the second one — but replay_wal() tallied every raw WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate made new_job_submits disagree with jobmgr's real growth, firing a false APPLY_DIVERGENT and deadlocking promotion. Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release, claim_committed_job_submits, and their wiring): three attempts at preventing the duplicate at submit/dispatch/commit time all failed to close the gap, because the duplicate isn't a bug to prevent — a WAL entry can't be retracted once appended, so any submit-time guard is structurally the wrong layer. Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected apply-seam/reapply-idempotency tests for the new return contract, and reverses two apply-correctness tests whose premise (duplicate-jobid replay must diverge) is now understood to be the actual bug rather than the expected behavior — they now force divergence directly via the same production fencing seam instead of relying on that trigger. Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job loss, no election stall (previously reproduced within ~5 rounds).
This commit is contained in:
parent
f28ce57993
commit
11244dd654
7 changed files with 197 additions and 232 deletions
|
|
@ -73,21 +73,6 @@ extern int handle_join(cluster_t*, endpoint_t*, xce_request_t*);
|
|||
extern int handle_reply(endpoint_t*, struct Request*, cluster_t*);
|
||||
extern int handle_jobdispatch(cluster_t*, endpoint_t*, xce_request_t*);
|
||||
extern int handle_jobsubmit(cluster_t*, endpoint_t*, xce_request_t*);
|
||||
/* bug-1419 layer 3: claim/release a single jobid in job.c's in-flight
|
||||
* registry from OUTSIDE handle_jobsubmit -- used by apply_committed_entry()
|
||||
* (persist.c) to make a WAL_JOB_SUBMIT entry's jobid visible as
|
||||
* "committed, dispatch in progress" for the whole duration of its apply,
|
||||
* regardless of whether this node originated the submission or received it
|
||||
* via replication/replay. Closes the cross-primary-retry gap: a client
|
||||
* resend landing on a NEW primary after the old one committed the entry but
|
||||
* died before acking sees jobid_range_exists() find nothing (not yet
|
||||
* applied to c->jobmgr) unless this claim is also visible.
|
||||
* job_wal_commit_claim returns true iff IT added the entry (false if
|
||||
* something else -- e.g. this node's own handle_jobsubmit -- already holds
|
||||
* an overlapping claim); only call job_wal_commit_release when the claim
|
||||
* call returned true, so a caller never removes an entry it does not own. */
|
||||
extern bool job_wal_commit_claim(uint64_t jobid);
|
||||
extern void job_wal_commit_release(uint64_t jobid);
|
||||
#ifdef XCE_TESTING
|
||||
/* bug-1419: clears job.c's process-lifetime in-flight-jobid registry.
|
||||
* Tests that deliberately leave a submission's commit continuation
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue