fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap
jobid_range_exists() (c->jobmgr) and the in-flight registry (job.c,
per-process, only populated by this node's own handle_jobsubmit calls)
both miss a real cross-node case: a client's XCECLI_FAILOVER_CALL retry
resends the same client-generated jobid after a connection drop, and the
resend lands on a newly elected primary that never processed the original
request. If the original was already committed before the old primary
died, the new primary's own local WAL already holds it (Raft Leader
Completeness) but jobmgr hasn't applied it yet — both existing guards see
nothing, so the resend appends a second, genuinely committed WAL_JOB_SUBMIT
entry for the same jobid. A live chaotic soak reproduced this exactly
(waldump: two committed entries, same term, matching apiserverId/timestamp/
sequence, one second apart) — apply's own idempotent dedup keeps jobmgr
correct, but replay's raw WAL_JOB_SUBMIT count no longer matches jobmgr's
growth, firing a false APPLY_DIVERGENT fence and deadlocking promotion.
Fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now claims the jobid in
job.c's in-flight registry (new job_wal_commit_claim/release, wrapping the
existing inflight_range_try_add/remove) for the duration of dispatch,
regardless of whether this node originated the submission or is applying a
replicated/replayed entry. A resend landing mid-dispatch on any node now
sees the same -EEXIST rejection self-originated in-flight submits already
get. Release only fires when the claim call added the entry, so a
self-originated submission's own claim/release lifecycle (unchanged) is
never double-released or stolen.
Also fixes 4 pre-existing tests that broke under the earlier checkpoint_lsn
fix (0eb1054) and were missed in that commit's regression sweep: cluster-
less fixtures where last_applied_lsn never advances now correctly get
checkpoint_lsn=0 unless the test simulates apply catch-up, matching the
new (correct) semantics. One assertion in
test_checkpoint_tail_with_no_snapshot_is_apply_clean compared Phase 1's
checkpoint_lsn against Phase 2's independently-reconstructed physical WAL
tip — that equality only held under the pre-fix current_lsn-tracking
behavior and is removed, not patched around.
Also hardens run-ha-soak.sh's start_node() with setsid+nohup: a forked/
background test-harness invocation whose session tears down mid-run was
silently killing every controller process via SIGHUP forwarding (observed
twice — deploy completes, ~1s of activity, then all 3 nodes die together
with no error logged).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
This commit is contained in:
parent
1b6041acef
commit
f28ce57993
10 changed files with 206 additions and 16 deletions
|
|
@ -239,19 +239,33 @@ start_node() {
|
|||
# yama scope 1) instead of via start.sh, so a hang can be interrupted
|
||||
# (SIGINT to gdb's own pid) for a live thread backtrace. Not for normal
|
||||
# soak use -- temporary investigation-only seam.
|
||||
# Forked/background-agent execution note: a plain trailing `&` keeps the
|
||||
# child in the SAME session as this script. If the session that launched
|
||||
# run-ha-soak.sh tears down mid-run (e.g. a forked skill invocation whose
|
||||
# controlling session ends once its own turn returns), the kernel sends
|
||||
# SIGHUP to everything still attached to it -- bash forwards that to
|
||||
# unprotected background jobs by default, killing every node silently
|
||||
# (no error, no log line, just simultaneous death) well before the soak
|
||||
# can finish. setsid detaches into a brand-new session (no controlling
|
||||
# terminal to ever receive a SIGHUP from) and nohup additionally makes
|
||||
# the process ignore SIGHUP outright -- belt and suspenders, since which
|
||||
# exact signal/mechanism a given forked-agent runtime uses to tear down
|
||||
# is not something this script can assume. </dev/null keeps a detached
|
||||
# process from blocking on a stdin that no longer has a reader.
|
||||
if [ -n "${GDB_WRAP:-}" ] && [ "$GDB_WRAP" = "$1" ]; then
|
||||
gdb --batch -ex run -ex "thread apply all bt full" -ex "kill inferior 1" -ex quit \
|
||||
setsid nohup gdb --batch -ex run -ex "thread apply all bt full" -ex "kill inferior 1" -ex quit \
|
||||
--args "$BIN" --myname "$1" --peers "$PEERS" \
|
||||
--workdir "$TOP/controller" --certdir "$TOP/certs" \
|
||||
--logfile "$TOP/logs/$1.log" --vcpus 4 -vvv "${extra_args[@]}" \
|
||||
>>"$TOP/logs/$1.gdbwrap" 2>&1 &
|
||||
</dev/null >>"$TOP/logs/$1.gdbwrap" 2>&1 &
|
||||
return
|
||||
fi
|
||||
if [ ${#extra_args[@]} -gt 0 ]; then
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv "${extra_args[@]}" \
|
||||
>>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
setsid nohup bash "$TOP/scripts/start.sh" "$1" -vvv "${extra_args[@]}" \
|
||||
</dev/null >>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
else
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv >>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
setsid nohup bash "$TOP/scripts/start.sh" "$1" -vvv \
|
||||
</dev/null >>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
fi
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue