Queue's (name, status) round-trip through write_snapshot's raw on-disk
bytes -- a plain fixed record (no capnp; queue definitions are
config-owned, only runtime admin status is WAL/snapshot-tracked, per
design.md's corrected queue model). Fails today (Expected 4 Was 0) since
write_snapshot still hardcodes queue_count=0 from task 2.2's placeholder.
Does not call persist_load_snapshot, following task 2.2's established
raw-byte-parsing pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot now Cap'n-Proto-encodes every retained job under nested
table+per-entity RTTI_LOCK_READ (table lock alone does not exclude a
concurrent per-entity writer -- design.md's central correctness finding),
appends length-prefixed job frames after a new format_version marker, and
fixes a previously-unchecked fdatasync return. Any failure anywhere in the
path (encode, write, sync) discards the whole checkpoint -- no partial
file at the final path, proven by a real mid-encode-loop fault-injection
test (forces one specific job's encode to fail among N, asserts nothing
survives).
Also fixed two pre-existing bugs in task 1.1's test fixture: terminal-state
seeded jobs need a non-zero finish_time or job_should_retain's retention
window silently excludes them from any checkpoint; the fixture needs
xstring_table_init or every seeded job's submit/start_info silently
decodes as NULL.
Round 1 of this task also touched load_snapshot/persist.h to make the
round-trip tests pass end-to-end, but review caught a live hazard: it
advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran, so
a decode failure still left the position advanced -- reachable today
through the unmodified ha_handle_snapshot_chunk production path, not
hypothetical. Remediated by fully reverting load_snapshot/persist.h to
byte-identical pre-task-2.2 state and rewriting the round-trip tests to
parse write_snapshot's raw on-disk bytes directly (read_Job/decode_Job)
instead of round-tripping through persist_load_snapshot -- proves the
write side without depending on load_snapshot, which stays task 5.2's
scope entirely, not started here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Three tests against the current sentinel-only write_snapshot/load_snapshot:
job content round-trip and concurrent-mutation-not-torn genuinely FAIL
(RED) since no entity content is written today; write-failure-discard
already passes (write_snapshot's open()-failure path already discards
correctly) -- flagged transparently as a GREEN-from-day-one regression
guard rather than faked RED, per real fault injection (directory removal
-> genuine ENOENT).
Carries forward two notes for task 2.2: fdatasync's return value is
currently unchecked in write_snapshot (persist.c:1505), and a mid-encode-
loop fault-injection test should be added once the per-job encode loop
exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds the ha-entity-snapshots Kiro spec (requirements, design, research,
tasks) for task 5.10a / raft-ha-core-safety: making the HA checkpoint an
actually-restorable copy of committed job/queue state instead of a
position-only sentinel, closing the residual chaotic-soak divergence gap.
Task 1.1 (foundation): reusable test fixture snapent_make/snapent_teardown
in a new test_persist_snapshot_entities.c, seeding N jobs (varying state)
and M queues (varying status) into a real persist_t+cluster_t for later
write/load/rotation-guard tests to build on. Build clean; new suite 1/1;
regression spot-checks (test_persist_hooks 19/19, test_commit_barrier_reply
22/22) clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A dead primary's control fd RDHUPs faster than an election settles. The 5.6
guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/
prevote; an RDHUP landing in the gap after a lost election (role briefly back
to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real
election (term++). One bump per RDHUP -> unbounded escalation under a sustained
cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the
harness window).
Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover;
ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the
last trigger (closes the dead fd, returns) so the election's own jittered retry
(5.7) runs without the external RDHUP cadence adding term bumps. First trigger
never debounced; genuine loss still fails over, at most once per window.
Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent
last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary
(last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect,
terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no
regression. The stochastic storm did not recur this run (debounce fired 0x), so
soak-reproduction of the storm is still pending; the mechanism rests on the
unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0,
election 28/0, failover_prevote 6/0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
The round-4 term ladder is driven by repeated RDHUP->begin_failover->
PreVote-won->real-election(term++) cycles from the dead primary's socket, not
a behind-node vote loop. 5.6's instantaneous 'suppress while electing' guard is
insufficient under a sustained RDHUP cadence. Fix direction: per-failover
cooldown/debounce + hold the failover latch across the full prevote->election->
settle window. Needs a deterministic dead-primary-cadence repro.
freshness-deny escalation loop under a just-restarted-survivor overlap: a
behind-log candidate is freshness-denied every term (18->24 observed), times
out, re-campaigns at term+1 -> ~18 escalations before convergence. NOT RNG
(seed is /dev/urandom); 5.7 jitter bounds retry rate, not the escalation.
Fix direction: a repeatedly-freshness-denied candidate must recognize it is
behind and defer/catch-up instead of laddering the cluster term. Needs a
deterministic 2-survivor repro before a fix (5.5/5.6/5.7 family).
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).
- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
step (WAL append + majority-commit barrier) returning the commit verdict
directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
reject on <0 before firing the now notification-only AFTER hook
(canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
commit_barrier_reply 22/0, commit_durability 38/0.
Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
10-round soak waldump+log bisect: the chaotic committed_job_loss is NOT the
catch-up/entity-snapshot prefix gap. All 3 WALs gapless/CRC-intact/full
payloads; lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority.
The durable-majority gate worked (commit_submit -> -ETIMEDOUT), but
xce_run_hooks (hook.c:134-157) returns only the LAST-registered hook's rc, so a
later JOB AFTER_EVENT hook (sched/apiserver) returning 0 clobbered persist's
-ETIMEDOUT -> handle_jobsubmit replied jobsubmit ok for an uncommitted entry ->
term-10 leader legally overwrote it. Fix per raft-apply-decoupling item #1:
move the WAL append + commit barrier out of the hook into an explicit
handle_jobsubmit-driven raft step whose rc is read directly (down payment on
the full apply-after-commit inversion). Entity-snapshot gap is a separate,
still-real defect (not this soak's mechanism).
5.8 endpoint refcount fix validated: 182 unit tests/0 fail, 6-round ASAN chaotic
soak ASan-clean, all rounds elect. bug-1298 marked fixed.
The soak now reaches rounds 4-6 for the first time (every prior run crashed at
round 4) and FAILs the verdict on committed_job_loss=4/divergence=7. Evidence shows
this is the pre-existing bug-795/828/851 failover catch-up gap, NOT a regression:
one job missing from round 4 onward across 3 primaries; rounds 1-3 clean; ZERO
WAL/commit/catch-up code changed this session; ASAN clean. Routed to task 5.9 /
ha-commit-durability.
5.7 election-timeout resonance FIXED + soak-confirmed: 6-round chaotic soak now
elects with bounded terms (rounds 1-3 at terms 4/5/6, not 25-30+), zero job loss.
The election-liveness layer is done.
Round-4 stall re-roots to a NON-election crash: ctrl-3 ASAN heap-use-after-free
in endpoint teardown — io_worker (endpoint_killer, cluster.c:813) frees an endpoint
while the vtask dispatcher reads ep->busy (endpoint_is_task_blocked, netio.c:1008).
ctrl-3 dies -> cluster drops to 1/3 -> no quorum -> no primary. Same async-io
dispatch surface as bug-1282/1197; UPSTREAM-owned (raft-drive-async / libxce).
Logged bug-1298; tracked as task 5.8.
5.6's RDHUP-no-preempt guard (81cb09a) is unit-verified and slows the round-4 term
churn, but the soak still stalls — a 5th layer (task 5.7). The freshest survivor
ctrl-2 (15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it
(vote_granted) — but ctrl-2 never tallies a majority: every grant arrives as
vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term
escalates via repeated election timeouts faster than ctrl-3 (cold peer
connection, fresh SSL connect per vote, plus dead-ctrl-1 connect timeouts) can
reply, so each grant is one term stale -> dropped -> timeout -> term++.
Self-perpetuating; DropStaleResponse itself is correct Raft.
Adds task 5.7 (don't re-escalate the term while a vote RTT is outstanding /
warm-reuse the peer vote connection / wider jitter). 5 layers of the round-4
stall now peeled, 4 fixed (deadlock, segfault, advertised-term, RDHUP-preempt);
the cold-connection term-lag remains. Consensus SAFETY intact throughout.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
5.5's real root cause was a stale advertised lastLogTerm (c->persist.term vs
persist_entry_term), not the last_activity hypothesis — fixed in b8453b2 and
VERIFIED: with it, the freshest survivor now wins PreVote and enters the real
election (it could not before). But round 4 still does not complete: a deeper
layer (task 5.6) — under the dead-primary RDHUP cadence (~15s) the two survivors
duel in lockstep and the term escalates unbounded (21->25->28->29...) without
reaching majority. Adds task 5.6 (randomized failover backoff to desync the duel
/ don't re-trigger failover mid-real-election). Consensus SAFETY intact; this is
election LIVENESS under sustained churn. 4 layers of the round-4 stall now peeled
(deadlock, segfault, advertised-term fixed; dueling-candidate remains).
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
6-round chaotic soak with both fixes: rounds 1-3 elect; round 4 has NO segfault,
NO fdatasync, NO iomgr deadlock — both fixes held. The round-4 stall is now the
pure election-liveness cross-deny that the crash/deadlock masked: the freshest
survivor (floor 15) is the legitimate winner, but the other survivor (floor 12)
denies it on eligibility because it still believes its dead primary is alive
(in_catch_up + heartbeat-not-yet-stale). Suspect 5.2's last_activity re-arm
keeping a node falsely-fresh after primary death.
Adds task 5.5 (release the heartbeat-fresh/in_catch_up deny on genuine primary
death within the failover window) as the remaining 7.2 blocker. bug-1282 and
bug-1286 are fixed and verified; consensus safety unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
bug-1282 (ABBA iomgr deadlock) is FIXED and verified (commit 25e741d): zero
__lll_lock_wait threads post-fix, unit suite green. But a plain release soak WITH
the fix still segfaults a restarted standby at round 4 — and deadlocks hang, they
do not SIGSEGV, so that crash is a SEPARATE bug (bug-1283), most likely an
endpoint-lifetime UAF under chaotic restart (bug-1197 family).
bug-1283 is a heisenbug (invisible under ASAN and gdb; no usable core in the
sandbox) and likely UPSTREAM raft-drive-async / core libxce endpoint quiescence.
7.2 stays blocked on it. Consensus correctness (Phase-2/5.2/5.3) is unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Using gdb (SIGQUIT all-thread backtrace on the wedged nodes), nailed the round-4
blocker to a deterministic ABBA lock-order deadlock in src/lib async dispatch —
identical on both wedged survivors:
- io_worker (netio.c:672): holds ep->lock, blocks on q->lock (vtask_enq->dataq_enq)
- vdispatcher dataq_deq (dataq.c:65): holds q->lock, blocks on ep->lock via the
vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) cb
io_worker wedges -> node stops accepting/answering PreVotes -> two restarted
survivors can't elect -> round-4 stall. The release SIGSEGV is the same ep/q race
corrupting under churn.
Defect: dataq_deq runs its match callback while holding q->lock, and the callback
takes ep->lock — inverting io_worker's ep->lock->q->lock order. Minimal fix: make
ep->busy atomic + lockless in endpoint_is_task_blocked (or io_worker drops ep->lock
before dataq_enq, or one global lock order).
Ownership: src/lib async-I/O dispatch (vtaskmgr/dataq/netio) = raft-drive-async /
core libxce, a Non-Goal of this spec. Per upstream-ownership protocol, routed to
raft-drive-async (bug-1282); 7.2 stays blocked until the dispatch lock-order is
repaired, then re-run the gate. The gate worked as designed — 7.2 must be GREEN
before raft-drive-async flips dispatch flags ON, and it caught a real dispatch
deadlock. Task 5.4 (root-cause + assign ownership) is DONE. Consensus correctness
(Phase-2/5.2/5.3) is unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Used the now-available gdb (wrapped the release controller under gdb -batch via
XCE_HA_BIN) to expose the round-4 blocker's second face. Findings:
- The release-build SIGSEGV and a gdb-run accept-loop HANG are the SAME restart-
time I/O-subsystem corruption: under gdb the memory bug is perturbed away from
a crash and instead wedges the inbound-accept path (ctrl-3 stops accepting at
03:42:03 while still firing its own election timer), so two restarted survivors
can never exchange PreVotes -> permanent round-N stall.
- This is a restart-STABILITY bug, NOT a consensus-logic gap: PreVote/eligibility/
freshness (incl. 5.3) are correct; the restarted node is simply I/O-dead. It
predates this session (baseline 21e26a1 failed round 3-4).
- Heisenbug: invisible under ASAN and under gdb; no usable core (read-only
core_pattern -> non-running systemd-coredump). Needs a non-ASAN DEBUG (-O0 -g)
build under gdb, or a host with a live coredump pipeline, to backtrace.
Updates research.md, refines task 5.4 (the unified restart-stability bug +
the debug-build path), updates bug-1281. Consensus correctness (Phase-2/5.2/5.3)
is unaffected and validated for every round that completes (2-round soak PASSES;
6-round rounds 1-3 elect; zero job loss; committed prefix byte-identical).
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.
Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
after being restarted into the churning cluster (recovery->failover/PreVote),
dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
debug+core+gdb env.
Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
5.3 fix (bf88681) lands and is unit-validated (test_ha_core_safety 20 Tests /
0 Failures). Re-ran the 7.2 chaotic soak with all Phase-2+5.2+5.3 fixes: it
still stalls at round 4, but bisecting the frozen logs shows the cause is the
bug-924 WAL-fdatasync wedge, NOT consensus — a survivor's persist flusher spins
on `fdatasync FAILED (errno=22 EINVAL)` and stops answering PreVotes, so the
round times out (presents as an election stall, is a durability-layer wedge).
A file-level repro of the controller's exact WAL pattern (open + posix_fallocate
64MB + write + fdatasync) SUCCEEDS on every available ext4/tmpfs path, so it is
the live overlay-backed WAL fd under chaotic restart — the documented bug-924,
a Non-Goal of this spec (owned by ha-commit-durability, excluded from the
verdict).
Re-attributes the round-4 stall (the first §"Task 7.2" note guessed in_catch_up;
5.3 fixes that real latent eligibility deadlock at unit level but it is not the
soak blocker). 7.2 is now BLOCKED ON ENVIRONMENT, not a consensus NO-GO; the
consensus work this spec owns (Phase-2 payload integrity, 5.2, 5.3) is
unit-complete. research.md §"Task 5.3 + 7.2 re-run"; bug-1272 updated.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
First acceptance chaotic soak (6 rounds, real-fs ext4, release binaries rebuilt
at HEAD 01419e1 with Phase-2 payload fix + Phase-5 5.2 suppression). Outcome:
PASS (job safety + landed fixes):
- Zero committed-job loss (all tracked jobs present on the new primary each round).
- Committed prefix [1..11] byte-identical across all 3 members — Phase-2 payload-loss
signature absent. All counted walcompare divergences are present=2 catch-up-LAG
MISSING tails of restarted-uncaught followers (tips 19/15/11), not corruption.
- Rounds 1-3 elect a strictly higher-term primary; 5.2 spurious-failover suppression
confirmed working (round-4 genuine-loss takes the failover path, not suppression).
NO-GO (R8.3 liveness):
- Round-4 election STALL after 3 overlapping kills: the up-to-date leader (tip 19)
is killed, both survivors are lagging (15, 11) but each holds the full committed
prefix; their PreVote rounds repeatedly time out with no grant quorum, so no
primary is elected. Distinct from the 5.2 flap. Contributors: freshness
mutual-denial, in_catch_up-denies-all-votes (Req 2.5, obs 1240) with no live
primary to catch up from, and/or unestablished peer vote connections.
Per the chaotic-harness contingency: recorded full classification in research.md
§"Task 7.2"; added task 5.3 (lagging-but-safe survivor must be electable under
overlapping churn); 7.2 marked _Blocked_ on 5.3; logged bug-1272. Frozen evidence
at /home/claude/tmp/ha-7p2-frozen.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
A recovered standby whose control channel flaps (transient RDHUP) while a
current-term primary is alive treated each flap as "primary died" and launched
an unwinnable PreVote failover (peers see the live primary and deny, the round
times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun
forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5").
Add a freshness guard at the RDHUP failover-initiation path
(ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh,
stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the
primary's reconnect instead of campaigning — the same guard
ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine
loss (stale heartbeat) falls through to failover unchanged; no term bump / no
role change, so Election Safety is untouched.
Keeping the standby attached lets catch-up complete, which clears in_catch_up
and lets votes flow again — so this one guard also resolves the downstream
catching-up-node vote-deny deadlock (obs 1240).
Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures
(spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 →
1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover
root cause (research.md §"Phase 4/5"):
- CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a
control-channel flap (transient RDHUP) while a current-term primary is alive
(fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline
(Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup
-> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no
ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path.
GREEN when task 5.2 adds that guard.
- LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale
heartbeat) still launches a real failover; a failed round re-arms; rapid
rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3).
- SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest
the same term; the real handle_request_vote grants at most one vote per term
(A=3/B=0), so at most one committing primary per term. Falsifiable Election
Safety check mapped to raft/raft.tla (Req 4.1/4.3).
Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean
ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the
intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
5.2: clean failover STRICT_TERM verified — committed prefix byte-identical
(term,pcrc) across all 3 members; standbys persist non-zero term;
0 committed-job loss.
5.4: chaotic 3-round soak — zero job loss; committed prefix consistent
at every walcompare checkpoint; all 3 nodes led; round-3 election
stall is pre-existing bug-851/795 family (NOT regression).
5.5: build+unit suite green (19 groups 0 failures, ASAN clean).
Ha-term-safety: ALL 21 tasks complete. Spec done.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase-4/5 investigation found the live stall is a recovered standby with a
divergent uncommitted tail (former-primary shape) that never reconverges:
loops on ae_reject_logmismatch, its control channel flaps ~15s, each flap
spawns an unwinnable PreVote failover while a live primary exists.
- Phase 4 (Standby Reconvergence): lead with divergent-tail truncate+catch-up
(make the AE backtrack + follower-conflict-truncate fire for a former-primary
tail, not just a short suffix), keep the applied/log_len healthy gate.
- Phase 5 (Election Liveness): retarget from 'missing election-restart' (retry
works) to spurious-failover suppression — reconnect+catch-up on transient
RDHUP, don't storm unwinnable PreVotes while a current-term primary is
reachable. 5.2 now _Depends 4.2_.
Post-Phase-2 chaotic wedge root-caused from frozen soak logs: primary stable
at term 6, but recovered standby ctrl-2 (divergent uncommitted tail 12-15)
cannot reconverge; its primary control channel RDHUP-flaps ~15s, each flap
spawns an unwinnable PreVote failover (peers see primary alive, no grant) ->
timeout -> reconnect -> repeat, so the cluster never reaches stable 3-of-3.
Fix spans Phase 4 (truncate divergent tail + catch up like ctrl-3 does) and
Phase 5 (don't launch unwinnable failover storm on transient RDHUP while a
current-term primary is reachable; reconnect+catch-up first). RED direction
recorded. (Dispatched investigator hung; reconstructed by parent.)
Rebuilt release at HEAD (2.2+2.4), chaotic soak on real ext4. Universally-
held committed prefix (1..11) content-identical across all members; the
present-but-empty committed divergence (bug-1.3 signature) is ABSENT. The
fix eliminates the cross-term payload-loss divergence. Remaining issues:
chaotic soak wedges on election stall (Phase 5) before its verdict; tail
MISSING = catch-up lag (Phase 4). Clean soak verdict gated on Phase 5/4,
not Phase 3 (snapshot, off the critical path).
Root-cause source fix: ha_stream_wal_to_standby reads the entry body via
stream_read_entry_body (bounded retry max 4, no sleep); on persistent short
pread/malloc-fail it fails loud + ABORTS the stream (flush valid prefix,
drop unreadable entry, -EIO) instead of shipping PLEN=0. Incomplete stream
re-driven by ACK-driven catch-up. With the 2.2 guard this closes the live
cross-term payload-loss divergence. 2.3 CORE RED -> GREEN (fail-loud: no
empty persisted; + retry-recover test ships full 64). Term-blind structural
retirement deferred to 6.2 (empty-over-full safety enforced by 2.2 guard +
pinned). 12/12 core suite; siblings green; no regression introduced.
Phase 2 complete.
Minimal XCE_TESTING fault-injection seam in ha_stream_wal_to_standby
(NULL/production-inert) forces a short body read on a cross-term entry.
CORE RED proves the streamer ships PLEN=0 for a full (source_len=64)
cross-term entry: instrumentation source_len=64 SEND_len=0 RECV_len=0 ->
loss is SEND-side (answers task 1.3's open send-vs-transitive question).
2 honest pins (term-blind route guarded by 2.2; below-tip fence). 11
tests/1 RED; production xce_controller links clean without the seam.
Case (a) empty-then-full = genuine RED (payload-blind guard at persist.c:822
no-ops the corrective write, Expected 64 Was 0). Cases (b) full-then-empty
and (c) identical-rewrite = baseline-passing pins that constrain the 2.2
fix asymmetry (empty must never overwrite stored full; identical re-stream
stays idempotent). 7 tests/1 RED/no ASAN.
Release build block-buffers --logfile (flush on signal only), so log-grep
primary/convergence detection false-aborted. Replace all log-grep detection
with client probes (xce_test ctrlstatus): probe_primary/node_probe/
cluster_committed_converged. Both modes use shared helpers; 1.2 verdict
self-test 17/17 intact; bug-854 isolation preserved. Live release-binary
runs reach the verdict (chaotic re-confirms committed_divergence=3 / job
loss 0; sequential PASS). No production code touched. Phase 1 complete.
New test_ha_core_safety.c with deterministic gate/probe seams (no sleeps,
drain-before-fini per bug-1224): payload-controllable WAL writer, on-disk
(term,pcrc,len) readback probe, walcompare observer, apply-gap + kill/
election injection for phases 2.3/4.1/5.1. Smoke-RED (Expected 64 Was 0)
proves the payload-blind (term,lsn)-only guard at persist.c:822 drops a
corrective full-payload write at a committed LSN. 5 tests/1 RED/4 seam
PASS, no ASAN errors.
Reproduced committed divergence at LSN 9 (real-fs release soak): all 3
members term=3, minority ctrl-1 pcrc=0/PLEN=0 vs majority PLEN=3560 =
PAYLOAD loss, term correct. Bisect: arrived via the reconcile seam under
cross-term catch-up (stored term 3, leader term 4 / bug-1067 path), NOT
the term-blind bypass. Real defect = cross-term catch-up payload loss in
ha_stream_wal_to_standby (silent empty ship on short pread/malloc-fail).
Independently verified against frozen WALs. Secondary: release-build log
buffering breaks soak primary-detection.
Per-member (lsn,term,pcrc,present) report + committed-vs-uncommitted
divergence classification via walcompare_report()/--report. Soak signal
(OK/exit-code) preserved. 4 new unit tests, all GREEN.
New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.
Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Verdict: raft-drive-async introduces NO new regression.
- Unit: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites green;
dispatch/snapshot/configpush flags default OFF, inert).
- Live kill-based failover soak on a real fs (~/tmp), release + ASAN builds:
zero committed-JOB loss every completed round (jobs 3->12 monotonic).
- The 6-round CHAOTIC soak does not run clean to round 6 (fails ~round 4), but a
baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4
IDENTICALLY (election stall / reconverge). The round-4 committed-prefix
divergence at LSN 9 is the documented pre-existing bug-851/795 family
(chaotic-failover silent divergence), present before Phase 3 and out of scope
for this spec. Logged as bug-1225 (baseline-confirmed).
Also: make the xce-test-ha soak driver's controller/xce_test paths
env-overridable (XCE_HA_BIN / XCE_HA_XT, backward-compatible defaults) — used to
run the baseline binaries through the same driver for the regression comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
With the dispatch flags flipped ON in-test, prove the three receiver-side
liveness properties (production defaults stay OFF):
- P1: while a snapshot (and config-push) transfer is parked in flight on the
slow lane, another endpoint's peer-control message is still processed
promptly on the vtask pool (different thread), with the transfer still
parked — the offloaded transfer does not block the drive path.
- P2: a peer-control ack (classified VTASK_LIVENESS) runs promptly even while
the pool is saturated to the LivenessAdmission reservation boundary with
REQ_BLOCK tasks (one more REQ_BLOCK fast-rejects with -E_XCE_NOT_READY,
proving real saturation). Narrowed to the liveness-dispatch-under-saturation
assertion; commit-advance itself is covered by the commit-engine tests.
- P3: while one endpoint's peer-control body is parked (holding a worker + that
ep busy), another endpoint's body runs concurrently on a different worker —
per-endpoint FIFO does not serialize across endpoints.
All deterministic (cond-var gates, bounded polls). Fixed a flaky teardown-race
hang (bug-1224): the tests released a parked vtask body then immediately called
the cancel-based vtaskmgr_fini while the just-released body was still unwinding
(P2's REQ_BLOCK bodies racing blk_lock teardown the worst). Fix adds end-of-body
completion counters polled to the full expected count before fini, so every
pool body has fully returned (worker idle) before the cancel. slowlane_fini is
graceful (join) so the lane side was already safe.
Verified: 5/5 deterministic isolated runs; full controller suite EXIT=0;
ASan-clean. Completes task 3.4 (3.4a UAF fix + 3.4b tests); the production
flag-flip remains deferred to after the 4.1 partition soak + 4.2 regression
sweep.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).
Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):
- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
FIFO makes it run strictly after any in-flight/queued same-ep peer-control
vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
run_task's post-body endpoint_unblock_tasks (which would touch the freed
ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.
- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
until dispatch_inflight drains to 0. So every dispatch that read the flag
false finishes its enqueue before mark returns (FIFO-before the teardown
vtask); any later dispatch reads the flag set and drops the packet. ep->lock
is deliberately NOT held across the enqueue (it inverts with the taskq lock in
vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
held window is just a calloc + non-blocking enqueue on the single io_worker.
Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.
Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
push_config_transport_only spawns one thread per standby then joins them with a
CONF_PUSH_TIMEOUT_MS bounded wait — blocking the caller. On the requester-less
drive-thread paths (promotion + heartbeat convergence) that wait stalls the
io_worker. Route those through ha_config_push_offloadable, gated by
g_drive_async_configpush_offload (default OFF → inline, byte-for-byte legacy;
ON → slow lane, non-blocking).
Flag ON deep-copies the config bytes + sha into a lane ctx (caller keeps owning
and freeing its own buffer); a single cluster-wide single-flight guard
(ha_state_t.configpush_inflight scalar CAS + lane key=1) bounds it to one
config-push fan-out in flight; the in-flight flag is cleared on every terminal
path (two alloc failures, -EEXIST, hard submit error, lane completion). Send
body factored into ha_config_push_do_send.
Three requester-less drive-thread call sites converted: raft_cb_become_leader
(promotion), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY
FSM promotion branch. The admin-upgrade barrier (push_config_to_standbys, has a
waiting requester) and join.c are untouched.
Lands DORMANT like tasks 3.1/3.2: the lane reads c->ha.standby_endpoints and
writes those borrowed standby endpoints, racing io_worker RDHUP teardown if run
off-thread (bug-1197 family). Flag OFF keeps push+teardown serialized on the
io_worker. Task 3.4 flips this ON together with the snapshot and peerctrl flags
after establishing endpoint-lifetime discipline.
Tests (controller suite EXIT=0): test_configpush_offload_runs_on_lane,
test_configpush_inline_when_flag_off, test_configpush_offload_dedups.
Adversarial cpp-review: no CRITICAL/HIGH/MEDIUM findings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Part B (LIVE, default-ON): trim_wal_segments now bounds WAL-segment deletion
by repl_trim_lsn(&p->repl, active_seg_lsn) = min(active segment LSN, every
active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn),
after repl_expire_disconnect_slots ages out stale slot guards. floor <=
active_seg_lsn structurally, so the guard can only preserve more segments,
never delete one a live standby still needs — closes the bug-795/828
trim-vs-cursor race for connected/recently-disconnected standbys (bug-1147
stopgap; empty/cursor=0 join still needs entity-bearing snapshots).
Part A (DORMANT, g_drive_async_snapshot_offload default OFF): ha_snapshot_send_
dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy,
on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body
factored into ha_snapshot_do_send. Per-peer single-flight via
ha_state_t.snapshot_inflight[] CAS + slowlane key; in-flight flag cleared on
all four terminal paths, mirroring the reconnect offload (task 2.1).
Lands dormant like task 3.1: snapshot_lane_fn borrows a standby endpoint it
does not own, so flag-ON would race io_worker RDHUP teardown (cross-thread UAF,
bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker.
Task 3.4 is the single gate that flips both this and the peerctrl flag ON after
establishing endpoint-lifetime discipline; bug-1198 stays open by design until
then. The offload flag was initially authored default-ON (would ship the UAF)
and corrected to OFF in ha.c and the ha.h accessor doc (bug-1210).
Tests (controller 196/0, 0 new leaks): test_snapshot_offload_runs_on_lane,
test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer,
test_trim_guard_preserves_standby_segments,
test_trim_without_standbys_proceeds_normally. Adversarial cpp-review: GO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Heartbeat timer detects a disconnected peer, CAS-acquires a per-peer
single-flight flag, slowlane_submits the bounded connect, and returns — the
delaymgr never blocks on connect. Lane worker (reconnect_lane_fn) does the
bounded endpoint_connect + lock-guarded iomgr_monitor; flag cleared on all
terminal paths (incl. -EEXIST race fix). Default-ON kill-switch retains the
inline path for TDD-RED/fallback. Never self-terminates on failure. 2.2 adds
the inbound/outbound race + failure hardening (documented seam). 196/0 green.
