5.5's real root cause was a stale advertised lastLogTerm (c->persist.term vs
persist_entry_term), not the last_activity hypothesis — fixed in b8453b2 and
VERIFIED: with it, the freshest survivor now wins PreVote and enters the real
election (it could not before). But round 4 still does not complete: a deeper
layer (task 5.6) — under the dead-primary RDHUP cadence (~15s) the two survivors
duel in lockstep and the term escalates unbounded (21->25->28->29...) without
reaching majority. Adds task 5.6 (randomized failover backoff to desync the duel
/ don't re-trigger failover mid-real-election). Consensus SAFETY intact; this is
election LIVENESS under sustained churn. 4 layers of the round-4 stall now peeled
(deadlock, segfault, advertised-term fixed; dueling-candidate remains).
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
6-round chaotic soak with both fixes: rounds 1-3 elect; round 4 has NO segfault,
NO fdatasync, NO iomgr deadlock — both fixes held. The round-4 stall is now the
pure election-liveness cross-deny that the crash/deadlock masked: the freshest
survivor (floor 15) is the legitimate winner, but the other survivor (floor 12)
denies it on eligibility because it still believes its dead primary is alive
(in_catch_up + heartbeat-not-yet-stale). Suspect 5.2's last_activity re-arm
keeping a node falsely-fresh after primary death.
Adds task 5.5 (release the heartbeat-fresh/in_catch_up deny on genuine primary
death within the failover window) as the remaining 7.2 blocker. bug-1282 and
bug-1286 are fixed and verified; consensus safety unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
bug-1282 (ABBA iomgr deadlock) is FIXED and verified (commit 25e741d): zero
__lll_lock_wait threads post-fix, unit suite green. But a plain release soak WITH
the fix still segfaults a restarted standby at round 4 — and deadlocks hang, they
do not SIGSEGV, so that crash is a SEPARATE bug (bug-1283), most likely an
endpoint-lifetime UAF under chaotic restart (bug-1197 family).
bug-1283 is a heisenbug (invisible under ASAN and gdb; no usable core in the
sandbox) and likely UPSTREAM raft-drive-async / core libxce endpoint quiescence.
7.2 stays blocked on it. Consensus correctness (Phase-2/5.2/5.3) is unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Using gdb (SIGQUIT all-thread backtrace on the wedged nodes), nailed the round-4
blocker to a deterministic ABBA lock-order deadlock in src/lib async dispatch —
identical on both wedged survivors:
- io_worker (netio.c:672): holds ep->lock, blocks on q->lock (vtask_enq->dataq_enq)
- vdispatcher dataq_deq (dataq.c:65): holds q->lock, blocks on ep->lock via the
vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) cb
io_worker wedges -> node stops accepting/answering PreVotes -> two restarted
survivors can't elect -> round-4 stall. The release SIGSEGV is the same ep/q race
corrupting under churn.
Defect: dataq_deq runs its match callback while holding q->lock, and the callback
takes ep->lock — inverting io_worker's ep->lock->q->lock order. Minimal fix: make
ep->busy atomic + lockless in endpoint_is_task_blocked (or io_worker drops ep->lock
before dataq_enq, or one global lock order).
Ownership: src/lib async-I/O dispatch (vtaskmgr/dataq/netio) = raft-drive-async /
core libxce, a Non-Goal of this spec. Per upstream-ownership protocol, routed to
raft-drive-async (bug-1282); 7.2 stays blocked until the dispatch lock-order is
repaired, then re-run the gate. The gate worked as designed — 7.2 must be GREEN
before raft-drive-async flips dispatch flags ON, and it caught a real dispatch
deadlock. Task 5.4 (root-cause + assign ownership) is DONE. Consensus correctness
(Phase-2/5.2/5.3) is unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Used the now-available gdb (wrapped the release controller under gdb -batch via
XCE_HA_BIN) to expose the round-4 blocker's second face. Findings:
- The release-build SIGSEGV and a gdb-run accept-loop HANG are the SAME restart-
time I/O-subsystem corruption: under gdb the memory bug is perturbed away from
a crash and instead wedges the inbound-accept path (ctrl-3 stops accepting at
03:42:03 while still firing its own election timer), so two restarted survivors
can never exchange PreVotes -> permanent round-N stall.
- This is a restart-STABILITY bug, NOT a consensus-logic gap: PreVote/eligibility/
freshness (incl. 5.3) are correct; the restarted node is simply I/O-dead. It
predates this session (baseline 21e26a1 failed round 3-4).
- Heisenbug: invisible under ASAN and under gdb; no usable core (read-only
core_pattern -> non-running systemd-coredump). Needs a non-ASAN DEBUG (-O0 -g)
build under gdb, or a host with a live coredump pipeline, to backtrace.
Updates research.md, refines task 5.4 (the unified restart-stability bug +
the debug-build path), updates bug-1281. Consensus correctness (Phase-2/5.2/5.3)
is unaffected and validated for every round that completes (2-round soak PASSES;
6-round rounds 1-3 elect; zero job loss; committed prefix byte-identical).
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.
Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
after being restarted into the churning cluster (recovery->failover/PreVote),
dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
debug+core+gdb env.
Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
5.3 fix (bf88681) lands and is unit-validated (test_ha_core_safety 20 Tests /
0 Failures). Re-ran the 7.2 chaotic soak with all Phase-2+5.2+5.3 fixes: it
still stalls at round 4, but bisecting the frozen logs shows the cause is the
bug-924 WAL-fdatasync wedge, NOT consensus — a survivor's persist flusher spins
on `fdatasync FAILED (errno=22 EINVAL)` and stops answering PreVotes, so the
round times out (presents as an election stall, is a durability-layer wedge).
A file-level repro of the controller's exact WAL pattern (open + posix_fallocate
64MB + write + fdatasync) SUCCEEDS on every available ext4/tmpfs path, so it is
the live overlay-backed WAL fd under chaotic restart — the documented bug-924,
a Non-Goal of this spec (owned by ha-commit-durability, excluded from the
verdict).
Re-attributes the round-4 stall (the first §"Task 7.2" note guessed in_catch_up;
5.3 fixes that real latent eligibility deadlock at unit level but it is not the
soak blocker). 7.2 is now BLOCKED ON ENVIRONMENT, not a consensus NO-GO; the
consensus work this spec owns (Phase-2 payload integrity, 5.2, 5.3) is
unit-complete. research.md §"Task 5.3 + 7.2 re-run"; bug-1272 updated.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
First acceptance chaotic soak (6 rounds, real-fs ext4, release binaries rebuilt
at HEAD 01419e1 with Phase-2 payload fix + Phase-5 5.2 suppression). Outcome:
PASS (job safety + landed fixes):
- Zero committed-job loss (all tracked jobs present on the new primary each round).
- Committed prefix [1..11] byte-identical across all 3 members — Phase-2 payload-loss
signature absent. All counted walcompare divergences are present=2 catch-up-LAG
MISSING tails of restarted-uncaught followers (tips 19/15/11), not corruption.
- Rounds 1-3 elect a strictly higher-term primary; 5.2 spurious-failover suppression
confirmed working (round-4 genuine-loss takes the failover path, not suppression).
NO-GO (R8.3 liveness):
- Round-4 election STALL after 3 overlapping kills: the up-to-date leader (tip 19)
is killed, both survivors are lagging (15, 11) but each holds the full committed
prefix; their PreVote rounds repeatedly time out with no grant quorum, so no
primary is elected. Distinct from the 5.2 flap. Contributors: freshness
mutual-denial, in_catch_up-denies-all-votes (Req 2.5, obs 1240) with no live
primary to catch up from, and/or unestablished peer vote connections.
Per the chaotic-harness contingency: recorded full classification in research.md
§"Task 7.2"; added task 5.3 (lagging-but-safe survivor must be electable under
overlapping churn); 7.2 marked _Blocked_ on 5.3; logged bug-1272. Frozen evidence
at /home/claude/tmp/ha-7p2-frozen.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
A recovered standby whose control channel flaps (transient RDHUP) while a
current-term primary is alive treated each flap as "primary died" and launched
an unwinnable PreVote failover (peers see the live primary and deny, the round
times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun
forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5").
Add a freshness guard at the RDHUP failover-initiation path
(ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh,
stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the
primary's reconnect instead of campaigning — the same guard
ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine
loss (stale heartbeat) falls through to failover unchanged; no term bump / no
role change, so Election Safety is untouched.
Keeping the standby attached lets catch-up complete, which clears in_catch_up
and lets votes flow again — so this one guard also resolves the downstream
catching-up-node vote-deny deadlock (obs 1240).
Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures
(spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 →
1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover
root cause (research.md §"Phase 4/5"):
- CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a
control-channel flap (transient RDHUP) while a current-term primary is alive
(fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline
(Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup
-> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no
ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path.
GREEN when task 5.2 adds that guard.
- LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale
heartbeat) still launches a real failover; a failed round re-arms; rapid
rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3).
- SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest
the same term; the real handle_request_vote grants at most one vote per term
(A=3/B=0), so at most one committing primary per term. Falsifiable Election
Safety check mapped to raft/raft.tla (Req 4.1/4.3).
Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean
ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the
intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
5.2: clean failover STRICT_TERM verified — committed prefix byte-identical
(term,pcrc) across all 3 members; standbys persist non-zero term;
0 committed-job loss.
5.4: chaotic 3-round soak — zero job loss; committed prefix consistent
at every walcompare checkpoint; all 3 nodes led; round-3 election
stall is pre-existing bug-851/795 family (NOT regression).
5.5: build+unit suite green (19 groups 0 failures, ASAN clean).
Ha-term-safety: ALL 21 tasks complete. Spec done.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase-4/5 investigation found the live stall is a recovered standby with a
divergent uncommitted tail (former-primary shape) that never reconverges:
loops on ae_reject_logmismatch, its control channel flaps ~15s, each flap
spawns an unwinnable PreVote failover while a live primary exists.
- Phase 4 (Standby Reconvergence): lead with divergent-tail truncate+catch-up
(make the AE backtrack + follower-conflict-truncate fire for a former-primary
tail, not just a short suffix), keep the applied/log_len healthy gate.
- Phase 5 (Election Liveness): retarget from 'missing election-restart' (retry
works) to spurious-failover suppression — reconnect+catch-up on transient
RDHUP, don't storm unwinnable PreVotes while a current-term primary is
reachable. 5.2 now _Depends 4.2_.
Post-Phase-2 chaotic wedge root-caused from frozen soak logs: primary stable
at term 6, but recovered standby ctrl-2 (divergent uncommitted tail 12-15)
cannot reconverge; its primary control channel RDHUP-flaps ~15s, each flap
spawns an unwinnable PreVote failover (peers see primary alive, no grant) ->
timeout -> reconnect -> repeat, so the cluster never reaches stable 3-of-3.
Fix spans Phase 4 (truncate divergent tail + catch up like ctrl-3 does) and
Phase 5 (don't launch unwinnable failover storm on transient RDHUP while a
current-term primary is reachable; reconnect+catch-up first). RED direction
recorded. (Dispatched investigator hung; reconstructed by parent.)
Rebuilt release at HEAD (2.2+2.4), chaotic soak on real ext4. Universally-
held committed prefix (1..11) content-identical across all members; the
present-but-empty committed divergence (bug-1.3 signature) is ABSENT. The
fix eliminates the cross-term payload-loss divergence. Remaining issues:
chaotic soak wedges on election stall (Phase 5) before its verdict; tail
MISSING = catch-up lag (Phase 4). Clean soak verdict gated on Phase 5/4,
not Phase 3 (snapshot, off the critical path).
Root-cause source fix: ha_stream_wal_to_standby reads the entry body via
stream_read_entry_body (bounded retry max 4, no sleep); on persistent short
pread/malloc-fail it fails loud + ABORTS the stream (flush valid prefix,
drop unreadable entry, -EIO) instead of shipping PLEN=0. Incomplete stream
re-driven by ACK-driven catch-up. With the 2.2 guard this closes the live
cross-term payload-loss divergence. 2.3 CORE RED -> GREEN (fail-loud: no
empty persisted; + retry-recover test ships full 64). Term-blind structural
retirement deferred to 6.2 (empty-over-full safety enforced by 2.2 guard +
pinned). 12/12 core suite; siblings green; no regression introduced.
Phase 2 complete.
Minimal XCE_TESTING fault-injection seam in ha_stream_wal_to_standby
(NULL/production-inert) forces a short body read on a cross-term entry.
CORE RED proves the streamer ships PLEN=0 for a full (source_len=64)
cross-term entry: instrumentation source_len=64 SEND_len=0 RECV_len=0 ->
loss is SEND-side (answers task 1.3's open send-vs-transitive question).
2 honest pins (term-blind route guarded by 2.2; below-tip fence). 11
tests/1 RED; production xce_controller links clean without the seam.
Case (a) empty-then-full = genuine RED (payload-blind guard at persist.c:822
no-ops the corrective write, Expected 64 Was 0). Cases (b) full-then-empty
and (c) identical-rewrite = baseline-passing pins that constrain the 2.2
fix asymmetry (empty must never overwrite stored full; identical re-stream
stays idempotent). 7 tests/1 RED/no ASAN.
Release build block-buffers --logfile (flush on signal only), so log-grep
primary/convergence detection false-aborted. Replace all log-grep detection
with client probes (xce_test ctrlstatus): probe_primary/node_probe/
cluster_committed_converged. Both modes use shared helpers; 1.2 verdict
self-test 17/17 intact; bug-854 isolation preserved. Live release-binary
runs reach the verdict (chaotic re-confirms committed_divergence=3 / job
loss 0; sequential PASS). No production code touched. Phase 1 complete.
New test_ha_core_safety.c with deterministic gate/probe seams (no sleeps,
drain-before-fini per bug-1224): payload-controllable WAL writer, on-disk
(term,pcrc,len) readback probe, walcompare observer, apply-gap + kill/
election injection for phases 2.3/4.1/5.1. Smoke-RED (Expected 64 Was 0)
proves the payload-blind (term,lsn)-only guard at persist.c:822 drops a
corrective full-payload write at a committed LSN. 5 tests/1 RED/4 seam
PASS, no ASAN errors.
Reproduced committed divergence at LSN 9 (real-fs release soak): all 3
members term=3, minority ctrl-1 pcrc=0/PLEN=0 vs majority PLEN=3560 =
PAYLOAD loss, term correct. Bisect: arrived via the reconcile seam under
cross-term catch-up (stored term 3, leader term 4 / bug-1067 path), NOT
the term-blind bypass. Real defect = cross-term catch-up payload loss in
ha_stream_wal_to_standby (silent empty ship on short pread/malloc-fail).
Independently verified against frozen WALs. Secondary: release-build log
buffering breaks soak primary-detection.
Per-member (lsn,term,pcrc,present) report + committed-vs-uncommitted
divergence classification via walcompare_report()/--report. Soak signal
(OK/exit-code) preserved. 4 new unit tests, all GREEN.
New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.
Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Verdict: raft-drive-async introduces NO new regression.
- Unit: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites green;
dispatch/snapshot/configpush flags default OFF, inert).
- Live kill-based failover soak on a real fs (~/tmp), release + ASAN builds:
zero committed-JOB loss every completed round (jobs 3->12 monotonic).
- The 6-round CHAOTIC soak does not run clean to round 6 (fails ~round 4), but a
baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4
IDENTICALLY (election stall / reconverge). The round-4 committed-prefix
divergence at LSN 9 is the documented pre-existing bug-851/795 family
(chaotic-failover silent divergence), present before Phase 3 and out of scope
for this spec. Logged as bug-1225 (baseline-confirmed).
Also: make the xce-test-ha soak driver's controller/xce_test paths
env-overridable (XCE_HA_BIN / XCE_HA_XT, backward-compatible defaults) — used to
run the baseline binaries through the same driver for the regression comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
With the dispatch flags flipped ON in-test, prove the three receiver-side
liveness properties (production defaults stay OFF):
- P1: while a snapshot (and config-push) transfer is parked in flight on the
slow lane, another endpoint's peer-control message is still processed
promptly on the vtask pool (different thread), with the transfer still
parked — the offloaded transfer does not block the drive path.
- P2: a peer-control ack (classified VTASK_LIVENESS) runs promptly even while
the pool is saturated to the LivenessAdmission reservation boundary with
REQ_BLOCK tasks (one more REQ_BLOCK fast-rejects with -E_XCE_NOT_READY,
proving real saturation). Narrowed to the liveness-dispatch-under-saturation
assertion; commit-advance itself is covered by the commit-engine tests.
- P3: while one endpoint's peer-control body is parked (holding a worker + that
ep busy), another endpoint's body runs concurrently on a different worker —
per-endpoint FIFO does not serialize across endpoints.
All deterministic (cond-var gates, bounded polls). Fixed a flaky teardown-race
hang (bug-1224): the tests released a parked vtask body then immediately called
the cancel-based vtaskmgr_fini while the just-released body was still unwinding
(P2's REQ_BLOCK bodies racing blk_lock teardown the worst). Fix adds end-of-body
completion counters polled to the full expected count before fini, so every
pool body has fully returned (worker idle) before the cancel. slowlane_fini is
graceful (join) so the lane side was already safe.
Verified: 5/5 deterministic isolated runs; full controller suite EXIT=0;
ASan-clean. Completes task 3.4 (3.4a UAF fix + 3.4b tests); the production
flag-flip remains deferred to after the 4.1 partition soak + 4.2 regression
sweep.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).
Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):
- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
FIFO makes it run strictly after any in-flight/queued same-ep peer-control
vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
run_task's post-body endpoint_unblock_tasks (which would touch the freed
ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.
- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
until dispatch_inflight drains to 0. So every dispatch that read the flag
false finishes its enqueue before mark returns (FIFO-before the teardown
vtask); any later dispatch reads the flag set and drops the packet. ep->lock
is deliberately NOT held across the enqueue (it inverts with the taskq lock in
vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
held window is just a calloc + non-blocking enqueue on the single io_worker.
Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.
Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
push_config_transport_only spawns one thread per standby then joins them with a
CONF_PUSH_TIMEOUT_MS bounded wait — blocking the caller. On the requester-less
drive-thread paths (promotion + heartbeat convergence) that wait stalls the
io_worker. Route those through ha_config_push_offloadable, gated by
g_drive_async_configpush_offload (default OFF → inline, byte-for-byte legacy;
ON → slow lane, non-blocking).
Flag ON deep-copies the config bytes + sha into a lane ctx (caller keeps owning
and freeing its own buffer); a single cluster-wide single-flight guard
(ha_state_t.configpush_inflight scalar CAS + lane key=1) bounds it to one
config-push fan-out in flight; the in-flight flag is cleared on every terminal
path (two alloc failures, -EEXIST, hard submit error, lane completion). Send
body factored into ha_config_push_do_send.
Three requester-less drive-thread call sites converted: raft_cb_become_leader
(promotion), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY
FSM promotion branch. The admin-upgrade barrier (push_config_to_standbys, has a
waiting requester) and join.c are untouched.
Lands DORMANT like tasks 3.1/3.2: the lane reads c->ha.standby_endpoints and
writes those borrowed standby endpoints, racing io_worker RDHUP teardown if run
off-thread (bug-1197 family). Flag OFF keeps push+teardown serialized on the
io_worker. Task 3.4 flips this ON together with the snapshot and peerctrl flags
after establishing endpoint-lifetime discipline.
Tests (controller suite EXIT=0): test_configpush_offload_runs_on_lane,
test_configpush_inline_when_flag_off, test_configpush_offload_dedups.
Adversarial cpp-review: no CRITICAL/HIGH/MEDIUM findings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Part B (LIVE, default-ON): trim_wal_segments now bounds WAL-segment deletion
by repl_trim_lsn(&p->repl, active_seg_lsn) = min(active segment LSN, every
active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn),
after repl_expire_disconnect_slots ages out stale slot guards. floor <=
active_seg_lsn structurally, so the guard can only preserve more segments,
never delete one a live standby still needs — closes the bug-795/828
trim-vs-cursor race for connected/recently-disconnected standbys (bug-1147
stopgap; empty/cursor=0 join still needs entity-bearing snapshots).
Part A (DORMANT, g_drive_async_snapshot_offload default OFF): ha_snapshot_send_
dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy,
on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body
factored into ha_snapshot_do_send. Per-peer single-flight via
ha_state_t.snapshot_inflight[] CAS + slowlane key; in-flight flag cleared on
all four terminal paths, mirroring the reconnect offload (task 2.1).
Lands dormant like task 3.1: snapshot_lane_fn borrows a standby endpoint it
does not own, so flag-ON would race io_worker RDHUP teardown (cross-thread UAF,
bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker.
Task 3.4 is the single gate that flips both this and the peerctrl flag ON after
establishing endpoint-lifetime discipline; bug-1198 stays open by design until
then. The offload flag was initially authored default-ON (would ship the UAF)
and corrected to OFF in ha.c and the ha.h accessor doc (bug-1210).
Tests (controller 196/0, 0 new leaks): test_snapshot_offload_runs_on_lane,
test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer,
test_trim_guard_preserves_standby_segments,
test_trim_without_standbys_proceeds_normally. Adversarial cpp-review: GO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Heartbeat timer detects a disconnected peer, CAS-acquires a per-peer
single-flight flag, slowlane_submits the bounded connect, and returns — the
delaymgr never blocks on connect. Lane worker (reconnect_lane_fn) does the
bounded endpoint_connect + lock-guarded iomgr_monitor; flag cleared on all
terminal paths (incl. -EEXIST race fix). Default-ON kill-switch retains the
inline path for TDD-RED/fallback. Never self-terminates on failure. 2.2 adds
the inbound/outbound race + failure hardening (documented seam). 196/0 green.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single
source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied
last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved
reservation logged at init; under-provisioned (n_vcpus<n_peers) clamp WARNs
loudly, never self-terminates. 8 unit cases, mutation-verified.
Adds two controller-level regression tests driving the production raft_on_vote_reply
via ha->raft (test_ha_election.c): a duplicated vote-grant from one peer must not
double-count toward quorum (bug-1095) and a stale-term reply must be dropped
(bug-1096) — both asserted on role/promotion (no two leaders per term). Formally
absorbs bug-1095/1096 into the ha-term-safety spec (Implementation Note) mapping to
its Requirement 1 / Objective; the raft.c fix (b5b7e01) is owned there with
protocol-suite regression at both the libxce (test_vote_reply_*) and controller
(test_ha_election) levels. No production source changed.
Adds raft_sim_scenario_figure8: a previous-term entry on a quorum under a
new-term leader is NOT committed until a current-term entry commits (library-
gated by raft_advance_commit; verified by a mutation test). Also restores the
raft_sim_scenario_drop_dup_reorder implementation that commit e445404 (task 5.3)
silently dropped from raft_sim.c (a stray git checkout reverted the uncommitted
impl), which had left the lib build broken with an undefined reference.
Build green: test_sim_5_3_drop_dup_reorder + test_sim_5_4_figure8 both PASS.
