Root-caused via gdb (all-thread backtrace on two independently-wedged controllers
during chaotic failover): an ABBA lock-order deadlock in the async dispatch path.
io_worker (netio.c): holds ep->lock, blocks on q->lock (vtask_enq -> dataq_enq)
vdispatcher: dataq_deq holds q->lock, blocks on ep->lock via the
vtask_eligible -> endpoint_is_task_blocked match callback
endpoint_is_task_blocked took ep->lock while the caller (dataq_deq) held the dataq
q->lock, inverting io_worker's ep->lock -> q->lock order. Under churn the two
orders collide and the iomgr wedges (no more accepts / vote replies).
Fix (mirrors the dispatch_inflight precedent, bug-1197): promote endpoint_t.busy
from a 1-bit bitfield to _Atomic int and make endpoint_block_tasks /
endpoint_unblock_tasks / endpoint_is_task_blocked lockless (release/acquire). The
ep->lock that wrapped each access protected nothing else (busy only gates the
vtask queue), so removing it is safe; the inherent set-then-check TOCTOU is
unchanged (the lock never prevented it — the decision is advisory).
Verified: post-fix gdb soak shows ZERO threads in __lll_lock_wait and
endpoint_is_task_blocked no longer appears in any lock path (deadlock gone);
controller_unit_test all green (no regression). NOTE: a SEPARATE release-only
restart segfault remains and still blocks the raft-ha-core-safety 7.2 gate
(tracked separately) — deadlocks hang, they do not SIGSEGV, so it is a distinct
memory bug.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).
Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):
- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
FIFO makes it run strictly after any in-flight/queued same-ep peer-control
vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
run_task's post-body endpoint_unblock_tasks (which would touch the freed
ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.
- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
until dispatch_inflight drains to 0. So every dispatch that read the flag
false finishes its enqueue before mark returns (FIFO-before the teardown
vtask); any later dispatch reads the flag set and drops the packet. ep->lock
is deliberately NOT held across the enqueue (it inverts with the taskq lock in
vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
held window is just a calloc + non-blocking enqueue on the single io_worker.
Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.
Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Lands the PeerCtrlDispatch machinery behind a DEFAULT-OFF flag
(g_drive_async_peerctrl_dispatch_enabled); production behavior is
UNCHANGED. peer_ctrl_msg_handler becomes a thin shim that takes
ownership of ep->fsm_in.pkt and either runs the routing body inline
(flag OFF, byte-for-byte the prior path) or dispatches it to the vtask
pool via vtask_enq_classified (flag ON). The 3 routers now take pkt as
an explicit param (no cross-thread ep->fsm_in.pkt re-read). Per-endpoint
arrival order is enforced by bind_vtask's endpoint_block_tasks.
Packet ownership: shim clears ep->fsm_in.pkt at dispatch (sole-owner
transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees
it once. New vtask_t.owns_io_pkt (default 0) lets taskq_item_killer free
the carried packet for an un-executed queued task at vtaskmgr_fini WITHOUT
double-freeing the libxce endpoint-owned path (bug-1199).
Adversarial review found two FLAG-ON hazards, documented with loud
comments (flag is OFF so production is unaffected), to be resolved before
task 3.4 flips the flag ON:
- bug-1197 (CRITICAL): raft_cb_become_follower endpoint_fini()s peer
vote-eps from a pool worker, racing the io_worker's read of ep->data;
needs iomgr_unmonitor before endpoint_fini in every pool-reachable
teardown.
- bug-1198 (HIGH): peer_ctrl_classify tags all work VTASK_LIVENESS, but
the CTRL_PRIMARY repl-ack path can run a blocking snapshot checkpoint
inline; resolved by task 3.2 (offload snapshot to SlowLane).
Per spec, 3.1 is not 'done' until the task 3.4 deadlock gate is green.
Tests: 3 new peerctrl tests; controller 196/196, lib 282/0; 0 net leaks.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
- sim_tla_action_name: drop brittle per-case 'raft.tla:NNN' line-number comments
(some had drifted a few lines off; the return string IS the stable raft.tla
Next-disjunct name, and the function header already cites raft.tla:454).
- test_sim_5_1_partition_heal: the per-follower convergence loop was vacuous —
it bounded by commit_index, which stays 0 for followers under the leader-only
commit model, so the loop never ran. Bound by log_count instead (followers DO
advance the replicated log); Log Matching still guarantees the shared-prefix
terms agree. Now non-vacuous; still passes.
Test/comment-only; no behavior change. Lib suite Ok:1 Fail:0.
Adds raft_sim_scenario_figure8: a previous-term entry on a quorum under a
new-term leader is NOT committed until a current-term entry commits (library-
gated by raft_advance_commit; verified by a mutation test). Also restores the
raft_sim_scenario_drop_dup_reorder implementation that commit e445404 (task 5.3)
silently dropped from raft_sim.c (a stray git checkout reverted the uncommitted
impl), which had left the lib build broken with an undefined reference.
Build green: test_sim_5_3_drop_dup_reorder + test_sim_5_4_figure8 both PASS.
raft_on_vote_reply double-counted duplicated vote-responses (counter ignored
from_idx) and failed to drop stale-term replies, allowing two leaders at the
same term (bug-851 split-brain family). Mirror the existing pre-vote per-peer
dedup pattern (new vote_replies[] set) and add DropStaleResponse (term <
currentTerm). Surfaced by the raft-sim-conformance oracle (task 4.2).
47 raft / 264 overall lib tests pass.
Harness surfaced a real upstream raft.c split-brain bug (bug-1095/1096,
bug-851 family) in raft_on_vote_reply: counter-based vote tally double-counts
a duplicated vote-response and no stale-term drop -> two leaders same term.
Out of this spec's boundary (raft.c protocol); recorded as Implementation Note.
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
7.1 Figure-8 commit refusal (G4): prior-term entry at a genuine quorum HELD (only the
current-term gate keeps commit_index at 0), then a current-term entry advances over
both (indirect commit) with one fire. Closes the Req 6.3 two-phase coverage.
7.2 Leader Completeness: committed prefix on {0,1}; node 0 fails; the stale node 2 is
DENIED by the freshness gate so only the up-to-date node 1 can win — and it holds the
full committed prefix (term+bytes).
7.3 freshness lexicographic (vote + prevote): stale-term-higher-index DENIED,
fresher-term-lower-index GRANTED, tie decided by index; prevote side-effect-free.
7.4 on_commit_advanced trigger: fire-count tracks STRICT advances — held(0)/advance(1)/
no-op-redrive(1)/advance(2)/non-leader(2).
Test-only (sim_set_log_entry local helper); reviewed in-parent (transient API 529 on the
review subagent). ASAN green (243 tests).
logOk (raft.tla:285, closes G6): local_last_log_term = log_len>0 ? on_get_entry_term(log_len):0;
logOk = cand_last_log_term > local || (== && cand_last_log_index >= log_len). Lexicographic:
term dominates, tie decided by index. Shared raft_log_ok helper.
Vote (4.2): logOk deny inserted after stale-term/UpdateTerm/role-gate/one-vote-per-term and
BEFORE on_vote_eligible (denies even when the app gate would grant); persist-before-grant intact.
PreVote (5.2): identical logOk, fully side-effect-free (no term/vote/role/commit mutation, no
on_persist). Signatures extended (1.3) on raft_on_vote_request/pre_vote_request; all 39 lib call
sites + sim routing updated; sim captures the candidate's send-time last-log tuple (raft_sim_rpc_t).
Empty-log call sites pass (0,0) → logOk true → existing outcomes preserved. 9 freshness tests.
NOTE: controller cluster.c vote callers still use the old signatures and will NOT compile — an
accepted, design-documented revalidation trigger; the controller adapter plumbing is owned by the
downstream ha-term-safety spec (to be implemented next). Library suite ASAN green (239 tests).
AdvanceCommitIndex (raft.tla:259/270, closes G4): leader-only (-EPERM); Max(agreeIndexes)
= highest N with a quorum ((n_peers/2)+1) of match_index>=N, self counted as log_len;
scan log_len downward. CURRENT-TERM RULE: advance commit_index to N only when
on_get_entry_term(N)==s->term, else HOLD (Figure-8 safety — never commit a previous-term
entry on replication count alone; a later current-term commit indirectly commits it).
Monotonic; fires on_commit_advanced once per strict advance; commit_index<=log_len.
3.3: raft_on_append_entries_reply left untouched — verified it never advances commit
(separation, Req 2.6). 6.1: sim commit-driver helper raft_sim_advance_commit added.
Req 6.3 two-phase indirect-commit transition covered end-to-end by task 7.1 (sim).
ASAN green (230 tests).
1.1: raft_state_t.commit_index (monotonic, ==WAL LSN, <=log_len, advanced only via
raft_advance_commit, untouched by AE reply) + raft_config_t.initial_commit_index.
1.2: on_commit_advanced(ctx, commit_index) REQUIRED callback — fired only on strict
advance, a TRIGGER for the controller commit barrier (never does durability itself).
2.1: raft_init requires it (-EINVAL if NULL) + restores commit_index from
initial_commit_index. raft_advance_commit prototype declared (impl in 3.2).
Coupled callback wiring to keep the required-callback ripple green: sim wires
sim_on_commit_advanced (+per-node commit_index mirror & fire-count), all test cbs
builders set it, raft_sim_restart_node restores the durable commit position. Vote
signatures unchanged; no commit-advance logic yet. ASAN green (224 tests).
raft_sim_restart_node models a process restart: the durable in-memory log[]
(WAL) and persisted (term, voted_for) survive, while volatile consensus state
is cleared by re-running raft_init (returns the node to FOLLOWER). Callbacks,
ctx, idx, and partition flag are preserved.
test_sim_9_5_node_restart_resumes_replication restarts a follower mid-replication
(partial 2-of-3 log), asserts the log survived (load-bearing: log_count==2 would
fail if wiped), role/term restored, then replication resumes and converges with
Log Matching. Closes the 8.3 'node restart while replication proceeds' sub-clause
(8/8 requirement sections now fully covered). ASAN green (220 tests).
9.1 replication + partition catch-up: 3-entry quorum replication then Log Matching;
isolate a follower, heal, assert next_index walks back, catch-up, Log Matching.
9.2 divergent-suffix truncation (G3): follower with a same-index/different-term tail
converges via conflict-truncate; byte-for-byte prefix equality + Log Matching.
9.3 drop/dup/reorder convergence: dropped AE recovered; duplicate already-have AE is
a no-op; genuine out-of-order delivery (entry-3 AE before entry-2) rejected via the
logOk gap with no phantom gap; converges + Log Matching (Req 8.3).
9.4 chaotic-churn G5/bug-851 regression: stale-term node rejoins under a higher-term
leader, adopts the term via the AE/heartbeat UpdateTerm path (load-bearing: would
fail as stale-reject if UpdateTerm-on-AE regressed), voted_for cleared, no second
leader, and a post-converge election campaigns strictly above the old term (cannot
reuse). Log Matching holds throughout.
Review-applied: genuine reorder in 9.3; active re-election + voted_for assertions in
9.4. ASAN green (219 tests).
raft_sim_check_log_matching: for every node pair and every shared index i, if
both entries at i have equal term, assert the whole prefix 1..i is term-and-byte
identical (Req 8.4, raft.tla Log Matching). Read-only, framework-agnostic (returns
1/0), bounded. Self-tests: pass on a converged cluster, fail on a hand-divergent
pair sharing (i=2,term=7) but differing at the index-1 prefix. ASAN green.
raft_sim_deliver_append_entries drains the AE queue, routes each buffered AE to
the (non-partitioned) destination via raft_on_append_entries_request, and routes
the result back to the source via raft_on_append_entries_reply — mirroring the
vote-RPC routing; partitioned endpoints are skipped.
Review-applied (HIGH): raft_sim_ae_t now captures the leader's SEND-time term
(leader_term) at buffer time and delivery uses it instead of the live source
state.term — without this, reorder/churn delivery (9.3/9.4) would inject the
wrong term and defeat UpdateTerm/stale-drop (G5). Smoke test asserts replication
to a connected follower + isolation of a partitioned one. ASAN green.
HandleAppendEntriesResponse (raft.tla:393): stale-term drop first (applied only
when reply_term==s->term, raft.tla:415, Req 5.4 — both lower and higher dropped);
success -> next_index[peer]=match_index+1, match_index[peer]=match_index (Req
5.1-5.2); failure -> next_index[peer]=Max(next-1,1) floored, match_index unchanged
(Req 5.3); no commit advance. uint64 underflow blocked by the >1 floor guard.
3 tests incl. floor-at-1 and lower/higher stale. ASAN green.
Complete HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (raft.tla:406),
implemented as one function to avoid a misleading partial handler. Fixed step
order: UpdateTerm FIRST on every RPC incl. heartbeat (G5, strictly-greater,
clears voted_for, persists, leader step-down fires on_become_follower) ->
stale reject -> candidate step-down -> logOk (G2) -> reject if !logOk ->
accept (already-have / conflict-truncate-then-repersist (G3) / append).
Never advances commit, never does transport I/O.
Review-applied: logOk now includes the explicit prev_log_index <= log_len bound
(raft.tla:330 / Req 4.1) instead of relying solely on the 0-absent sentinel;
added a candidate-equal-term-logOk-fails step-ordering test. 15 tests, ASAN green.
Leader-only emit (raft.tla AppendEntries:204, Req 3.5 — no-op off leader);
prev_log_index=next_index[j]-1, prev_log_term via on_get_entry_term (0/unqueried
at index 0, Req 3.2-3.3); n_entries=1 when next_index[j]<=log_len else 0 heartbeat
(Req 3.4); leader_commit opaque passthrough. Trace fires only on successful send
(consistency with raft_log_append) and non-leader RAFT_OK asserted — both from
review. 4 spy-based tests. ASAN green.
Leader-only (-EPERM for follower/candidate, no mutation/callback); index=
log_len+1, term=currentTerm (raft.tla ClientRequest:246, Req 2.1-2.4);
persist-before-act: on_persist_entry called once BEFORE log_len advances,
non-zero return propagated with no phantom entry (Req 2.5); no commit-index
advance, no transport I/O. Spy-based tests for all four branches. ASAN green.
next_index[]/match_index[]/pre_vote_replies[] are fixed-size RAFT_MAX_PEERS
arrays indexed by peer; without an upper-bound check a caller passing
n_peers > 8 would OOB-write in the leader-progress and pre-vote loops.
Add the guard (returns -EINVAL, no mutation) + canary-checked test.
Latent since task 1.1; surfaced by task 2.2 review (first loop writes into
the new arrays). No current caller exceeds the bound (sim caps at 8, HA <= 4).
next_index[j]=log_len+1, match_index[j]=0 for every peer at both win paths
(single-node RAFT_ELECTED_IMMEDIATELY in raft_start_election; multi-node
RAFT_BECAME_LEADER in raft_on_vote_reply) via shared raft_init_leader_progress
helper (raft.tla BecomeLeader, Req 3.1). Suite green under ASAN.
2.1: raft_init returns -EINVAL when any of on_persist_entry/on_get_entry_term/
on_send_append_entries/on_truncate_from is NULL (Req 7.1-7.4), before any write
to *s; restores s->log_len = cfg->initial_log_len (Req 1.1).
8.1: raft_sim gains a per-node in-memory log (term+bytes per 1-based index) and
an AE buffer queue; wires all four callbacks (persist/get-term/truncate/send).
Pulled forward to land with 2.1 so the stricter init does not leave the
sim/proto suite red between tasks. AE delivery deferred to 8.2.
Suite green under ASAN (Ok:1 Fail:0).
Commit ca1d9ae raised the commit-barrier default 50->500ms in config.h
but left test_config.c asserting ==50, leaving test_persistence_constants
failing on HEAD. Sync the assertion to the committed constant.
Unrelated to raft-log-replication; fixed in isolation so the library
suite has a clean baseline for the spec's task-10.1 green gate.
