Adds raft_sim_scenario_figure8: a previous-term entry on a quorum under a
new-term leader is NOT committed until a current-term entry commits (library-
gated by raft_advance_commit; verified by a mutation test). Also restores the
raft_sim_scenario_drop_dup_reorder implementation that commit e445404 (task 5.3)
silently dropped from raft_sim.c (a stray git checkout reverted the uncommitted
impl), which had left the lib build broken with an undefined reference.
Build green: test_sim_5_3_drop_dup_reorder + test_sim_5_4_figure8 both PASS.
raft_on_vote_reply double-counted duplicated vote-responses (counter ignored
from_idx) and failed to drop stale-term replies, allowing two leaders at the
same term (bug-851 split-brain family). Mirror the existing pre-vote per-peer
dedup pattern (new vote_replies[] set) and add DropStaleResponse (term <
currentTerm). Surfaced by the raft-sim-conformance oracle (task 4.2).
47 raft / 264 overall lib tests pass.
Harness surfaced a real upstream raft.c split-brain bug (bug-1095/1096,
bug-851 family) in raft_on_vote_reply: counter-based vote tally double-counts
a duplicated vote-response and no stale-term drop -> two leaders same term.
Out of this spec's boundary (raft.c protocol); recorded as Implementation Note.
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
7.1 Figure-8 commit refusal (G4): prior-term entry at a genuine quorum HELD (only the
current-term gate keeps commit_index at 0), then a current-term entry advances over
both (indirect commit) with one fire. Closes the Req 6.3 two-phase coverage.
7.2 Leader Completeness: committed prefix on {0,1}; node 0 fails; the stale node 2 is
DENIED by the freshness gate so only the up-to-date node 1 can win — and it holds the
full committed prefix (term+bytes).
7.3 freshness lexicographic (vote + prevote): stale-term-higher-index DENIED,
fresher-term-lower-index GRANTED, tie decided by index; prevote side-effect-free.
7.4 on_commit_advanced trigger: fire-count tracks STRICT advances — held(0)/advance(1)/
no-op-redrive(1)/advance(2)/non-leader(2).
Test-only (sim_set_log_entry local helper); reviewed in-parent (transient API 529 on the
review subagent). ASAN green (243 tests).
logOk (raft.tla:285, closes G6): local_last_log_term = log_len>0 ? on_get_entry_term(log_len):0;
logOk = cand_last_log_term > local || (== && cand_last_log_index >= log_len). Lexicographic:
term dominates, tie decided by index. Shared raft_log_ok helper.
Vote (4.2): logOk deny inserted after stale-term/UpdateTerm/role-gate/one-vote-per-term and
BEFORE on_vote_eligible (denies even when the app gate would grant); persist-before-grant intact.
PreVote (5.2): identical logOk, fully side-effect-free (no term/vote/role/commit mutation, no
on_persist). Signatures extended (1.3) on raft_on_vote_request/pre_vote_request; all 39 lib call
sites + sim routing updated; sim captures the candidate's send-time last-log tuple (raft_sim_rpc_t).
Empty-log call sites pass (0,0) → logOk true → existing outcomes preserved. 9 freshness tests.
NOTE: controller cluster.c vote callers still use the old signatures and will NOT compile — an
accepted, design-documented revalidation trigger; the controller adapter plumbing is owned by the
downstream ha-term-safety spec (to be implemented next). Library suite ASAN green (239 tests).
AdvanceCommitIndex (raft.tla:259/270, closes G4): leader-only (-EPERM); Max(agreeIndexes)
= highest N with a quorum ((n_peers/2)+1) of match_index>=N, self counted as log_len;
scan log_len downward. CURRENT-TERM RULE: advance commit_index to N only when
on_get_entry_term(N)==s->term, else HOLD (Figure-8 safety — never commit a previous-term
entry on replication count alone; a later current-term commit indirectly commits it).
Monotonic; fires on_commit_advanced once per strict advance; commit_index<=log_len.
3.3: raft_on_append_entries_reply left untouched — verified it never advances commit
(separation, Req 2.6). 6.1: sim commit-driver helper raft_sim_advance_commit added.
Req 6.3 two-phase indirect-commit transition covered end-to-end by task 7.1 (sim).
ASAN green (230 tests).
1.1: raft_state_t.commit_index (monotonic, ==WAL LSN, <=log_len, advanced only via
raft_advance_commit, untouched by AE reply) + raft_config_t.initial_commit_index.
1.2: on_commit_advanced(ctx, commit_index) REQUIRED callback — fired only on strict
advance, a TRIGGER for the controller commit barrier (never does durability itself).
2.1: raft_init requires it (-EINVAL if NULL) + restores commit_index from
initial_commit_index. raft_advance_commit prototype declared (impl in 3.2).
Coupled callback wiring to keep the required-callback ripple green: sim wires
sim_on_commit_advanced (+per-node commit_index mirror & fire-count), all test cbs
builders set it, raft_sim_restart_node restores the durable commit position. Vote
signatures unchanged; no commit-advance logic yet. ASAN green (224 tests).
raft_sim_restart_node models a process restart: the durable in-memory log[]
(WAL) and persisted (term, voted_for) survive, while volatile consensus state
is cleared by re-running raft_init (returns the node to FOLLOWER). Callbacks,
ctx, idx, and partition flag are preserved.
test_sim_9_5_node_restart_resumes_replication restarts a follower mid-replication
(partial 2-of-3 log), asserts the log survived (load-bearing: log_count==2 would
fail if wiped), role/term restored, then replication resumes and converges with
Log Matching. Closes the 8.3 'node restart while replication proceeds' sub-clause
(8/8 requirement sections now fully covered). ASAN green (220 tests).
9.1 replication + partition catch-up: 3-entry quorum replication then Log Matching;
isolate a follower, heal, assert next_index walks back, catch-up, Log Matching.
9.2 divergent-suffix truncation (G3): follower with a same-index/different-term tail
converges via conflict-truncate; byte-for-byte prefix equality + Log Matching.
9.3 drop/dup/reorder convergence: dropped AE recovered; duplicate already-have AE is
a no-op; genuine out-of-order delivery (entry-3 AE before entry-2) rejected via the
logOk gap with no phantom gap; converges + Log Matching (Req 8.3).
9.4 chaotic-churn G5/bug-851 regression: stale-term node rejoins under a higher-term
leader, adopts the term via the AE/heartbeat UpdateTerm path (load-bearing: would
fail as stale-reject if UpdateTerm-on-AE regressed), voted_for cleared, no second
leader, and a post-converge election campaigns strictly above the old term (cannot
reuse). Log Matching holds throughout.
Review-applied: genuine reorder in 9.3; active re-election + voted_for assertions in
9.4. ASAN green (219 tests).
raft_sim_check_log_matching: for every node pair and every shared index i, if
both entries at i have equal term, assert the whole prefix 1..i is term-and-byte
identical (Req 8.4, raft.tla Log Matching). Read-only, framework-agnostic (returns
1/0), bounded. Self-tests: pass on a converged cluster, fail on a hand-divergent
pair sharing (i=2,term=7) but differing at the index-1 prefix. ASAN green.
raft_sim_deliver_append_entries drains the AE queue, routes each buffered AE to
the (non-partitioned) destination via raft_on_append_entries_request, and routes
the result back to the source via raft_on_append_entries_reply — mirroring the
vote-RPC routing; partitioned endpoints are skipped.
Review-applied (HIGH): raft_sim_ae_t now captures the leader's SEND-time term
(leader_term) at buffer time and delivery uses it instead of the live source
state.term — without this, reorder/churn delivery (9.3/9.4) would inject the
wrong term and defeat UpdateTerm/stale-drop (G5). Smoke test asserts replication
to a connected follower + isolation of a partitioned one. ASAN green.
HandleAppendEntriesResponse (raft.tla:393): stale-term drop first (applied only
when reply_term==s->term, raft.tla:415, Req 5.4 — both lower and higher dropped);
success -> next_index[peer]=match_index+1, match_index[peer]=match_index (Req
5.1-5.2); failure -> next_index[peer]=Max(next-1,1) floored, match_index unchanged
(Req 5.3); no commit advance. uint64 underflow blocked by the >1 floor guard.
3 tests incl. floor-at-1 and lower/higher stale. ASAN green.
Complete HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (raft.tla:406),
implemented as one function to avoid a misleading partial handler. Fixed step
order: UpdateTerm FIRST on every RPC incl. heartbeat (G5, strictly-greater,
clears voted_for, persists, leader step-down fires on_become_follower) ->
stale reject -> candidate step-down -> logOk (G2) -> reject if !logOk ->
accept (already-have / conflict-truncate-then-repersist (G3) / append).
Never advances commit, never does transport I/O.
Review-applied: logOk now includes the explicit prev_log_index <= log_len bound
(raft.tla:330 / Req 4.1) instead of relying solely on the 0-absent sentinel;
added a candidate-equal-term-logOk-fails step-ordering test. 15 tests, ASAN green.
Leader-only emit (raft.tla AppendEntries:204, Req 3.5 — no-op off leader);
prev_log_index=next_index[j]-1, prev_log_term via on_get_entry_term (0/unqueried
at index 0, Req 3.2-3.3); n_entries=1 when next_index[j]<=log_len else 0 heartbeat
(Req 3.4); leader_commit opaque passthrough. Trace fires only on successful send
(consistency with raft_log_append) and non-leader RAFT_OK asserted — both from
review. 4 spy-based tests. ASAN green.
Leader-only (-EPERM for follower/candidate, no mutation/callback); index=
log_len+1, term=currentTerm (raft.tla ClientRequest:246, Req 2.1-2.4);
persist-before-act: on_persist_entry called once BEFORE log_len advances,
non-zero return propagated with no phantom entry (Req 2.5); no commit-index
advance, no transport I/O. Spy-based tests for all four branches. ASAN green.
next_index[]/match_index[]/pre_vote_replies[] are fixed-size RAFT_MAX_PEERS
arrays indexed by peer; without an upper-bound check a caller passing
n_peers > 8 would OOB-write in the leader-progress and pre-vote loops.
Add the guard (returns -EINVAL, no mutation) + canary-checked test.
Latent since task 1.1; surfaced by task 2.2 review (first loop writes into
the new arrays). No current caller exceeds the bound (sim caps at 8, HA <= 4).
next_index[j]=log_len+1, match_index[j]=0 for every peer at both win paths
(single-node RAFT_ELECTED_IMMEDIATELY in raft_start_election; multi-node
RAFT_BECAME_LEADER in raft_on_vote_reply) via shared raft_init_leader_progress
helper (raft.tla BecomeLeader, Req 3.1). Suite green under ASAN.
2.1: raft_init returns -EINVAL when any of on_persist_entry/on_get_entry_term/
on_send_append_entries/on_truncate_from is NULL (Req 7.1-7.4), before any write
to *s; restores s->log_len = cfg->initial_log_len (Req 1.1).
8.1: raft_sim gains a per-node in-memory log (term+bytes per 1-based index) and
an AE buffer queue; wires all four callbacks (persist/get-term/truncate/send).
Pulled forward to land with 2.1 so the stricter init does not leave the
sim/proto suite red between tasks. AE delivery deferred to 8.2.
Suite green under ASAN (Ok:1 Fail:0).
Commit ca1d9ae raised the commit-barrier default 50->500ms in config.h
but left test_config.c asserting ==50, leaving test_persistence_constants
failing on HEAD. Sync the assertion to the committed constant.
Unrelated to raft-log-replication; fixed in isolation so the library
suite has a clean baseline for the spec's task-10.1 green gate.
Send the HA heartbeat as OP_nop + the existing HeartbeatMsg body carrying
{term, committed_lsn, committed_config_tag} (senderPriority unused). Generic
xce_nop_req untouched; C-side body support pre-existed (no typemap edit). Req 1.1
producer half. Round-trip test proves the 3 fields survive the wire; lib suite
0 failures under ASAN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The E_XCE_* enum (errnox.h) gained NOMASTER/REDIRECT/NOT_READY/STANDBY
(10011-10014) but xce_err_str[] still had only 10 strings + NULL, while the
bounds check used E_XCE_MAX. Formatting any of those codes ran off the array:
NOMASTER -> strncpy(NULL) deref; NOT_READY (10013, returned by ctrlstatus on a
failover-exhausted cluster) -> read 8 bytes past the 88-byte table = the ASAN
global-buffer-overflow at utils.c:335.
Fix: fill the table for all 14 codes; bound xce_strerror_r against the actual
array size + NULL (degrade to 'unknown error', never OOB); add a _Static_assert
tying the table length to E_XCE_MAX so any future enum drift fails the build.
Pre-existing foundation bug surfaced (not caused) by controller-status 6.4.
RED: test_strerror_all_xce_codes SEGV'd at utils.c:335 under ASAN; GREEN: lib
suite Ok:1 Fail:0. New tests in test_error.c cover all in-range + out-of-range codes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add io_classify_io_error and propagate endpoint/epoll errors through
endpoint_update_epoll and cbuf_get instead of swallowing them, so client
failover surfaces the real error rather than hanging. Adds netio arm-failure,
close-propagation, and request send-propagation tests.
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
Live 3-node E2E revealed the reported 'Invalid argument' bug was still
reachable: after the primary dies, handle_disconnect leaves ctx.state=
XCE_CTX_ERR, so a FRESH xce_* call returns -EINVAL from the xceapi guard
before reaching a primary. The classifier mapped -EINVAL to TERMINAL, so the
loop bailed without rotating/recovering. decide() now detects the known-dead
ctx and treats it as a prompt reconnect cause (RETRY_NOW), surfacing the
distinct cluster-not-ready code on budget exhaustion, never -EINVAL.
Regression test: test_decide_known_dead_ctx_retries_not_terminal.
