9.1 replication + partition catch-up: 3-entry quorum replication then Log Matching;
isolate a follower, heal, assert next_index walks back, catch-up, Log Matching.
9.2 divergent-suffix truncation (G3): follower with a same-index/different-term tail
converges via conflict-truncate; byte-for-byte prefix equality + Log Matching.
9.3 drop/dup/reorder convergence: dropped AE recovered; duplicate already-have AE is
a no-op; genuine out-of-order delivery (entry-3 AE before entry-2) rejected via the
logOk gap with no phantom gap; converges + Log Matching (Req 8.3).
9.4 chaotic-churn G5/bug-851 regression: stale-term node rejoins under a higher-term
leader, adopts the term via the AE/heartbeat UpdateTerm path (load-bearing: would
fail as stale-reject if UpdateTerm-on-AE regressed), voted_for cleared, no second
leader, and a post-converge election campaigns strictly above the old term (cannot
reuse). Log Matching holds throughout.
Review-applied: genuine reorder in 9.3; active re-election + voted_for assertions in
9.4. ASAN green (219 tests).
raft_sim_check_log_matching: for every node pair and every shared index i, if
both entries at i have equal term, assert the whole prefix 1..i is term-and-byte
identical (Req 8.4, raft.tla Log Matching). Read-only, framework-agnostic (returns
1/0), bounded. Self-tests: pass on a converged cluster, fail on a hand-divergent
pair sharing (i=2,term=7) but differing at the index-1 prefix. ASAN green.
raft_sim_deliver_append_entries drains the AE queue, routes each buffered AE to
the (non-partitioned) destination via raft_on_append_entries_request, and routes
the result back to the source via raft_on_append_entries_reply — mirroring the
vote-RPC routing; partitioned endpoints are skipped.
Review-applied (HIGH): raft_sim_ae_t now captures the leader's SEND-time term
(leader_term) at buffer time and delivery uses it instead of the live source
state.term — without this, reorder/churn delivery (9.3/9.4) would inject the
wrong term and defeat UpdateTerm/stale-drop (G5). Smoke test asserts replication
to a connected follower + isolation of a partitioned one. ASAN green.
HandleAppendEntriesResponse (raft.tla:393): stale-term drop first (applied only
when reply_term==s->term, raft.tla:415, Req 5.4 — both lower and higher dropped);
success -> next_index[peer]=match_index+1, match_index[peer]=match_index (Req
5.1-5.2); failure -> next_index[peer]=Max(next-1,1) floored, match_index unchanged
(Req 5.3); no commit advance. uint64 underflow blocked by the >1 floor guard.
3 tests incl. floor-at-1 and lower/higher stale. ASAN green.
Complete HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (raft.tla:406),
implemented as one function to avoid a misleading partial handler. Fixed step
order: UpdateTerm FIRST on every RPC incl. heartbeat (G5, strictly-greater,
clears voted_for, persists, leader step-down fires on_become_follower) ->
stale reject -> candidate step-down -> logOk (G2) -> reject if !logOk ->
accept (already-have / conflict-truncate-then-repersist (G3) / append).
Never advances commit, never does transport I/O.
Review-applied: logOk now includes the explicit prev_log_index <= log_len bound
(raft.tla:330 / Req 4.1) instead of relying solely on the 0-absent sentinel;
added a candidate-equal-term-logOk-fails step-ordering test. 15 tests, ASAN green.
Leader-only emit (raft.tla AppendEntries:204, Req 3.5 — no-op off leader);
prev_log_index=next_index[j]-1, prev_log_term via on_get_entry_term (0/unqueried
at index 0, Req 3.2-3.3); n_entries=1 when next_index[j]<=log_len else 0 heartbeat
(Req 3.4); leader_commit opaque passthrough. Trace fires only on successful send
(consistency with raft_log_append) and non-leader RAFT_OK asserted — both from
review. 4 spy-based tests. ASAN green.
Leader-only (-EPERM for follower/candidate, no mutation/callback); index=
log_len+1, term=currentTerm (raft.tla ClientRequest:246, Req 2.1-2.4);
persist-before-act: on_persist_entry called once BEFORE log_len advances,
non-zero return propagated with no phantom entry (Req 2.5); no commit-index
advance, no transport I/O. Spy-based tests for all four branches. ASAN green.
next_index[]/match_index[]/pre_vote_replies[] are fixed-size RAFT_MAX_PEERS
arrays indexed by peer; without an upper-bound check a caller passing
n_peers > 8 would OOB-write in the leader-progress and pre-vote loops.
Add the guard (returns -EINVAL, no mutation) + canary-checked test.
Latent since task 1.1; surfaced by task 2.2 review (first loop writes into
the new arrays). No current caller exceeds the bound (sim caps at 8, HA <= 4).
next_index[j]=log_len+1, match_index[j]=0 for every peer at both win paths
(single-node RAFT_ELECTED_IMMEDIATELY in raft_start_election; multi-node
RAFT_BECAME_LEADER in raft_on_vote_reply) via shared raft_init_leader_progress
helper (raft.tla BecomeLeader, Req 3.1). Suite green under ASAN.
2.1: raft_init returns -EINVAL when any of on_persist_entry/on_get_entry_term/
on_send_append_entries/on_truncate_from is NULL (Req 7.1-7.4), before any write
to *s; restores s->log_len = cfg->initial_log_len (Req 1.1).
8.1: raft_sim gains a per-node in-memory log (term+bytes per 1-based index) and
an AE buffer queue; wires all four callbacks (persist/get-term/truncate/send).
Pulled forward to land with 2.1 so the stricter init does not leave the
sim/proto suite red between tasks. AE delivery deferred to 8.2.
Suite green under ASAN (Ok:1 Fail:0).
Commit ca1d9ae raised the commit-barrier default 50->500ms in config.h
but left test_config.c asserting ==50, leaving test_persistence_constants
failing on HEAD. Sync the assertion to the committed constant.
Unrelated to raft-log-replication; fixed in isolation so the library
suite has a clean baseline for the spec's task-10.1 green gate.
Send the HA heartbeat as OP_nop + the existing HeartbeatMsg body carrying
{term, committed_lsn, committed_config_tag} (senderPriority unused). Generic
xce_nop_req untouched; C-side body support pre-existed (no typemap edit). Req 1.1
producer half. Round-trip test proves the 3 fields survive the wire; lib suite
0 failures under ASAN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The E_XCE_* enum (errnox.h) gained NOMASTER/REDIRECT/NOT_READY/STANDBY
(10011-10014) but xce_err_str[] still had only 10 strings + NULL, while the
bounds check used E_XCE_MAX. Formatting any of those codes ran off the array:
NOMASTER -> strncpy(NULL) deref; NOT_READY (10013, returned by ctrlstatus on a
failover-exhausted cluster) -> read 8 bytes past the 88-byte table = the ASAN
global-buffer-overflow at utils.c:335.
Fix: fill the table for all 14 codes; bound xce_strerror_r against the actual
array size + NULL (degrade to 'unknown error', never OOB); add a _Static_assert
tying the table length to E_XCE_MAX so any future enum drift fails the build.
Pre-existing foundation bug surfaced (not caused) by controller-status 6.4.
RED: test_strerror_all_xce_codes SEGV'd at utils.c:335 under ASAN; GREEN: lib
suite Ok:1 Fail:0. New tests in test_error.c cover all in-range + out-of-range codes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add io_classify_io_error and propagate endpoint/epoll errors through
endpoint_update_epoll and cbuf_get instead of swallowing them, so client
failover surfaces the real error rather than hanging. Adds netio arm-failure,
close-propagation, and request send-propagation tests.
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
xce_failover_rotate reconnected with endpoint_fini+endpoint_init+endpoint_connect
but never re-bound ep->iomgr (reset by endpoint_init) nor re-armed the owner/
msg/err handlers + iomgr_monitor. endpoint_connect then dereferenced a NULL
iomgr (->ssl_ctx) → SEGV at netio.c:672 during live master failover. Added
xce_ctx_reconnect() in xcectx.c that mirrors xcecli_start's establishment
(without restarting iomgr/vtaskmgr or touching ctx->state); rotate delegates to
it. Found via live 3-node E2E. bug-610.
Live 3-node E2E revealed the reported 'Invalid argument' bug was still
reachable: after the primary dies, handle_disconnect leaves ctx.state=
XCE_CTX_ERR, so a FRESH xce_* call returns -EINVAL from the xceapi guard
before reaching a primary. The classifier mapped -EINVAL to TERMINAL, so the
loop bailed without rotating/recovering. decide() now detects the known-dead
ctx and treats it as a prompt reconnect cause (RETRY_NOW), surfacing the
distinct cluster-not-ready code on budget exhaustion, never -EINVAL.
Regression test: test_decide_known_dead_ctx_retries_not_terminal.
Four property-style raft tests proving the PreVote API preserves the safety
invariants: term/voted_for/role never change and on_persist never fires across
mixed begin/request/reply sequences (WON and LOST); a WON round creates no leader
(stays FOLLOWER); and the real-vote higher-term-adopt + step-down + quorum-elect
path is unchanged (bug-339/340 intact). Records the TLA+/checklist consistency
argument (RN-1) in test_raft.c and research.md, citing the tests as evidence.
Raft suite 123/0/0; zero production change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_reply: per-peer-deduped tally returning WON at quorum
((n_peers/2)+1), LOST when all non-self peers reply without quorum, else
PENDING; bounds-guards peer_idx. Never fires become-leader/follower, never
mutates term/voted_for/role, never persists. 8 RED-first tests (3- and 5-node,
dedup, out-of-range, purity). Raft PreVote pure core complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_begin_pre_vote: compute prospective term (term+1) locally without
persisting, arm the pre_vote_* tally with a self-grant, NULL-safely probe each
non-self peer, return RAFT_PRE_VOTE_WON for single-node else RAFT_OK. Stays
FOLLOWER, schedules no timeout, mutates no persistent state. 6 RED-first tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_request: deny on stale term, else reuse on_vote_eligible
(primary/heartbeat-stickiness/catch-up gates). Never mutates term/voted_for/
role and never persists or adopts a higher term. 6 RED-first raft unit tests
assert the purity invariant; libxce suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
Root cause: task_cmp in delay.c compared nextrun (always 0 for ms-tasks)
instead of nextrun_ts, making all ms-tasks appear equal in the ordered
queue. The heartbeat task was re-enqueued correctly but stuck behind the
liveness poll task in the broken ordering.
Changes:
- delay.c: fix task_cmp to compare nextrun_ts for ms-tasks (ns precision)
- ha.c: use runcount=-1 (infinite repeat) instead of one-shot+re-arm;
simpler and avoids the broken re-arm path entirely
- ha.c: add static atomic idempotency guard to prevent duplicate heartbeat
tasks on re-election
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
