raft_on_pre_vote_reply: per-peer-deduped tally returning WON at quorum
((n_peers/2)+1), LOST when all non-self peers reply without quorum, else
PENDING; bounds-guards peer_idx. Never fires become-leader/follower, never
mutates term/voted_for/role, never persists. 8 RED-first tests (3- and 5-node,
dedup, out-of-range, purity). Raft PreVote pure core complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_begin_pre_vote: compute prospective term (term+1) locally without
persisting, arm the pre_vote_* tally with a self-grant, NULL-safely probe each
non-self peer, return RAFT_PRE_VOTE_WON for single-node else RAFT_OK. Stays
FOLLOWER, schedules no timeout, mutates no persistent state. 6 RED-first tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_request: deny on stale term, else reuse on_vote_eligible
(primary/heartbeat-stickiness/catch-up gates). Never mutates term/voted_for/
role and never persists or adopts a higher term. 6 RED-first raft unit tests
assert the purity invariant; libxce suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append preVoteRequest @38 / preVoteReply @39 to the OPCode enum, reusing the
existing RequestVoteRequest/RequestVoteReply payload structs and union arms
(no new structs, no isPreVote flag). Keeps the real-vote wire path unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).
bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.
bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).
Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.
Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
persist pcfg (override applied before persist_init); 0 = use default, positive
overrides — so operators can tune per environment.
TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
Live symptom: every write (jobsub) on a healthy 3-node cluster returned
E_XCE_NOT_READY (10013) — the primary had no write quorum and committed_lsn was
stuck at 0, even though both standbys were connected, heartbeating, streamed, and
the primary RECEIVED their WAL ACKs.
Root cause: raft_cb_become_leader registers a surviving vote endpoint with
repl_add_standby only when ectx->votes[i].peer != NULL, but raft_cb_send_vote_request
set votes[i].ep and never votes[i].peer. So at promotion the guard failed, the
standbys were added to ha->standby_endpoints (streaming) but NOT to the replicator
array c->persist.repl.standbys[] (commit/quorum). Consequently repl_on_ack could
not find the ack's ep (slot_idx=-1) -> commit_on_ack never ran -> committed_lsn
never advanced; majority_candidate/repl_active_standby_count saw 0 standbys ->
no write quorum -> the dispatch gate rejected every write with -XCE_ERR_NOT_READY.
(Streaming/persistence worked, which masked it: reslist/workerlist succeed, only
writes fail.)
Fix: raft_cb_send_vote_request now also sets ectx->votes[votes_idx].peer = peer
(the &candidates[peer_idx] pointer already computed for the connect) at the same
success site as .ep. peer points into the persistent myconf candidates[];
repl_add_standby copies host/port, so no lifetime dependency. Promotion now
registers each surviving vote ep -> standbys enter the replicator -> WAL ACKs
advance committed_lsn -> own-term gate opens -> write quorum established.
TDD: drive the real raft_cb_become_leader over a real persist/repl/commit fixture
and assert repl_active_standby_count post-promotion == N when votes[].peer is set,
== 0 when NULL (the bug). Full controller suite green under ASAN; election/commit/
repl/wire untouched apart from the one-line .peer assignment + a test seam.
The xce_test client config previously used separate 'host' and 'port' settings
and hard-coded nmasters=1, so it could only target a single controller and
could not exercise controller HA failover.
Merge them into a single 'masters' setting (comma-separated "name=host:port"
or "host:port" entries — the same syntax as the controller --peers flag) and
parse it in enter_ctx() with the shared libxce parser xce_parse_peers(), filling
config.nmasters / config.masters[] / config.master_ports[] (capped at
N_MAX_MASTERS). The test client now fails over across controller candidates
exactly like the worker/scheduler/apiserver.
The parsed list lives in a static xce_cluster_config_t so its hostname buffers
outlive enter_ctx() (xcecli_ctx_init stores the config pointer, it does not copy
masters[]). enter_ctx() now returns -EINVAL on an invalid/empty masters setting
(callers already check rc<0). Default masters='localhost:9990' preserves the
prior single-controller default. Builds clean; 'set' shows the masters setting.
A standby applied replicated WAL entries via apply_wal_entry (the replay/
in-memory path) which restores entities + advances applied_lsn but NEVER writes
the entry to the standby's WAL segment. The standby then ACKed (feeding the
primary's commit majority) without persisting — a Raft durability violation:
entries counted toward a commit could be lost on standby restart, and standby
WAL files stayed empty (contradicting steering: log[i] = WAL segments for every
node). This is why ctrl-2/ctrl-3 received + acked the streamed own-term NOOP yet
kept empty seg-*.wal.
Fix: new persist_wal_write_replicated(p, lsn, type, payload, len) writes the
entry to the local WAL at the PRIMARY's lsn (no lsn-reassignment), with a
self-consistent intra-segment CRC chain (the streamed entry carries no term/crc;
each node chains independently and recovery validates intra-segment), fdatasyncs
for durability, advances current_lsn = max(current_lsn, lsn+1), rotates segments,
and does NOT re-broadcast. Wired into ha_apply_single_entry as
persist(durable) -> apply(in-memory) -> ACK; on persist failure: no apply, no ACK
(a standby never ACKs an entry it didn't persist). Covers single + batch
catch-up; OPCode_replEntry has no separate apply path (allow-list only), so all
live WAL apply now persists. Dedup guard (lsn+1<=current_lsn) makes re-streams
idempotent while still ACKing.
TDD: standby write -> drop persist_t -> reopen + persist_recover replays it
(single, 3-entry crc chain, post-dedup); checkpoint/oversized rejected. Full
controller suite green under ASAN; persist_wal_async / recovery / truncation /
streamer / wire untouched.
After reserve-lsn-0 (oldest retained WAL entry = lsn 1 on a fresh cluster), a
freshly-joined empty standby (applied_lsn=0) hit the snapshot-fallback branch
'standby_lsn < oldest_wal_lsn' (0 < 1) and was sent a non-existent snapshot
instead of streaming — so the own-term NOOP (lsn 1) never replicated, committed_lsn
stayed 0, the own-term gate (writable_lsn=1) stayed closed, and standbys kept
empty WALs. Live evidence:
ha_dispatch_repl_action_for_standby: standby[0] lsn=0 < oldest_wal_lsn=1 -> requesting snapshot transfer
The snapshot fallback should fire only when the standby's NEXT-needed entry
(standby_lsn+1) has been GC'd below the oldest retained entry. Fix the boundary
to 'standby_lsn + 1 < oldest_wal_lsn' in ha_evaluate_repl_action (and the parallel
dead repl_needs_snapshot predicate). An empty standby (next-needed=1, oldest=1)
now STREAMs lsn 1 -> acks -> committed_lsn 0->1 -> own-term gate opens -> standbys
receive the WAL. Snapshot still fires for genuinely GC'd-behind standbys
(standby_lsn+1 < oldest). No underflow (oldest=0 -> 1<0 false). Pre-reserve-lsn-0
this off-by-one was masked by oldest_wal_lsn=0 (0<0 false).
TDD: test_empty_standby_fresh_cluster_streams_not_snapshot (RED->GREEN);
snapshot + heartbeat pins strengthened to assert SEND_SNAPSHOT only for genuinely
GC'd standbys and STREAM_WAL at the empty/boundary case. Full controller suite
green under ASAN; commit engine / streamer / apply path / wire untouched.
Root cause (found via live 3-node E2E + a verification test): lsn was 0-based
(hdr.lsn = current_lsn++ from current_lsn=0), and 0 was overloaded as a
sentinel, causing three cold-start defects on a freshly-promoted leader:
- the first WAL entry (own-term promotion NOOP at lsn 0) was structurally
uncommittable: commit_advance_locked only stores when cand>cur and
committed_lsn starts at 0, so lsn 0 can never advance it;
- commit_set_writable_lsn(noop_lsn=0) set writable_lsn=0, which wait_done
reads as 'gate off' — the own-term safety gate was a no-op on the first leader;
- bug-543 (WAL hole): ha_stream_wal_to_standby filters hdr.lsn>standby_lsn;
a fresh empty standby reports standby_lsn=0 (ambiguous 'empty' vs 'has lsn 0'),
so lsn 0 failed 0>0 and was dropped -> non-contiguous standby WAL.
Fix: a fresh persist store initializes current_lsn=1, reserving lsn 0 as the
none/empty sentinel. The first real entry is lsn 1 -> committable (0->1),
writable_lsn=1 -> gate effective, and standby_lsn=0 unambiguously means 'empty'
so catch-up streams from the first entry with no hole. persist_recover is
data-driven (current_lsn=max(hdr.lsn+1)), so existing on-disk WALs and non-fresh
recovery are unaffected; the change only sets the base for brand-new stores.
No wire/capnp change.
Verification: test_coldstart_lsn0 drives the real commit engine + real streamer
and proves H1 (first entry commits 0->1), H3 (gate effective + blocks when
committed<writable), H4 (empty-standby catch-up is contiguous [1,2], control
standby_lsn=1 gets only [2]). Full controller suite green under ASAN. Tests that
relied on the lsn-0 base were shifted (several strengthened); commit.c /
config_repl.c / ha.c / wire format untouched.
Owning specs: ha-commit-engine / ha-wal-replication / persist. Follow-up:
sweep stale lsn-0 comments (repl.c:429-436, persist.c:1055-1075/1240-1243/1296).
Root cause (found via ha-convergence-recovery task 8 live 3-node E2E): a
controller with no local config (config_version_tag==0) could become PRIMARY
and push an empty config, breaking replication.
Two upstream election-safety gaps, both outside ha-convergence-recovery:
1. RDHUP re-election bypassed the no-config 'voter only' guard. The startup
path (ha_start_election_async) refuses to start an election when
my_config_tag==0 && n_peers>1, but the RDHUP failover path
(ha_on_primary_ctrl_fd_rdhup) called raft_on_leader_lost unconditionally.
Fix: mirror the guard before elect_ctx alloc / hook / raft_on_leader_lost;
a config-less node stays CTRL_STANDBY (voter only) and grants the
configured peer instead of starting (and winning) its own election.
2. Voter freshness gate compared the candidate only against the voter's LOCAL
config, ignoring the cluster's known committed state. Fix: ha_vote_self_freshness
floors the comparison at max(local, learned-committed config_tag / committed_lsn),
so a voter denies any candidate whose config/cluster state is outdated vs
what the cluster has committed. Strictly additive (never denies the freshest
node — Leader Completeness preserves liveness).
TDD: test_rdhup_noconfig_node_does_not_elect + configured-node contrast;
test_ha_vote_gate learned-floor deny + liveness + backward-compat. Full
controller suite green under ASAN. commit.c and ha-convergence-recovery
files untouched.
Full controller suite green under ASAN (508 PASS / 0 FAIL / 0 sanitizer
aborts, independently re-censused). All 16 tasks complete; barrier
before reply proven across all restructured opcodes; classification
guard active; barrier-before-state-removal precondition for
ha-single-leader-role satisfied. spec phase=implemented; roadmap
updated. Sub-spec 2 of 4 of the ha-consistency-invariants umbrella.
43-site classification (8 category-A barrier-gated, 6 category-B
receipt-ACK, ~29 category-C) recorded in tasks.md. Reviewer-confirmed
exhaustive from the producer side and the opcode dispatch: every
WAIT-class WAL producer reaches its requester reply only through a
commit barrier; no handler bypasses wal_classify. Zero production
changes. Noted: WORKER_UPDATE has a reply-less second producer
(mark_worker_back liveness sweep) — not a violation, future RETAIN
refinement candidate.
Guards at the top of mark_job_start/finish/update under the caller's
job lock, before BEFORE_EVENT: duplicate (FINISH on terminal) → INFO
no-op skipping the AFTER_EVENT hook (no WAL append, lsn frozen);
terminal→non-terminal backward (START/UPDATE) → WARN reject preserving
state. Shared job_state_is_backward predicate provably admits every
production forward merge (PDONE/PERR/suspend bit-OR) while rejecting
overwrites. mark_job_fail intentionally unguarded (its merges are
always terminal-forward; job_ctrl pre-gates terminal). 5/5 guard tests
green; full job-lifecycle suite green; census 0 FAIL.
3 RED carriers (duplicate FINISH on DONE appends a 2nd WAL entry +
overwrites state; START on terminal clobbers it; replay non-equivalent)
+ 2 preservation pins (forward postdone PDONE-merge, fail-after-finish
PERR-merge) that 6.2 must not break. Reviewer-confirmed: the
UPDATE-clears-terminal arm is production-unreachable (pinned, not
driven). 22 tests / 3 RED; census 508 distinct triples, only those 3
fail.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
notify/update/setpreason/clrpreason admitted in CTRL_PRIMARY without
write quorum (Req 3.2 / umbrella 2.8 / bug-396): all four terminate in
receipt-style RETAIN fact paths — reviewer-traced no-block proof (no
admitted opcode reaches a barrier/WAIT path; request opcodes stay
rejected). 21/21 group; census 494/494 zero FAIL; RED→GREEN
independently reproduced via stash-rebuild.
4 RED carriers (notify/update/setpreason/clrpreason must be admitted
without quorum — currently NOT_READY, the bug-396 hole) + 2 pins (six
request opcodes stay rejected; quorum-present admits all ten). Census
494 = 488 + 6; exactly 4 FAILs.
4 green pins: handle_jobdispatch replies receipt-only with zero WAL
appends through a real 2-alloc dispatch cycle (notify outcomes intact,
reply-precedes-processing); set/clr_preason and do_update_op rusage
stay receipt-style with RETAIN-pending fact appends ACK-committed.
Umbrella Reply-Ordering Contract cited per pin. DISCOVERY bug-451
(reviewer-verified): RUSAGE hook va_list type confusion — 4-byte OOB
read + garbage state in WAL_JOB_RUSAGE payloads; fix deferred to 7.1.
Census 488; zero failures.
