Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.
Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).
bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.
bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).
Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
Two Raft correctness fixes:
1. Priority staggering in ha_wait_for_heartbeat_or_timeout():
Add my_priority × election_priority_step_ms to the timeout so
ctrl-a (priority=0) times out first and starts election before
ctrl-b (priority=1) and ctrl-c (priority=2). This gives the
highest-priority node a head start and prevents simultaneous
campaigning that causes perpetual split votes.
2. candidate_id in vote RPC:
Pass the actual node name (myconf->name) as candidate_id instead
of "". This fixes the voted_for deduplication gate (Req 2.6):
a peer that already voted for ctrl-a will correctly deny a
second vote request from ctrl-b in the same term.
Thread context gets my_name field; am_i_primary_quorum() takes
my_name parameter; all call sites updated (myapp.c, ha.c RDHUP,
test_ha_election.c stubs).
Replace single 2-round election attempt with an infinite retry loop:
1. ha_wait_for_heartbeat_or_timeout() — catch existing primary's heartbeat
(→ STANDBY immediately) or timeout and proceed to election
2. am_i_primary_quorum() — run 2-round election
(→ PRIMARY if won, otherwise loop back to step 1)
Nodes now retry indefinitely until one wins a quorum. On split vote
or no-peer-reachable scenarios the node loops back; on the next
heartbeat wait, it either catches the winning primary's heartbeat
(→ STANDBY) or times out and tries again with a higher term and
randomized jitter (Raft's split-vote resolution mechanism).
This is the core Raft liveness property: as long as a majority of
nodes are reachable, the cluster eventually elects a leader.
1. iomgr_monitor double-register: cluster_start skips iomgr_monitor
if c->ep was already registered by cluster_register_listener
(ep->flags non-zero after successful registration). Previously
returned EEXIST → -1 → "Operation not permitted" crash.
2. ASAN stack-use-after-return: cluster_fini now calls iomgr_stop()
before iomgr_fini() to join the worker thread before the cluster_t
stack frame is reclaimed by the caller.
3. Election timing: add ha_wait_for_heartbeat_or_timeout() before
election in myapp_run. If an existing primary sends a heartbeat,
node becomes STANDBY immediately without competing in election.
If timeout expires (no primary running), proceed to election.
This prevents ctrl-a from immediately losing when ctrl-b/c are
not yet started — it now waits election_timeout_ms (2s) first.
vote_request_thread connects to peers via SSL. iomgr was started before
election but c->ep was only registered with epoll inside cluster_start(),
which runs after election. The TCP connect succeeded (kernel accepts it)
but SSL_accept was never called server-side, so SSL_connect failed
immediately — both vote threads went to deny in <1ms and ctrl-a could
never win a quorum.
Fix: add cluster_register_listener() that wires c->ep into iomgr's
epoll watch. Call it from myapp_run() right after iomgr_start/vtaskmgr_start,
before election. cluster_start() still calls iomgr_monitor again
(idempotent via EPOLL_CTL_MOD).
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
- Add --vcpus N argp option in main.c; store in myconf.vcpus
- Replace inline argp guard with validate_vcpus() call (returns -EINVAL for <=0)
- Add validate_startup_config() in startup_validation.c: returns 0; writes
deprecation warning into caller-supplied warn_buf when vcpus key present
- Wire validate_startup_config() into myapp_run() after conffile resolved;
emit WARN() if warn_buf non-empty (Req 4.4)
- Remove vcpus field from cluster_config_t (clusterconfig.h)
- Preserve capnp ordinal: vcpus @7 -> vcpusDeprecated @7: Void in controller.capnp
- vtaskmgr_init uses only myconf->vcpus with hardcoded default 8
- Add stub C types in typemap.h for capnp schema extensions from task 1.1
- 96 tests pass
- myapp.c: capture am_i_primary() into is_primary (avoids redundant
second probe call); after load_primary_config() succeeds, record
CTRL_PRIMARY role via atomic_store_explicit so cluster_init_phase2()
sees the correct role (ha_init() defaults to CTRL_STANDBY)
- myapp.c: add #include "ha.h" for CTRL_PRIMARY, atomic_store_explicit,
and memory_order_release
- cluster.c: cluster_init() wrapper now calls cluster_fini(c) before
returning on create_ssl_context failure, hook failure, and
cluster_init_phase2 failure — prevents SSL ctx and phase1 resource leak
