Root cause (found via ha-convergence-recovery task 8 live 3-node E2E): a
controller with no local config (config_version_tag==0) could become PRIMARY
and push an empty config, breaking replication.
Two upstream election-safety gaps, both outside ha-convergence-recovery:
1. RDHUP re-election bypassed the no-config 'voter only' guard. The startup
path (ha_start_election_async) refuses to start an election when
my_config_tag==0 && n_peers>1, but the RDHUP failover path
(ha_on_primary_ctrl_fd_rdhup) called raft_on_leader_lost unconditionally.
Fix: mirror the guard before elect_ctx alloc / hook / raft_on_leader_lost;
a config-less node stays CTRL_STANDBY (voter only) and grants the
configured peer instead of starting (and winning) its own election.
2. Voter freshness gate compared the candidate only against the voter's LOCAL
config, ignoring the cluster's known committed state. Fix: ha_vote_self_freshness
floors the comparison at max(local, learned-committed config_tag / committed_lsn),
so a voter denies any candidate whose config/cluster state is outdated vs
what the cluster has committed. Strictly additive (never denies the freshest
node — Leader Completeness preserves liveness).
TDD: test_rdhup_noconfig_node_does_not_elect + configured-node contrast;
test_ha_vote_gate learned-floor deny + liveness + backward-compat. Full
controller suite green under ASAN. commit.c and ha-convergence-recovery
files untouched.
repl_active_standby_count: registry truth (ep != NULL && !in_catch_up)
under r->lock. ha_state_t gains repl back-pointer (wired at init by
task 5.3); ha_has_write_quorum uses registry truth when wired, exact
legacy fallback when NULL; formula unchanged. Test 29 (wired gate,
catch-up-only standby → NOT_READY, admits after clear) captured RED
pre-fix. 29/29 group tests; suite at the 3-failure baseline.
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
- ha_state_t.n_peers now matches raft_config_t.n_peers convention:
total cluster size (including self), consistently across both layers
- ha_start_election_async: ha->n_peers = n_peers (candidates_count)
- Convergence/counting loops: i < n_peers - 1 (skip self for standby arrays)
- Single-node: n_peers=1, threshold=0 (same behavior, clear semantics)
- ha_elect_ctx_t.n_peers unchanged (separate field for vote array sizing)
- Tests: 3-node n_peers=2->3, single-node 0->1, loop bounds updated
E2E test 6.7: 3-node cluster lifecycle verified
- Initial: ctrl-1 PRIMARY via majority convergence, no degraded mode
- Kill primary: RDHUP detected on both standbys, ctrl-2 elected at term=4
- No data loss: config tag preserved across failover
- Full ASAN build green
6.3: Fix single-node bug — ha_compute_safe_committed_lsn now returns
self_lsn instead of 0 for n_peers=0 (prevents WAL truncation on
single-node re-election). Add self_lsn/self_tag to ha_elect_ctx_t,
populated unconditionally in ha_start_election_async. All 3 test
scenarios pass under ASAN.
6.4: Add test_snapshot_recv_128kb_two_chunks (128KB snapshot in
2x64KB chunks, verifies size and applied_lsn); 4 other test cases
already existed from tasks 3.5c/3.5d. All ASAN-clean.
- Change HA_DEFAULT_COMMITTING_TIMEOUT_MS from 10000 to 60000
- committing_timeout_fn: count converged/pending standbys from
standby_converged[], log WARN with counts, then promote (degraded mode)
- Replace hardcoded 10000 in test_ha_evaluate_committing.c with macro
- Three new tests: default=60000, degrade-promote with 1/2 and 0/2 converged
- Add ha_repl_send_snapshot_hook seam in ha.c/ha.h for XCE_TESTING
- test_dispatch_repl_action_send_snapshot: verifies SEND_SNAPSHOT
decision triggers hook when standby_lsn < oldest_wal_lsn
- test_snapshot_recv_two_chunks_assembles_and_renames: verifies standby
assembles snap.part across chunks and renames on isLast=true
- test_snapshot_recv_reconnect_discards_snappart: verifies O_TRUNC
semantics on chunk_index==0 restart after reconnect
- All tests pass under ASAN
ha_stream_wal_to_standby() reads WAL from committed_lsn range, accumulates
entries to 256/64KB per batch, enqueues all via xce_request() (non-blocking).
Partial final batch sent immediately. Wired into STREAM_WAL case.
3 tests: 500-entries/2-batches, partial batch, empty range.
Fixed stale task-number comments per review.
ha_dispatch_repl_action_for_standby() calls push_config_to_standbys()
when PUSH_CONFIG decision. Hook seam for unit tests. Wired into
repl_ack_decode_and_dispatch after ha_process_standby_state.
oldest_wal_lsn=0 placeholder documented for task 3.4.
ha_evaluate_repl_action() pure function: stale tag→PUSH_CONFIG,
lsn<oldest_wal→SEND_SNAPSHOT, lsn<committed→STREAM_WAL, else CONVERGED.
5 tests. Fixed misleading stream_wal_from_zero test name per review.
OPCode_nop handler now sends stateReport {applied_lsn, config_tag}
instead of nodata. Function in ha.c for unit-test linkability.
Two tests: warm state and zero-state fresh standby (Req 1.4).
Replace stub with ha_apply_wal_catchup_entry() calling apply_wal_entry().
CAS-advance applied_lsn on success; log ERROR and skip on failure.
Standby stays in CTRL_STANDBY throughout. 3 new unit tests.
Add standby_reported_tag/lsn/converged arrays to ha_state_t, sized
MAX_HA_CANDIDATES. Zero-init via existing memset in ha_init().
Unit test verifies zero-init and writability.
Add time_t last_activity to ha_state_t for standby liveness timestamp tracking.
Add time_t last_push_time to standby_ep_ctx_t for primary idle NOP suppression.
Zero-initialize last_activity in ha_init(); last_push_time zeroed by calloc.
Add test_ha_init_last_activity_is_zero to test_ha_startup.c.
Task 7.1 (unit tests) was already complete with 8 tests in
test_ha_connection_reuse.c. Added two new tests:
Task 7.2 — test_follower_passive_no_outbound_connect (Req 2.1, 2.4):
Verifies raft_cb_become_follower sets role=CTRL_STANDBY and nulls all
votes[i].ep. The old fetch_config_from_primary path is gone; standbys
never initiate outbound connects after election loss.
Task 7.3 — test_promoted_ep_repl_ack_promotes_to_primary (Req 5.2, 5.4):
Injects two OPCode_replCatchUpDone packets via peer_ctrl_msg_handler in
CTRL_PRIMARY_COMMITTING with n_peers=4 (5-node cluster, quorum=2 ACKs).
Verifies repl_ack_count reaches 2 and role transitions to CTRL_PRIMARY.
Added vote_ep_ctx_set_cluster_for_test() seam so vctx->c is non-NULL
when ha_repl_ack_received fires ha_promote_to_primary.
All 62 controller tests pass; ASAN clean.
spec: .kiro/specs/ha-peer-connection-reuse/tasks.md all tasks [x]
Non-blocking election initiator that returns immediately after registering
vote endpoints with iomgr and scheduling the election timeout watchdog.
- Single-node shortcircuit (n_peers==1): allocates self-only ha_elect_ctx_t,
calls ha_elect_evaluate_and_signal directly, frees ctx inline — no iomgr
or delaymgr involvement; returns in < 1ms
- Multi-node: allocates ha_elect_ctx_t, per-peer vote_ep_ctx_t, calls
endpoint_connect + iomgr_monitor for each peer; schedules election timeout
via delay_task_ms; skips peer on connect failure (counted as denied)
- election_timeout_fn: delaymgr callback that fires ha_elect_evaluate_and_signal
after election_timeout_ms, unless cancelled flag is set (lock-free cancellation)
- vote_ep_err_handler: counts unreachable peers as denied votes
- Declares ha_start_election_async in ha.h with full doc comment
- 5 Unity tests in test_ha_start_election_async.c: returns < 5ms, stores
n_peers, shortcircuit fires immediately, n_peers stored for multi-peer,
term incremented — all GREEN with ASAN enabled
Task 2.1 of election-iomgr-refactor spec. Adds the new ha_state_t fields
required for the COMMITTING phase FSM, the CTRL_PRIMARY_COMMITTING role enum
value, and the committing_timeout_ms config field with its default constant.
ha_init() explicitly zero-initialises all new fields. Unit test verifies
zero-init and enum distinctness.
Requirements: 5.2, 8.1 (election-iomgr-refactor spec)
Two Raft correctness fixes:
1. Priority staggering in ha_wait_for_heartbeat_or_timeout():
Add my_priority × election_priority_step_ms to the timeout so
ctrl-a (priority=0) times out first and starts election before
ctrl-b (priority=1) and ctrl-c (priority=2). This gives the
highest-priority node a head start and prevents simultaneous
campaigning that causes perpetual split votes.
2. candidate_id in vote RPC:
Pass the actual node name (myconf->name) as candidate_id instead
of "". This fixes the voted_for deduplication gate (Req 2.6):
a peer that already voted for ctrl-a will correctly deny a
second vote request from ctrl-b in the same term.
Thread context gets my_name field; am_i_primary_quorum() takes
my_name parameter; all call sites updated (myapp.c, ha.c RDHUP,
test_ha_election.c stubs).
- sem_timedwait on heartbeat_seen semaphore with CLOCK_REALTIME deadline
- deadline = now + election_timeout_ms + xorshift64 jitter
- Returns CTRL_STANDBY on heartbeat arrival (calls ha_do_demote)
- Returns CTRL_ELECTING on timeout (caller drives election)
- Allows OPCode_heartbeat through dispatch gate in CTRL_INIT state
- 3 new unit tests: standby-on-heartbeat, electing-on-timeout, gate-allows-heartbeat
- 24 Tests 0 Failures in test_ha_startup suite
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
(only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
(replication, election, and housekeeping ops pass; primary-only ops
return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
switch; gate failure sends error reply and returns early
All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
