- Strip handle_ctrlhello: remove repl_add_standby, WAL streaming,
snapshot check, push_config_to_standbys, and CtrlHelloReply
- Keep only: set primary_ctrl_fd, call ha_on_heartbeat_received,
arm ha_arm_heartbeat_timeout_vtask, log INFO
- All replication decisions now handled by heartbeat decision loop
- Test: call with NULL repl + config_tag mismatch → no crash, no repl
messages enqueued, primary_ctrl_fd set correctly
- In raft_cb_become_leader, after persist_truncate_wal_above(safe_lsn),
store safe_lsn to ha->last_known_committed_lsn via atomic release
- Test: 3-node voters [100,150,200] → safe_lsn=150; asserts
ha->last_known_committed_lsn == 150 after raft_cb_become_leader_for_test
- Add ha_repl_send_snapshot_hook seam in ha.c/ha.h for XCE_TESTING
- test_dispatch_repl_action_send_snapshot: verifies SEND_SNAPSHOT
decision triggers hook when standby_lsn < oldest_wal_lsn
- test_snapshot_recv_two_chunks_assembles_and_renames: verifies standby
assembles snap.part across chunks and renames on isLast=true
- test_snapshot_recv_reconnect_discards_snappart: verifies O_TRUNC
semantics on chunk_index==0 restart after reconnect
- All tests pass under ASAN
- SEND_SNAPSHOT uses checkpoint_lsn (not committed_lsn) for file lookup
- ha_send_snapshot_to_standby stats file for actual total_size
- ha_compute_oldest_wal_lsn scans WAL dir for min segment LSN
- Removed hardcoded 131072 and oldest_wal_lsn=0 placeholders
ha_stream_wal_to_standby() reads WAL from committed_lsn range, accumulates
entries to 256/64KB per batch, enqueues all via xce_request() (non-blocking).
Partial final batch sent immediately. Wired into STREAM_WAL case.
3 tests: 500-entries/2-batches, partial batch, empty range.
Fixed stale task-number comments per review.
ha_dispatch_repl_action_for_standby() calls push_config_to_standbys()
when PUSH_CONFIG decision. Hook seam for unit tests. Wired into
repl_ack_decode_and_dispatch after ha_process_standby_state.
oldest_wal_lsn=0 placeholder documented for task 3.4.
ha_evaluate_repl_action() pure function: stale tag→PUSH_CONFIG,
lsn<oldest_wal→SEND_SNAPSHOT, lsn<committed→STREAM_WAL, else CONVERGED.
5 tests. Fixed misleading stream_wal_from_zero test name per review.
OPCode_nop handler now sends stateReport {applied_lsn, config_tag}
instead of nodata. Function in ha.c for unit-test linkability.
Two tests: warm state and zero-state fresh standby (Req 1.4).
Extend ha_apply_wal_catchup_entry with batch branch:
- Check n_entries > 0, iterate entries[] list in order
- Apply each via apply_wal_entry(); skip NULL/errors without crash
- Advance applied_lsn once after full batch (not per-entry)
- n_entries == 0 falls through to task 2.1 single-entry path
4 new unit tests covering full batch, partial batch, error skip, fallback
Replace stub with ha_apply_wal_catchup_entry() calling apply_wal_entry().
CAS-advance applied_lsn on success; log ERROR and skip on failure.
Standby stays in CTRL_STANDBY throughout. 3 new unit tests.
Add standby_reported_tag/lsn/converged arrays to ha_state_t, sized
MAX_HA_CANDIDATES. Zero-init via existing memset in ha_init().
Unit test verifies zero-init and writability.
Verification task: confirmed config_repl.c handle_ctrlhello calls both
ha_on_heartbeat_received and ha_arm_heartbeat_timeout_vtask (delaymgr poll).
Confirmed ha_on_primary_ctrl_fd_rdhup leads with ha_disarm (Req 5.4).
No pthread_create in heartbeat arm path (Req 1.6). Add thread-count test.
Extend primary_heartbeat_fn to read last_push_time per standby and send NOP
only when elapsed >= heartbeat_interval_s. on_standby_connected writes
last_push_time before ctrlhello send. Remove workaround xce_nop_req at end
of on_standby_connected. Add ha_nop_req_hook XCE_TESTING seam. 3 new tests.
Remove hb_vtask_t, hb_timeout_thread, s_hb_vtask from ha.c. Replace with
liveness_poll_ctx_t + delaymgr liveness_poll_fn: checks cancelled flag and
role==CTRL_STANDBY before acting; re-arms at heartbeat_interval_ms/2.
ha_on_primary_ctrl_fd_rdhup calls ha_disarm_heartbeat_timeout_vtask first
to prevent duplicate failover (Req 5.4). ha_on_heartbeat_received writes
last_activity via clock_gettime. Add test_ha_liveness.c with 7 unit tests.
Enforce heartbeat_timeout_ms >= 2 * heartbeat_interval_ms at startup.
validate_heartbeat_config() in startup_validation.c returns -EINVAL with
error message on violation; wired in cluster_init_phase1 before ha_init().
4 unit tests added to test_ha_startup.c.
Add time_t last_activity to ha_state_t for standby liveness timestamp tracking.
Add time_t last_push_time to standby_ep_ctx_t for primary idle NOP suppression.
Zero-initialize last_activity in ha_init(); last_push_time zeroed by calloc.
Add test_ha_init_last_activity_is_zero to test_ha_startup.c.
Task 7.1 (unit tests) was already complete with 8 tests in
test_ha_connection_reuse.c. Added two new tests:
Task 7.2 — test_follower_passive_no_outbound_connect (Req 2.1, 2.4):
Verifies raft_cb_become_follower sets role=CTRL_STANDBY and nulls all
votes[i].ep. The old fetch_config_from_primary path is gone; standbys
never initiate outbound connects after election loss.
Task 7.3 — test_promoted_ep_repl_ack_promotes_to_primary (Req 5.2, 5.4):
Injects two OPCode_replCatchUpDone packets via peer_ctrl_msg_handler in
CTRL_PRIMARY_COMMITTING with n_peers=4 (5-node cluster, quorum=2 ACKs).
Verifies repl_ack_count reaches 2 and role transitions to CTRL_PRIMARY.
Added vote_ep_ctx_set_cluster_for_test() seam so vctx->c is non-NULL
when ha_repl_ack_received fires ha_promote_to_primary.
All 62 controller tests pass; ASAN clean.
spec: .kiro/specs/ha-peer-connection-reuse/tasks.md all tasks [x]
Non-blocking election initiator that returns immediately after registering
vote endpoints with iomgr and scheduling the election timeout watchdog.
- Single-node shortcircuit (n_peers==1): allocates self-only ha_elect_ctx_t,
calls ha_elect_evaluate_and_signal directly, frees ctx inline — no iomgr
or delaymgr involvement; returns in < 1ms
- Multi-node: allocates ha_elect_ctx_t, per-peer vote_ep_ctx_t, calls
endpoint_connect + iomgr_monitor for each peer; schedules election timeout
via delay_task_ms; skips peer on connect failure (counted as denied)
- election_timeout_fn: delaymgr callback that fires ha_elect_evaluate_and_signal
after election_timeout_ms, unless cancelled flag is set (lock-free cancellation)
- vote_ep_err_handler: counts unreachable peers as denied votes
- Declares ha_start_election_async in ha.h with full doc comment
- 5 Unity tests in test_ha_start_election_async.c: returns < 5ms, stores
n_peers, shortcircuit fires immediately, n_peers stored for multi-peer,
term incremented — all GREEN with ASAN enabled
Replace stub with full iomgr-native vote reply decoder:
- Reads RequestVoteReply fields into local scalars BEFORE capn_free (UAF fix)
- Term fencing: atomic update ha->term if peer term > ours
- Error path: decode failure counts peer as denied vote
- Test seam: xce_vote_reply_hook for unit testing without real packets
- 4 new ASAN-clean unit tests: field decode, denied vote, term fencing, no-UAF
The pre-existing vote_request_thread UAF in the old pthread path remains
but is targeted for removal in task 5.3 (old code removal).
- vote_ep_ctx_t: per-peer context (ha, c, ctx, peer_idx, term/tag/lsn/name)
- election_timeout_ctx_t: cancelled flag + context for delay watchdog
- on_vote_connected(): iomgr callback fires after SSL handshake; builds
xce_request_t{opcode=OPCode_requestVote}, calls xce_request, sets msg_handler
- vote_reply_msg_handler stub (returns 0; implemented in task 2.3)
- Test seam: xce_vote_request_hook for unit testing without live SSL
- 4 new unit tests: opcode, field encoding, msg_handler set, cancelled flag
Context note: vctx->c is NULL in test helper — task 2.3 must use real cluster_t*
Task 2.1 of election-iomgr-refactor spec. Adds the new ha_state_t fields
required for the COMMITTING phase FSM, the CTRL_PRIMARY_COMMITTING role enum
value, and the committing_timeout_ms config field with its default constant.
ha_init() explicitly zero-initialises all new fields. Unit test verifies
zero-init and enum distinctness.
Requirements: 5.2, 8.1 (election-iomgr-refactor spec)
Two Raft correctness fixes:
1. Priority staggering in ha_wait_for_heartbeat_or_timeout():
Add my_priority × election_priority_step_ms to the timeout so
ctrl-a (priority=0) times out first and starts election before
ctrl-b (priority=1) and ctrl-c (priority=2). This gives the
highest-priority node a head start and prevents simultaneous
campaigning that causes perpetual split votes.
2. candidate_id in vote RPC:
Pass the actual node name (myconf->name) as candidate_id instead
of "". This fixes the voted_for deduplication gate (Req 2.6):
a peer that already voted for ctrl-a will correctly deny a
second vote request from ctrl-b in the same term.
Thread context gets my_name field; am_i_primary_quorum() takes
my_name parameter; all call sites updated (myapp.c, ha.c RDHUP,
test_ha_election.c stubs).
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
