- Gap analysis: all 10 requirements verified implemented in live codebase
(ha.c, ha.h, config_repl.c, request.c).
- Fix parent group headers (1-10) to [x] since all leaf subtasks complete.
- Mark task 11.5 complete: /xce-build controller passes 1/1 tests, ASAN clean.
- Remaining: integration tests 11.1-11.4 (manual cluster tests, documented
in manual-tests.md).
Add src/tools/test-heartbeat-tolerance.sh: automated bash script that
deploys a 3-node cluster and runs two SIGSTOP/SIGCONT scenarios:
Scenario A (1 × heartbeat_interval_ms pause): verifies no RDHUP fires on
standbys — liveness_poll_fn must not trigger failover for a short silence
within the tolerance window (Req 2.2).
Scenario B (heartbeat_timeout_ms + 500ms pause): verifies exactly one RDHUP
fires on each standby — the activity-based poll fires after sustained silence
and does not loop (Req 2.3, Req 5.4).
Add .kiro/specs/ha-heartbeat-liveness/manual-tests.md: step-by-step manual
procedure, expected log patterns, and results template for both scenarios.
Mark task 4.5 complete in tasks.md.
Call ha_start_primary_heartbeat in raft_cb_become_leader when entering
CTRL_PRIMARY_COMMITTING so standbys receive NOPs during the committing
window. Fixes 60s stability: 0 false RDHUP events on standbys.
Verification task: confirmed config_repl.c handle_ctrlhello calls both
ha_on_heartbeat_received and ha_arm_heartbeat_timeout_vtask (delaymgr poll).
Confirmed ha_on_primary_ctrl_fd_rdhup leads with ha_disarm (Req 5.4).
No pthread_create in heartbeat arm path (Req 1.6). Add thread-count test.
Extend primary_heartbeat_fn to read last_push_time per standby and send NOP
only when elapsed >= heartbeat_interval_s. on_standby_connected writes
last_push_time before ctrlhello send. Remove workaround xce_nop_req at end
of on_standby_connected. Add ha_nop_req_hook XCE_TESTING seam. 3 new tests.
Remove hb_vtask_t, hb_timeout_thread, s_hb_vtask from ha.c. Replace with
liveness_poll_ctx_t + delaymgr liveness_poll_fn: checks cancelled flag and
role==CTRL_STANDBY before acting; re-arms at heartbeat_interval_ms/2.
ha_on_primary_ctrl_fd_rdhup calls ha_disarm_heartbeat_timeout_vtask first
to prevent duplicate failover (Req 5.4). ha_on_heartbeat_received writes
last_activity via clock_gettime. Add test_ha_liveness.c with 7 unit tests.
Enforce heartbeat_timeout_ms >= 2 * heartbeat_interval_ms at startup.
validate_heartbeat_config() in startup_validation.c returns -EINVAL with
error message on violation; wired in cluster_init_phase1 before ha_init().
4 unit tests added to test_ha_startup.c.
Add time_t last_activity to ha_state_t for standby liveness timestamp tracking.
Add time_t last_push_time to standby_ep_ctx_t for primary idle NOP suppression.
Zero-initialize last_activity in ha_init(); last_push_time zeroed by calloc.
Add test_ha_init_last_activity_is_zero to test_ha_startup.c.
Task 7.1 (unit tests) was already complete with 8 tests in
test_ha_connection_reuse.c. Added two new tests:
Task 7.2 — test_follower_passive_no_outbound_connect (Req 2.1, 2.4):
Verifies raft_cb_become_follower sets role=CTRL_STANDBY and nulls all
votes[i].ep. The old fetch_config_from_primary path is gone; standbys
never initiate outbound connects after election loss.
Task 7.3 — test_promoted_ep_repl_ack_promotes_to_primary (Req 5.2, 5.4):
Injects two OPCode_replCatchUpDone packets via peer_ctrl_msg_handler in
CTRL_PRIMARY_COMMITTING with n_peers=4 (5-node cluster, quorum=2 ACKs).
Verifies repl_ack_count reaches 2 and role transitions to CTRL_PRIMARY.
Added vote_ep_ctx_set_cluster_for_test() seam so vctx->c is non-NULL
when ha_repl_ack_received fires ha_promote_to_primary.
All 62 controller tests pass; ASAN clean.
spec: .kiro/specs/ha-peer-connection-reuse/tasks.md all tasks [x]
