Add src/tools/test-heartbeat-tolerance.sh: automated bash script that
deploys a 3-node cluster and runs two SIGSTOP/SIGCONT scenarios:
Scenario A (1 × heartbeat_interval_ms pause): verifies no RDHUP fires on
standbys — liveness_poll_fn must not trigger failover for a short silence
within the tolerance window (Req 2.2).
Scenario B (heartbeat_timeout_ms + 500ms pause): verifies exactly one RDHUP
fires on each standby — the activity-based poll fires after sustained silence
and does not loop (Req 2.3, Req 5.4).
Add .kiro/specs/ha-heartbeat-liveness/manual-tests.md: step-by-step manual
procedure, expected log patterns, and results template for both scenarios.
Mark task 4.5 complete in tasks.md.
Call ha_start_primary_heartbeat in raft_cb_become_leader when entering
CTRL_PRIMARY_COMMITTING so standbys receive NOPs during the committing
window. Fixes 60s stability: 0 false RDHUP events on standbys.
Verification task: confirmed config_repl.c handle_ctrlhello calls both
ha_on_heartbeat_received and ha_arm_heartbeat_timeout_vtask (delaymgr poll).
Confirmed ha_on_primary_ctrl_fd_rdhup leads with ha_disarm (Req 5.4).
No pthread_create in heartbeat arm path (Req 1.6). Add thread-count test.
Extend primary_heartbeat_fn to read last_push_time per standby and send NOP
only when elapsed >= heartbeat_interval_s. on_standby_connected writes
last_push_time before ctrlhello send. Remove workaround xce_nop_req at end
of on_standby_connected. Add ha_nop_req_hook XCE_TESTING seam. 3 new tests.
Remove hb_vtask_t, hb_timeout_thread, s_hb_vtask from ha.c. Replace with
liveness_poll_ctx_t + delaymgr liveness_poll_fn: checks cancelled flag and
role==CTRL_STANDBY before acting; re-arms at heartbeat_interval_ms/2.
ha_on_primary_ctrl_fd_rdhup calls ha_disarm_heartbeat_timeout_vtask first
to prevent duplicate failover (Req 5.4). ha_on_heartbeat_received writes
last_activity via clock_gettime. Add test_ha_liveness.c with 7 unit tests.
Enforce heartbeat_timeout_ms >= 2 * heartbeat_interval_ms at startup.
validate_heartbeat_config() in startup_validation.c returns -EINVAL with
error message on violation; wired in cluster_init_phase1 before ha_init().
4 unit tests added to test_ha_startup.c.
Add time_t last_activity to ha_state_t for standby liveness timestamp tracking.
Add time_t last_push_time to standby_ep_ctx_t for primary idle NOP suppression.
Zero-initialize last_activity in ha_init(); last_push_time zeroed by calloc.
Add test_ha_init_last_activity_is_zero to test_ha_startup.c.
Task 7.1 (unit tests) was already complete with 8 tests in
test_ha_connection_reuse.c. Added two new tests:
Task 7.2 — test_follower_passive_no_outbound_connect (Req 2.1, 2.4):
Verifies raft_cb_become_follower sets role=CTRL_STANDBY and nulls all
votes[i].ep. The old fetch_config_from_primary path is gone; standbys
never initiate outbound connects after election loss.
Task 7.3 — test_promoted_ep_repl_ack_promotes_to_primary (Req 5.2, 5.4):
Injects two OPCode_replCatchUpDone packets via peer_ctrl_msg_handler in
CTRL_PRIMARY_COMMITTING with n_peers=4 (5-node cluster, quorum=2 ACKs).
Verifies repl_ack_count reaches 2 and role transitions to CTRL_PRIMARY.
Added vote_ep_ctx_set_cluster_for_test() seam so vctx->c is non-NULL
when ha_repl_ack_received fires ha_promote_to_primary.
All 62 controller tests pass; ASAN clean.
spec: .kiro/specs/ha-peer-connection-reuse/tasks.md all tasks [x]
cleanup.sh is generated by deploy-ha.sh and already does the right thing:
keeps ctrl-1/conf/* and all certs, wipes ctrl-{2..N}/conf, all data/, and logs.
Update skill and CLAUDE.md to use it as the primary clean-restart command
instead of manual rm loops.
- .claude/skills/xce-deploy-ha/SKILL.md: new skill for deploying a
local 3-node XCE HA controller cluster. Covers build prerequisites,
deploy via src/tools/deploy-ha.sh, node startup with -vvv raft trace,
health verification checks, clean restart procedure, and a
troubleshooting table for common failure modes.
- src/controller/CLAUDE.md: add HA Cluster Deployment section with
quick-deploy commands, health check one-liners, and reference to the
xce-deploy-ha skill. Add src/tools/deploy-ha.sh to key files.
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
- ha.c: add raft_cb_trace callback wired to s_real_cbs.on_trace; emits
RAFT[pN] term=T event=E at DEBUG3 level, visible with -vvv flag.
- ha.c: repl_ack_decode_and_dispatch now silently drops OPCode_nop and
OPCode_reply instead of WARNING — heartbeat NOP replies from standbys
arrive on promoted endpoints and are expected.
- deploy-ha.sh: fix --vcpu -> --vcpus; add \$VERBOSE passthrough so
start.sh respects VERBOSE=-vvv from the environment.
- ha.c: arm heartbeat timeout watchdog after ha_find_and_connect_to_primary (all 3 tiers)
- ha.c: guard ha_arm_heartbeat_timeout_vtask with c!=NULL (prevents ASAN stack-use-after-return in unit tests)
- ha.c: add priority-based jitter to ha_on_primary_ctrl_fd_rdhup to avoid split-brain
- ha.c: clear voted_for before RDHUP jitter so check reflects new votes, not stale startup votes
