Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
Add src/tools/test-heartbeat-tolerance.sh: automated bash script that
deploys a 3-node cluster and runs two SIGSTOP/SIGCONT scenarios:
Scenario A (1 × heartbeat_interval_ms pause): verifies no RDHUP fires on
standbys — liveness_poll_fn must not trigger failover for a short silence
within the tolerance window (Req 2.2).
Scenario B (heartbeat_timeout_ms + 500ms pause): verifies exactly one RDHUP
fires on each standby — the activity-based poll fires after sustained silence
and does not loop (Req 2.3, Req 5.4).
Add .kiro/specs/ha-heartbeat-liveness/manual-tests.md: step-by-step manual
procedure, expected log patterns, and results template for both scenarios.
Mark task 4.5 complete in tasks.md.
- ha.c: add raft_cb_trace callback wired to s_real_cbs.on_trace; emits
RAFT[pN] term=T event=E at DEBUG3 level, visible with -vvv flag.
- ha.c: repl_ack_decode_and_dispatch now silently drops OPCode_nop and
OPCode_reply instead of WARNING — heartbeat NOP replies from standbys
arrive on promoted endpoints and are expected.
- deploy-ha.sh: fix --vcpu -> --vcpus; add \$VERBOSE passthrough so
start.sh respects VERBOSE=-vvv from the environment.
