repl_active_standby_count: registry truth (ep != NULL && !in_catch_up)
under r->lock. ha_state_t gains repl back-pointer (wired at init by
task 5.3); ha_has_write_quorum uses registry truth when wired, exact
legacy fallback when NULL; formula unchanged. Test 29 (wired gate,
catch-up-only standby → NOT_READY, admits after clear) captured RED
pre-fix. 29/29 group tests; suite at the 3-failure baseline.
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
- ha_state_t.n_peers now matches raft_config_t.n_peers convention:
total cluster size (including self), consistently across both layers
- ha_start_election_async: ha->n_peers = n_peers (candidates_count)
- Convergence/counting loops: i < n_peers - 1 (skip self for standby arrays)
- Single-node: n_peers=1, threshold=0 (same behavior, clear semantics)
- ha_elect_ctx_t.n_peers unchanged (separate field for vote array sizing)
- Tests: 3-node n_peers=2->3, single-node 0->1, loop bounds updated
E2E test 6.7: 3-node cluster lifecycle verified
- Initial: ctrl-1 PRIMARY via majority convergence, no degraded mode
- Kill primary: RDHUP detected on both standbys, ctrl-2 elected at term=4
- No data loss: config tag preserved across failover
- Full ASAN build green
6.3: Fix single-node bug — ha_compute_safe_committed_lsn now returns
self_lsn instead of 0 for n_peers=0 (prevents WAL truncation on
single-node re-election). Add self_lsn/self_tag to ha_elect_ctx_t,
populated unconditionally in ha_start_election_async. All 3 test
scenarios pass under ASAN.
6.4: Add test_snapshot_recv_128kb_two_chunks (128KB snapshot in
2x64KB chunks, verifies size and applied_lsn); 4 other test cases
already existed from tasks 3.5c/3.5d. All ASAN-clean.
- Change HA_DEFAULT_COMMITTING_TIMEOUT_MS from 10000 to 60000
- committing_timeout_fn: count converged/pending standbys from
standby_converged[], log WARN with counts, then promote (degraded mode)
- Replace hardcoded 10000 in test_ha_evaluate_committing.c with macro
- Three new tests: default=60000, degrade-promote with 1/2 and 0/2 converged
- Add ha_repl_send_snapshot_hook seam in ha.c/ha.h for XCE_TESTING
- test_dispatch_repl_action_send_snapshot: verifies SEND_SNAPSHOT
decision triggers hook when standby_lsn < oldest_wal_lsn
- test_snapshot_recv_two_chunks_assembles_and_renames: verifies standby
assembles snap.part across chunks and renames on isLast=true
- test_snapshot_recv_reconnect_discards_snappart: verifies O_TRUNC
semantics on chunk_index==0 restart after reconnect
- All tests pass under ASAN
ha_stream_wal_to_standby() reads WAL from committed_lsn range, accumulates
entries to 256/64KB per batch, enqueues all via xce_request() (non-blocking).
Partial final batch sent immediately. Wired into STREAM_WAL case.
3 tests: 500-entries/2-batches, partial batch, empty range.
Fixed stale task-number comments per review.
ha_dispatch_repl_action_for_standby() calls push_config_to_standbys()
when PUSH_CONFIG decision. Hook seam for unit tests. Wired into
repl_ack_decode_and_dispatch after ha_process_standby_state.
oldest_wal_lsn=0 placeholder documented for task 3.4.
ha_evaluate_repl_action() pure function: stale tag→PUSH_CONFIG,
lsn<oldest_wal→SEND_SNAPSHOT, lsn<committed→STREAM_WAL, else CONVERGED.
5 tests. Fixed misleading stream_wal_from_zero test name per review.
OPCode_nop handler now sends stateReport {applied_lsn, config_tag}
instead of nodata. Function in ha.c for unit-test linkability.
Two tests: warm state and zero-state fresh standby (Req 1.4).
Replace stub with ha_apply_wal_catchup_entry() calling apply_wal_entry().
CAS-advance applied_lsn on success; log ERROR and skip on failure.
Standby stays in CTRL_STANDBY throughout. 3 new unit tests.
Add standby_reported_tag/lsn/converged arrays to ha_state_t, sized
MAX_HA_CANDIDATES. Zero-init via existing memset in ha_init().
Unit test verifies zero-init and writability.
Add time_t last_activity to ha_state_t for standby liveness timestamp tracking.
Add time_t last_push_time to standby_ep_ctx_t for primary idle NOP suppression.
Zero-initialize last_activity in ha_init(); last_push_time zeroed by calloc.
Add test_ha_init_last_activity_is_zero to test_ha_startup.c.
Task 7.1 (unit tests) was already complete with 8 tests in
test_ha_connection_reuse.c. Added two new tests:
Task 7.2 — test_follower_passive_no_outbound_connect (Req 2.1, 2.4):
Verifies raft_cb_become_follower sets role=CTRL_STANDBY and nulls all
votes[i].ep. The old fetch_config_from_primary path is gone; standbys
never initiate outbound connects after election loss.
Task 7.3 — test_promoted_ep_repl_ack_promotes_to_primary (Req 5.2, 5.4):
Injects two OPCode_replCatchUpDone packets via peer_ctrl_msg_handler in
CTRL_PRIMARY_COMMITTING with n_peers=4 (5-node cluster, quorum=2 ACKs).
Verifies repl_ack_count reaches 2 and role transitions to CTRL_PRIMARY.
Added vote_ep_ctx_set_cluster_for_test() seam so vctx->c is non-NULL
when ha_repl_ack_received fires ha_promote_to_primary.
All 62 controller tests pass; ASAN clean.
spec: .kiro/specs/ha-peer-connection-reuse/tasks.md all tasks [x]
Non-blocking election initiator that returns immediately after registering
vote endpoints with iomgr and scheduling the election timeout watchdog.
- Single-node shortcircuit (n_peers==1): allocates self-only ha_elect_ctx_t,
calls ha_elect_evaluate_and_signal directly, frees ctx inline — no iomgr
or delaymgr involvement; returns in < 1ms
- Multi-node: allocates ha_elect_ctx_t, per-peer vote_ep_ctx_t, calls
endpoint_connect + iomgr_monitor for each peer; schedules election timeout
via delay_task_ms; skips peer on connect failure (counted as denied)
- election_timeout_fn: delaymgr callback that fires ha_elect_evaluate_and_signal
after election_timeout_ms, unless cancelled flag is set (lock-free cancellation)
- vote_ep_err_handler: counts unreachable peers as denied votes
- Declares ha_start_election_async in ha.h with full doc comment
- 5 Unity tests in test_ha_start_election_async.c: returns < 5ms, stores
n_peers, shortcircuit fires immediately, n_peers stored for multi-peer,
term incremented — all GREEN with ASAN enabled
Task 2.1 of election-iomgr-refactor spec. Adds the new ha_state_t fields
required for the COMMITTING phase FSM, the CTRL_PRIMARY_COMMITTING role enum
value, and the committing_timeout_ms config field with its default constant.
ha_init() explicitly zero-initialises all new fields. Unit test verifies
zero-init and enum distinctness.
Requirements: 5.2, 8.1 (election-iomgr-refactor spec)
Two Raft correctness fixes:
1. Priority staggering in ha_wait_for_heartbeat_or_timeout():
Add my_priority × election_priority_step_ms to the timeout so
ctrl-a (priority=0) times out first and starts election before
ctrl-b (priority=1) and ctrl-c (priority=2). This gives the
highest-priority node a head start and prevents simultaneous
campaigning that causes perpetual split votes.
2. candidate_id in vote RPC:
Pass the actual node name (myconf->name) as candidate_id instead
of "". This fixes the voted_for deduplication gate (Req 2.6):
a peer that already voted for ctrl-a will correctly deny a
second vote request from ctrl-b in the same term.
Thread context gets my_name field; am_i_primary_quorum() takes
my_name parameter; all call sites updated (myapp.c, ha.c RDHUP,
test_ha_election.c stubs).
- sem_timedwait on heartbeat_seen semaphore with CLOCK_REALTIME deadline
- deadline = now + election_timeout_ms + xorshift64 jitter
- Returns CTRL_STANDBY on heartbeat arrival (calls ha_do_demote)
- Returns CTRL_ELECTING on timeout (caller drives election)
- Allows OPCode_heartbeat through dispatch gate in CTRL_INIT state
- 3 new unit tests: standby-on-heartbeat, electing-on-timeout, gate-allows-heartbeat
- 24 Tests 0 Failures in test_ha_startup suite
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
(only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
(replication, election, and housekeeping ops pass; primary-only ops
return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
switch; gate failure sends error reply and returns early
All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
