- ha.c: arm heartbeat timeout watchdog after ha_find_and_connect_to_primary (all 3 tiers)
- ha.c: guard ha_arm_heartbeat_timeout_vtask with c!=NULL (prevents ASAN stack-use-after-return in unit tests)
- ha.c: add priority-based jitter to ha_on_primary_ctrl_fd_rdhup to avoid split-brain
- ha.c: clear voted_for before RDHUP jitter so check reflects new votes, not stale startup votes
Non-blocking election initiator that returns immediately after registering
vote endpoints with iomgr and scheduling the election timeout watchdog.
- Single-node shortcircuit (n_peers==1): allocates self-only ha_elect_ctx_t,
calls ha_elect_evaluate_and_signal directly, frees ctx inline — no iomgr
or delaymgr involvement; returns in < 1ms
- Multi-node: allocates ha_elect_ctx_t, per-peer vote_ep_ctx_t, calls
endpoint_connect + iomgr_monitor for each peer; schedules election timeout
via delay_task_ms; skips peer on connect failure (counted as denied)
- election_timeout_fn: delaymgr callback that fires ha_elect_evaluate_and_signal
after election_timeout_ms, unless cancelled flag is set (lock-free cancellation)
- vote_ep_err_handler: counts unreachable peers as denied votes
- Declares ha_start_election_async in ha.h with full doc comment
- 5 Unity tests in test_ha_start_election_async.c: returns < 5ms, stores
n_peers, shortcircuit fires immediately, n_peers stored for multi-peer,
term incremented — all GREEN with ASAN enabled
Replace stub with full iomgr-native vote reply decoder:
- Reads RequestVoteReply fields into local scalars BEFORE capn_free (UAF fix)
- Term fencing: atomic update ha->term if peer term > ours
- Error path: decode failure counts peer as denied vote
- Test seam: xce_vote_reply_hook for unit testing without real packets
- 4 new ASAN-clean unit tests: field decode, denied vote, term fencing, no-UAF
The pre-existing vote_request_thread UAF in the old pthread path remains
but is targeted for removal in task 5.3 (old code removal).
- vote_ep_ctx_t: per-peer context (ha, c, ctx, peer_idx, term/tag/lsn/name)
- election_timeout_ctx_t: cancelled flag + context for delay watchdog
- on_vote_connected(): iomgr callback fires after SSL handshake; builds
xce_request_t{opcode=OPCode_requestVote}, calls xce_request, sets msg_handler
- vote_reply_msg_handler stub (returns 0; implemented in task 2.3)
- Test seam: xce_vote_request_hook for unit testing without live SSL
- 4 new unit tests: opcode, field encoding, msg_handler set, cancelled flag
Context note: vctx->c is NULL in test helper — task 2.3 must use real cluster_t*
Task 2.1 of election-iomgr-refactor spec. Adds the new ha_state_t fields
required for the COMMITTING phase FSM, the CTRL_PRIMARY_COMMITTING role enum
value, and the committing_timeout_ms config field with its default constant.
ha_init() explicitly zero-initialises all new fields. Unit test verifies
zero-init and enum distinctness.
Requirements: 5.2, 8.1 (election-iomgr-refactor spec)
ha->ssl_ctx is created with TLS_server_method() (for incoming iomgr
connections). vote_request_thread was calling SSL_connect() on this
server-method context, which OpenSSL rejects immediately — causing
every vote thread to fail with SSL_ERROR_SSL and goto deny.
Fix: create a fresh TLS_client_method() context with SSL_VERIFY_NONE
for each outgoing vote connection. This is safe because the election
protocol is authenticated by the vote RPC payload (term, config_tag)
not by the TLS certificate chain.
The SSL_CTX_free after SSL_new is correct — SSL_new increments the
reference count internally, so freeing the ctx leaves the ssl object
valid.
Two Raft correctness fixes:
1. Priority staggering in ha_wait_for_heartbeat_or_timeout():
Add my_priority × election_priority_step_ms to the timeout so
ctrl-a (priority=0) times out first and starts election before
ctrl-b (priority=1) and ctrl-c (priority=2). This gives the
highest-priority node a head start and prevents simultaneous
campaigning that causes perpetual split votes.
2. candidate_id in vote RPC:
Pass the actual node name (myconf->name) as candidate_id instead
of "". This fixes the voted_for deduplication gate (Req 2.6):
a peer that already voted for ctrl-a will correctly deny a
second vote request from ctrl-b in the same term.
Thread context gets my_name field; am_i_primary_quorum() takes
my_name parameter; all call sites updated (myapp.c, ha.c RDHUP,
test_ha_election.c stubs).
The vote_request_thread was sending raw capnp(RequestVoteRequest) bytes
but the server-side iomgr decodes every incoming packet as a Request
struct. This caused the term field (uint64) to be misread as the opcode
field (uint16), so vote RPCs were silently misrouted (term=1 → OPCode_reply,
term=2 → OPCode_hello) and no node ever got a vote from its peers.
Fix:
- controller.capnp: add requestvotemsg @27 to Request.body and
requestvotereplymsg @14 to RequestReply.data
- typemap.h: move request_vote_req_t/reply_t before xce_request_t;
add requestvotemsg/requestvotereplymsg to body/data unions
- ha.c: rewrite vote_request_thread send/recv to use xce_encode()
wrapper and decode outer Request→inner RequestReply on receive
- cluster.c: add case OPCode_requestVote handler calling
handle_request_vote() and replying via xce_reply()
All 188 unit tests pass.
- sem_timedwait on heartbeat_seen semaphore with CLOCK_REALTIME deadline
- deadline = now + election_timeout_ms + xorshift64 jitter
- Returns CTRL_STANDBY on heartbeat arrival (calls ha_do_demote)
- Returns CTRL_ELECTING on timeout (caller drives election)
- Allows OPCode_heartbeat through dispatch gate in CTRL_INIT state
- 3 new unit tests: standby-on-heartbeat, electing-on-timeout, gate-allows-heartbeat
- 24 Tests 0 Failures in test_ha_startup suite
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
(only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
(replication, election, and housekeeping ops pass; primary-only ops
return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
switch; gate failure sends error reply and returns early
All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
- Implement validate_peer_count() in startup_validation.c: -EINVAL for even N>1
- Implement validate_cert_files() with err_buf output: access() check, writes
missing path into err_buf, returns -ENOENT
- Wire validate_peer_count() in main.c after peer list parse; fatal exit on fail
- Wire validate_cert_files() in main.c before SSL init; fatal exit on fail
- Fix ASAN leak: SSL_CTX_free in ha_fini() with NULL guard
- Fix double-free: null c->ha.ssl_ctx in cluster_fini() before ha_fini(),
so iomgr_fini() retains sole SSL_CTX ownership (design per ha.h comment)
- All 96 tests pass; all 12 ha_startup tests GREEN
Root cause: test_heartbeat_stale_no_crash_without_primary had
heartbeat_timeout_ms=100 < heartbeat_interval_ms=3000, causing ha_init
to return -EINVAL before memset. The test ignored the return value and
called ha_fini on garbage stack memory. The watchdog_thread field
happened to hold the value 3 (from a prior stack frame), so
ha_stop_watchdog called pthread_join(3, NULL) — crash at 0x2d3
(__pthread_clockjoin_ex accessing thread-struct field at offset 0x2d0).
Fix 1 (test): valid config (interval=2000, timeout=3000). Staleness is
injected via last_heartbeat -= 10s, so timeout value was irrelevant.
Added TEST_ASSERT_EQUAL_INT check on ha_init return.
Fix 2 (ha_init): move memset before validation so ha_fini is safe to
call even after a failed init — prevents the same bug class in callers
that check ha_init's return late.
Verified: test_ha_role 11/11 PASS in both non-ASAN and ASAN builds.
- Add HA_DEFAULT_* constants to ha.h; use them in cluster_init()
- Strip stale task-label comments (P1-3a, P1-5, P1-6, P3-6 refs)
- Remove self-documenting 'Step N:' annotations from ha_do_promote/demote
- Rewrite stale forward-reference notes as current WHY comments
- Remove resolved TODO P0-8b from cluster.c
Add test-only function ha_trigger_manual_failover_for_test() to exercise manual
failover state machine (demote → promote cycle) without requiring live cluster_t.
Test coverage:
- test_manual_failover_transitions_role_and_increments_term: Validates role
transitions (PRIMARY → STANDBY → PRIMARY) and term monotonicity
- test_manual_failover_resets_hooks_registered: Confirms demotion resets the
hooks_registered idempotency guard
- test_manual_failover_monotonic_term_across_multiple_cycles: Validates term
remains monotonically increasing across 5 consecutive failover cycles
Implementation:
- Exposed only under #ifdef XCE_TESTING in ha.h
- Function body (6 lines): calls ha_do_demote() then ha_do_promote(); returns
the new term value via atomic_load_explicit()
- Enables design validation (P3-6) without test-cluster infrastructure
All tests pass. Build clean.
Add redirect helpers to ha.c/ha.h so a standby controller never accepts
join requests as if it were the primary:
- ha_should_redirect(): returns 1 when role is STANDBY and primary_host/port
are known — caller should send E_XCE_REDIRECT.
- ha_no_primary_known(): returns 1 when role is STANDBY but primary address
is not yet known — caller should send E_XCE_NOMASTER (retry later).
- Add E_XCE_NOMASTER and E_XCE_REDIRECT to errnox.h.
- Wire both guards into handle_join_() in join.c (before the role switch)
so all join types (worker, scheduler, apiserver, client) are protected.
- Full redirect response sends status=-E_XCE_REDIRECT with nodata body;
the primary host:port are logged so operators can trace misrouted clients.
Full Cap'n Proto RedirectInfo response body is deferred to a follow-up
(schema union would require regenerating capnp bindings).
5 new TDD tests — 110 total, 0 failures.
- ha_do_promote(): atomic_fetch_add term, set role=PRIMARY, reset
last_received_crc to WAL_CRC_NONE (COH-5 chain restart on promotion)
- ha_do_demote(): set role=STANDBY, reset hooks_registered idempotency guard
- ha_install_sigusr1(): sigaction handler stores only atomic int —
async-signal-safe per POSIX.1-2017 §2.4.3; global g_sigusr1_ha set
once before signals are enabled
- ha_promote_to_primary() / ha_demote_to_standby() remain (void)c stubs
pending P1-5 cluster wiring
Tests: 3 new cases in test_ha_role.c
test_promote_increments_term: term > term_before, role == PRIMARY
test_demote_sets_standby_role: role reverts to STANDBY after promote+demote
test_sigusr1_sets_failover_requested: raise(SIGUSR1) sets atomic flag
Result: 100 Tests 0 Failures 0 Ignored (controller suite)
- Add ha_start_watchdog() / ha_stop_watchdog() — dedicated pthread with
nanosleep(heartbeat_interval_ms) loop; stop via atomic watchdog_stop +
pthread_join (no pthread_cancel, safe for OpenSSL)
- Add ha_check_heartbeat_nocluster() — reads role with
atomic_load_explicit(memory_order_acquire), guards on primary_host[0]=='\0',
computes elapsed_ms from CLOCK_MONOTONIC, sets failover_requested on timeout
- Update ha_fini() to call ha_stop_watchdog() before mutex destroy
- 3 new tests (TDD RED→GREEN): watchdog starts/stops, no-op when no primary,
stale heartbeat no crash; all 97 controller tests green
- Full ha.h: ctrl_role_t enum (CTRL_PRIMARY/CTRL_STANDBY), ha_config_t,
ha_state_t with _Atomic fields (term, role, applied_lsn, promotion
counters, watchdog_stop), pthread_mutex_t lock, and public API.
- endpoint_t* omitted from ha_state_t — including netio.h would pull
OpenSSL headers and risk circular deps; watchdog (P1-3) includes
netio.h directly where needed.
- ha_init validates: heartbeat_interval_ms >= 2000, heartbeat_timeout_ms
>= heartbeat_interval_ms, election_timeout_ms >= 500; returns -EINVAL
on bad config; zeros struct and sets initial role to CTRL_STANDBY.
- ha_fini destroys mutex; watchdog thread join deferred to P1-3.
- am_i_primary / ha_promote_to_primary / ha_demote_to_standby are
no-op stubs; cluster_t->ha field not yet wired (P1-5), so these avoid
incomplete-type dereferences until then.
- TDD: 5 new tests in test_ha_role.c confirmed RED before implementation,
all 5 GREEN after; all prior controller tests remain green.
