fix(ha): election-safety hardening — vote gates, restored-vote term, freshness source
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
This commit is contained in:
parent
fa66d1ba9c
commit
9821ec4f3c
7 changed files with 207 additions and 91 deletions
|
|
@ -151,7 +151,7 @@ typedef struct {
|
|||
_Atomic uint64_t applied_lsn; /* Standby: last applied WAL LSN */
|
||||
char primary_host[XCE_MAXNAME_LEN]; /* Hostname of current primary */
|
||||
uint32_t primary_port; /* Port of current primary */
|
||||
struct timespec last_heartbeat; /* CLOCK_MONOTONIC timestamp of last heartbeat */
|
||||
/* last_heartbeat deleted — heartbeat freshness now derives from last_activity */
|
||||
_Atomic int promotion_failed;
|
||||
_Atomic int promotion_retry_count;
|
||||
_Atomic int in_catch_up; /* Non-zero while processing historical WAL entries */
|
||||
|
|
@ -159,7 +159,7 @@ typedef struct {
|
|||
int hooks_registered; /* Idempotency guard for xce_add_hook calls */
|
||||
uint32_t last_received_crc; /* Chained CRC of last received replicated entry */
|
||||
ha_config_t config;
|
||||
pthread_mutex_t lock; /* Protects primary_host/port + last_heartbeat */
|
||||
pthread_mutex_t lock; /* Protects primary_host/port + voted_for_term/name */
|
||||
/* watchdog_stop / watchdog_thread deleted — replaced by ha_arm_heartbeat_timeout_vtask */
|
||||
void *ssl_ctx; /* SSL_CTX* cast; avoids pulling openssl headers here */
|
||||
uint64_t jitter_seed; /* xorshift64 PRNG state for election jitter */
|
||||
|
|
@ -441,6 +441,21 @@ int ha_handle_snapshot_chunk(ha_state_t *ha, persist_t *p,
|
|||
*/
|
||||
void ha_persist_term(ha_state_t *ha, const char *workdir);
|
||||
|
||||
/*
|
||||
* ha_init_raft: initialize the embedded raft state machine with cluster topology.
|
||||
*
|
||||
* Must be called ONCE at controller startup after config is loaded and
|
||||
* ha_restore_term() has run. Subsequent elections (initial and re-elections
|
||||
* after primary loss) reuse the same raft state — only raft_start_election()
|
||||
* and raft_on_leader_lost() are called for each election cycle.
|
||||
*
|
||||
* Calling raft_init() more than once would reset raft's internal state
|
||||
* (term, voted_for, role) and could lose durable election progress.
|
||||
*
|
||||
* n_peers: total cluster size including self (candidates_count from config).
|
||||
*/
|
||||
void ha_init_raft(ha_state_t *ha, cluster_t *c, int n_peers);
|
||||
|
||||
/*
|
||||
* ha_restore_term: read {workdir}/ha-term.bin at startup and restore
|
||||
* ha->term and ha->voted_for_* into ha_state_t.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue