lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/src/controller/ha.h
Rongsong Shen 74784b2b18 fix(raft-ha-core-safety): randomize per-campaign election timeout to [T,2T] (task 5.7) 
The per-campaign election watchdog was armed with a FIXED election_timeout_ms
(raft_start_election -> on_schedule_timeout -> raft_cb_schedule_timeout). Under
cold-connection churn a just-restarted survivor opens a fresh TCP+SSL endpoint
per vote round, so its single needed grant can land a hair AFTER a constant
deadline on EVERY campaign -- a resonance: the campaign times out, the term
re-escalates, and the now-stale grant is dropped (DropStaleResponse). Forever.
Terms climb unbounded; the freshest survivor never tallies quorum. This is the
final round-4 chaotic-soak election-liveness stall.

Fix (Ongaro thesis 3.4): re-randomize the watchdog per campaign to the canonical
[T, 2T] interval in the HA adapter, so some campaign's deadline falls AFTER its
grant RTT and the grant lands in-window. Done in raft_cb_schedule_timeout so the
pure (sim-tested) raft library stays deterministic. Independent of
election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter)
so it spreads even when jmax is small/0; never below the configured floor so
genuine-loss failover is not sped up past election_timeout_ms.

Preserves Election Safety (no change to term/vote/DropStaleResponse logic).

new ha_jittered_election_delay() helper + 2 unit tests (RED verified: stubbing
jitter to 0 fails test_5_7_campaign_timeout_is_randomized with the resonance
message; PIN guards floor + modulo-by-zero). Suite: 24 Tests / 0 Failures.
2026-06-30 13:29:15 +00:00

1741 lines
92 KiB
C
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#ifndef HA_H
#define HA_H
/*
* XCE Controller HA — role management and election (P1).
*
* P1-1 audit: the existing create_ssl_context() + iomgr_set_ssl_ctx() path
* in cluster.c (line 559, set at line 583) covers all controller-to-controller SSL.
* HA listeners (heartbeat, election probe, replication accept) reuse the same
* ssl_ctx already installed on c->iomgr.
* No new TLS infrastructure is needed — XCE runs on a trusted management LAN.
* Peer identity is validated by host:port match against the --peers candidate list.
* See design doc Part 5.9 and memory: Persistence & HA Trusted LAN Decision.
*
* P1-2: full struct definitions and API.
* endpoint_t* (primary_ep) is intentionally omitted here — including netio.h
* would pull in heavy OpenSSL headers and create circular dependency risks.
* The watchdog (P1-3) will include netio.h directly where it needs endpoint_t.
*/
#include <pthread.h>
#include <semaphore.h>
#include <stdbool.h>
#include <stdatomic.h>
#include <stdint.h>
#include <time.h>
#include "config.h" /* XCE_MAXNAME_LEN, MAX_STANDBYS, etc. */
#include "myconf.h" /* myconf_t, config_version_tag */
#include "persist.h" /* persist_t, apply_wal_entry */
#include "typemap.h" /* wal_catchup_entry_t, state_report_t, xce_reply_t */
#include "xce/cluster.h" /* xce_candidate_t */
#include "zarray.h"
#include "raft/raft.h" /* raft_state_t, raft_init */
/* Forward declaration — avoids circular include with cluster_def.h */
typedef struct cluster_s cluster_t;
/* ── Inline PRNG for election jitter ───────────────────────── */
static inline uint64_t xorshift64_next(uint64_t *state)
{
uint64_t x = *state;
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
*state = x;
return x;
}
/*
* ha_seed_jitter: seed the xorshift64 PRNG from /dev/urandom (8 bytes).
* Falls back to CLOCK_MONOTONIC ^ priority ^ pid if urandom is unavailable.
* Not inline — contains syscalls (open/read/close/clock_gettime/getpid).
*/
void ha_seed_jitter(uint64_t *seed, int my_priority);
/* ── Role enum ─────────────────────────────────────────────── */
typedef enum {
CTRL_PRIMARY = 0,
CTRL_STANDBY = 1,
CTRL_INIT = 2, /* Startup: listener open, awaiting heartbeat or election timeout */
CTRL_ELECTING = 3 /* Running quorum election via OP_requestVote */
} ctrl_role_t;
/* ── Election types ────────────────────────────────────────── */
/*
* ha_vote_entry_t: per-peer state accumulated by the election candidate.
* One entry per peer in ha_elect_ctx_t.votes[] (flexible array).
*/
/* Forward declaration to avoid pulling netio.h into ha.h */
struct endpoint_t;
typedef struct {
xce_candidate_t *peer; /* Peer being solicited */
uint64_t my_term; /* Term sent in OP_requestVote */
uint64_t my_tag; /* config_tag sent in request */
uint64_t my_lsn; /* WAL LSN sent in request */
int vote_granted; /* 1 if peer granted vote */
int replied; /* 1 if peer replied (granted or denied) */
uint64_t peer_tag; /* peer's config_tag from reply */
uint64_t peer_lsn; /* peer's WAL LSN from reply */
struct endpoint_t *ep; /* Outgoing vote endpoint; NULL if connect failed or not yet connected */
} ha_vote_entry_t;
/*
* ha_elect_ctx_t: shared context for one quorum election round.
* Allocated as: calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))
* Stored in ha_state_t.elect_ctx (void * _Atomic) during election; NULL otherwise.
*/
typedef struct {
int n_peers; /* Total peer count (excluding self) */
int votes_granted; /* Votes received; starts at 1 (self-vote) */
int n_replied; /* Number of peers that replied */
uint64_t max_resp_tag; /* Highest config_tag seen in any reply */
uint64_t max_resp_lsn; /* Highest WAL LSN seen among max_resp_tag peers */
int result; /* 1=primary, 0=standby, <0=error */
int done_posted; /* Guard: non-zero once sem_post(&done) has been called */
pthread_mutex_t lock; /* Protects votes_granted/n_replied/max_resp_tag/max_resp_lsn/result */
sem_t done; /* Startup thread blocks; dispatch posts when done */
/* Fix 4 (HIGH): pointer back to the election_timeout_ctx_t so that task 4.1
* (quorum-reached path) can set timeout_ctx->cancelled = 1 to cancel the
* watchdog. Set by ha_start_election_async after scheduling the timeout. */
void *timeout_ctx; /* election_timeout_ctx_t*; NULL until watchdog scheduled */
/* Task 4.1: cluster_t* so ha_elect_evaluate_and_signal can call
* cluster_init_phase2(c).
* NULL in unit tests (c=NULL bypasses phase2). */
cluster_t *c;
/* Task 6.3: candidate's own LSN/tag, stored unconditionally so that
* ha_compute_safe_committed_lsn can use them even for single-node elections
* where votes[] is empty (n_peers=0). */
uint64_t self_lsn; /* Candidate's own WAL LSN at election start */
uint64_t self_tag; /* Candidate's own config_tag at election start */
ha_vote_entry_t votes[]; /* C99 flexible array: n_peers-1 entries */
} ha_elect_ctx_t;
/* ── HA configuration ──────────────────────────────────────── */
/* Default values used by cluster_init(). Tuned for a LAN with ≤5ms RTT. */
#define HA_DEFAULT_HEARTBEAT_INTERVAL_MS 3000
#define HA_DEFAULT_HEARTBEAT_TIMEOUT_MS 10000
#define HA_DEFAULT_ELECTION_TIMEOUT_MS 5000
#define HA_DEFAULT_PROBE_CONNECT_TIMEOUT_MS 500
/* Log the first connect failure per peer, then every Nth attempt. */
#define HA_CONNECT_LOG_THROTTLE 10
#define HA_DEFAULT_MAX_PROMOTION_RETRIES 5
#define HA_DEFAULT_ELECTION_JITTER_MAX_MS 200
#define HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500
typedef struct {
uint32_t heartbeat_interval_ms; /* Primary heartbeat send interval (min 2000 ms) */
uint32_t heartbeat_timeout_ms; /* Standby timeout before promoting (>= interval) */
uint32_t election_timeout_ms; /* Election probe shared deadline (min 500 ms) */
uint32_t probe_connect_timeout_ms; /* Per-probe TCP connect timeout */
uint32_t max_promotion_retries; /* Max consecutive failed promotion attempts */
uint32_t election_jitter_max_ms; /* random jitter added to election timeout (default 200) */
uint32_t election_priority_step_ms; /* extra ms per priority level (default 500) */
} ha_config_t;
/* probe_ctx_t deleted — only used by am_i_primary() (election-iomgr-refactor) */
/* ── HA runtime state (embedded in cluster_t as `ha_state_t ha` in P1-5) ── */
typedef struct {
_Atomic uint64_t term; /* Election term — fencing token */
_Atomic int32_t role; /* Cast to ctrl_role_t at access sites */
int my_priority; /* Index in --peers list (lower = higher priority) */
_Atomic uint64_t applied_lsn; /* Standby: last applied WAL LSN */
char primary_host[XCE_MAXNAME_LEN]; /* Hostname of current primary */
uint32_t primary_port; /* Port of current primary */
/* last_heartbeat deleted — heartbeat freshness now derives from last_activity */
_Atomic int promotion_failed;
_Atomic int promotion_retry_count;
_Atomic int in_catch_up; /* Non-zero while processing historical WAL entries */
/* failover_requested deleted — watchdog replaced by ha_arm_heartbeat_timeout_vtask */
int hooks_registered; /* Idempotency guard for xce_add_hook calls */
uint32_t last_received_crc; /* Chained CRC of last received replicated entry */
ha_config_t config;
pthread_mutex_t lock; /* Protects primary_host/port + voted_for_term/name */
/* watchdog_stop / watchdog_thread deleted — replaced by ha_arm_heartbeat_timeout_vtask */
void *ssl_ctx; /* SSL_CTX* cast; avoids pulling openssl headers here */
uint64_t jitter_seed; /* xorshift64 PRNG state for election jitter */
zarray_t *standby_endpoints; /* connected standby controller endpoints for config push */
/* ── New fields added by ha-config-bootstrap spec ──────── */
uint64_t voted_for_term; /* Req 9: term of last granted vote */
char voted_for_name[XCE_MAXNAME_LEN]; /* Req 9: candidate of last granted vote */
int primary_ctrl_fd; /* Req 6: persistent channel fd (-1 = none) */
bool defer_config_fetch; /* Req 3: tag=0 node defers fetch to post-election */
void * _Atomic elect_ctx; /* Req 2: ha_elect_ctx_t* during election; NULL otherwise */
_Atomic uint64_t last_known_committed_lsn; /* Req 5: from primary heartbeats */
sem_t heartbeat_seen; /* Req 1: posted by heartbeat handler during CTRL_INIT */
_Atomic uint64_t last_known_committed_config_tag; /* Req 8: from primary heartbeats */
_Atomic uint64_t committed_config_tag; /* Req 8.1: set on quorum ACK */
/* ── New fields added by election-iomgr-refactor spec (Task 2.1) ── */
void *client_ssl_ctx; /* SSL_CTX* (TLS_client_method); created in myapp_run */
/* ── New field added by raft-drive-async spec (Task 3.1) ──────── */
void *pool; /* vtaskmgr_t* (cast; avoids pulling vtaskmgr.h
* into every ha.h consumer). Set once at init
* (cluster_init_phase2 → &c->vtaskmgr). Used ONLY
* by peer_ctrl_msg_handler when peer-control
* dispatch is enabled; NULL otherwise (gate OFF). */
int phase2_done; /* 1 after cluster_init_phase2 runs */
int n_peers; /* total cluster size (including self), matches raft convention */
/* ── New fields added by raft-library spec (Task 6.1) ──────── */
raft_state_t raft; /* Raft consensus state (wired in task 6.2) */
/* ── New fields added by raft-library spec (Task 6.2) ──────── */
cluster_t *raft_c; /* cluster_t* set by ha_start_election_async; used by raft callbacks */
void *raft_election_timeout_ctx; /* election_timeout_ctx_t* for cancel (task 6.2) */
/* ── New fields added by ha-heartbeat-liveness spec (Task 1.1) ── */
time_t last_activity; /* CLOCK_MONOTONIC seconds of last primary message; 0 = none yet */
/* ── New fields added by ha-repl-hardening spec (Task 1.4) ──────── */
uint64_t standby_reported_tag[MAX_HA_CANDIDATES]; /* Last config_tag reported by each standby */
uint64_t standby_reported_lsn[MAX_HA_CANDIDATES]; /* Last applied_lsn reported by each standby */
bool standby_converged[MAX_HA_CANDIDATES]; /* Whether standby[i] has converged with primary */
int peer_connect_fail_count[MAX_HA_CANDIDATES]; /* throttle log spam on repeated connect failure */
/* ── New field added by ha-commit-engine spec (Task 4.2) ──────── */
replicator_t *repl; /* Back-pointer to the standby registry (persist_t.repl).
* When set, ha_has_write_quorum counts registry truth via
* repl_active_standby_count (ep != NULL && !in_catch_up);
* NULL → legacy standby_endpoints->count fallback.
* (Req 3.1-3.3, 6.1, 6.2; wired at init by task 5.3.) */
/* ── New field added by ha-convergence-recovery spec (Task 3.x) ──────── */
_Atomic int rejoin_reconciled; /* Defensive once-guard for rejoin
* reconciliation (Req 2.1): set after the
* first truncate+self-restart so a second
* CtrlHelloReply in the same process
* lifetime cannot re-fire the restart.
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-convergence-recovery spec (Task 4.x) ──────── */
_Atomic int catchup_inflight[MAX_STANDBYS]; /* Per-standby single-flight
* guard for ACK-driven catch-up
* continuation (Req 5.2): set when a
* continuation is scheduled for a
* standby slot, cleared when it
* completes; a duplicate walAck while
* the flag is set schedules nothing.
* Slot identity (the repl registry slot
* index) is stable across schedule/run.
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-convergence-recovery spec (Task 5.x) ──────── */
_Atomic int quorum_lost; /* Quorum-transition edge detector (Req
* 8.1-8.3): 0 = quorum held (or unknown
* at init), 1 = quorum lost. Flipped by
* ha_log_quorum_transition via CAS so the
* loss/restore WARN is emitted exactly
* once per edge (no log storms across
* repeated dispatch-gate rejections).
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-commit-durability spec (Task 4.4) ──────── */
void * _Atomic promotion_fence_ctx; /* promotion_fence_ctx_t* for the
* bounded-window catch-up timer (Req
* 2.2/2.4): a recovery-truncated/behind
* member that wins an election with no
* reachable peer is NOT FATAL'd at the
* promotion instant; it is fenced
* (in_catch_up) and this one-shot timer
* is armed for DEFAULT_RECOVERY_CATCHUP_-
* WINDOW_MS. The 3.3 FATAL is reached
* ONLY from that timer iff still
* hollow + no peer + no catch-up
* progress when it fires; cleared/
* cancelled on recovery or shutdown so a
* caught-up or stopped member never
* FATALs. NULL at init / when no fence
* timer is pending. */
/* ── New field added by ha-term-safety spec (Task 4.3) ──────── */
_Atomic int config_serve_fence; /* Config byte-delivery serve-fence
* (R7.9): set when the latest committed
* WAL_CONFIG_MARKER references config
* bytes whose sha256 does NOT match the
* locally-held bytes (or the bytes are
* absent locally). While set, the member
* MUST NOT serve under the new
* configuration; the bytes travel
* out-of-band via the existing config-push
* transport, so the member STAYS ALIVE and
* config_activate_committed KEEPS RETRYING
* until the matching bytes arrive and
* hash-verify, at which point the fence is
* cleared and the config is activated.
* NEVER self-terminates (loud non-serving
* fence only). 0 at init (ha_init memset). */
/* ── New field added by raft-drive-async spec (Task 2.1) ──────── */
_Atomic int reconnect_inflight[MAX_HA_CANDIDATES]; /* Per-peer
* single-flight guard for the offloaded
* delaymgr reconnect (Req 2.5): set (CAS
* 0->1) when a reconnect to peer[i] is
* submitted to the slow lane, cleared by
* the lane worker on completion (success
* or failure) so the next heartbeat tick
* can retry. While set, the heartbeat
* reconnect scan enqueues no duplicate for
* that peer. Indexed by ABSOLUTE candidate
* index (peers_cfg.candidates[]). 0 at
* init (ha_init memset). */
/* ── New field added by raft-drive-async spec (Task 3.2) ──────── */
_Atomic int snapshot_inflight[MAX_HA_CANDIDATES]; /* Per-peer
* single-flight guard for the offloaded
* snapshot send (Req 1.5): set (CAS
* 0->1) when a snapshot to peer[i] is
* submitted to the slow lane, cleared by
* the lane worker on completion (success
* or failure). While set, the heartbeat
* decision loop enqueues no duplicate
* snapshot for that peer. Indexed by
* ABSOLUTE candidate index. 0 at
* init (ha_init memset). */
/* ── New field added by raft-drive-async spec (Task 3.3) ──────── */
_Atomic int configpush_inflight; /* CLUSTER-WIDE single-flight guard
* for the offloaded requester-less
* config push. Config push is a
* cluster-wide fan-out (one push thread
* per standby), so at most ONE is in
* flight at a time — hence a scalar, not
* a per-peer array. Set (CAS 0->1) when
* a config push is submitted to the slow
* lane, cleared by the lane worker on
* completion (success or failure). While
* set, the heartbeat decision loop /
* promotion enqueues no duplicate config
* push. 0 at init (ha_init memset). */
} ha_state_t;
/* ── Public API ────────────────────────────────────────────── */
/*
* ha_init: zero-initialise *ha, validate cfg, store config and my_priority.
* Returns 0 on success, -EINVAL on bad config.
* Initial role is always CTRL_STANDBY.
*/
int ha_init(ha_state_t *ha, const ha_config_t *cfg, int my_priority);
/*
* ha_fini: destroy mutex. Watchdog thread join added in P1-3.
*/
void ha_fini(ha_state_t *ha);
/* am_i_primary() deleted — replaced by ha_start_election_async (election-iomgr-refactor) */
/*
* ha_promote_to_primary: stub — atomically sets role to PRIMARY.
* Full promotion sequence (hooks, WAL flush, peer notify) added in P1-3a.
*/
int ha_promote_to_primary(cluster_t *c);
/*
* ha_demote_to_standby: stub — atomically sets role to STANDBY.
* Full demotion sequence added in P1-3a.
*/
int ha_demote_to_standby(cluster_t *c);
/* ── Internal promotion/demotion (P1-3a) ───────────────────── */
/*
* ha_do_promote: increment term, set role=PRIMARY, reset last_received_crc.
* No cluster_t dependency — fully testable standalone.
* Callers: ha_promote_to_primary() (P1-5), watchdog failover path.
*/
void ha_do_promote(ha_state_t *ha);
/*
* ha_do_demote: set role=STANDBY, reset hooks_registered idempotency guard.
* Hook disable requires cluster_t and is done in ha_demote_to_standby() (P1-5).
*/
void ha_do_demote(ha_state_t *ha);
/* ── Demotion convergence + self-restart primitive ─────────────
* (ha-convergence-recovery Requirement 1: a demoted ex-primary discards its
* uncommitted, potentially divergent state and rejoins from committed state
* only via truncation + process self-restart through boot recovery.)
*/
/*
* controller_saved_argv: accessor for the NULL-terminated argv captured by
* main() before argp_parse, used by ha_self_restart to re-exec the controller
* (ha-convergence-recovery Req 1.3, design "Self-Restart Primitive").
* Returns NULL if argv was never saved (e.g. unit-test binaries).
*/
char **controller_saved_argv(void);
/*
* ha_self_restart: the truncate-and-restart primitive's restart half.
*
* Production contract (Req 1.3, 1.4): flush logs, execv(controller_saved_argv())
* to rebuild in-memory state through boot recovery from the persisted committed
* state only; if exec cannot be performed, terminate FATALLY (abort) rather than
* continue running with potentially divergent state — NEVER returns in
* production.
*
* Test contract (Req 1.6): under XCE_TESTING, invoke ha_restart_hook instead of
* execv and return its result (0 on the success path); a failing ha_restart_hook
* routes to ha_fatal_hook in place of abort so tests can assert the FATAL path
* without dying.
*
* Preconditions: persistence hooks already disabled; truncation completed
* before this call. Invariant: no new WAL entry is appended by this node
* between hook-disable and exec.
*/
int ha_self_restart(cluster_t *c);
/*
* ha_primary_log_append (ha-term-safety task 3.1, R1.1/R1.2/R2.1): the single
* primary append entry point. Routes a local primary append (job-state entry or
* the own-term promotion no-op) through the raft library (raft_log_append) so
* the library stamps the live election term and assigns the LSN, firing
* on_persist_entry for the actual WAL write — making the library the sole
* LSN/term source and keeping p->current_lsn co-tracked with s->log_len. When
* the raft library is not RAFT_LEADER (the non-raft promotion road / engine-less
* tests), falls back to a legacy local persist_wal_async append (the library is
* not tracking the log, so no co-tracking hazard). `type` is a wal_entry_type;
* `payload`/`len` may be NULL/0 (e.g. WAL_NOOP). out_lsn (optional) receives the
* assigned 1-based LSN. Returns 0 on success; propagates a non-zero persist rc.
*/
int ha_primary_log_append(cluster_t *c, uint32_t type,
const void *payload, uint32_t len, uint64_t *out_lsn);
/*
* ha_raft_on_repl_ack (ha-term-safety task 3.3, R4.1/R4.5): drive the library on
* a replication ACK. Routes a successful WalAck through raft_on_append_entries_reply
* (updating the library's match_index[]/next_index[] at the absolute peer index)
* then raft_advance_commit, so the library recomputes the current-term-gated
* commit (the Figure-8 rule) and MAY fire on_commit_advanced — the task-2.4
* callback that feeds the existing synchronous commit barrier. A current-term
* quorum advances committed only when the agreed entry is current-term; a
* prior-term agreed entry is held; a stale-term reply is dropped by the library.
*
* `abs_peer_idx` is the ABSOLUTE library peer index (j != self_idx), the same
* index space as match_index[]/next_index[]. The CALLER resolves it per endpoint
* type via ha_repl_ack_abs_peer_idx — promoted-vote eps carry a relative votes
* index, heartbeat standby eps carry an already-absolute index. This seam must
* NOT remap again. An out-of-range / self index is rejected loudly (returns
* -EINVAL, no library accounting) so the library's unchecked match_index[] write
* is never corrupted (no false quorum, no OOB). `acked_lsn` is the standby's
* matched index (WalAckReply.lsn); `reply_term` is the reply's term.
*
* Leader-only effect: raft_advance_commit returns -EPERM off the leader (no
* commit change, no callback) — harmless on a node that is no longer primary.
* Non-blocking (ha-async-io): the library performs no I/O; on_commit_advanced
* only records the barrier ceiling. Returns 0 on success; -EINVAL on a
* no-cluster context or a rejected peer index. */
int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
uint64_t reply_term);
/*
* ha_repl_ack_abs_peer_idx (ha-term-safety task 3.3 remediation): resolve the
* ABSOLUTE library peer index from an ACK endpoint's stored peer_idx, branching
* on endpoint type (mirrors the standby_endpoints scan at ha.c:6187-6189):
* - ep_is_vote != 0 (promoted vote ep, vote_ep_ctx_t): peer_idx is a
* self-EXCLUDED votes index → remap abs = peer_idx < self_idx ? peer_idx
* : peer_idx+1.
* - ep_is_vote == 0 (heartbeat standby ep, standby_ep_ctx_t, ctx==NULL):
* peer_idx is ALREADY absolute → pass through.
* self_idx == ha->my_priority. Pure (no side effects); the single source of
* truth shared by the production ACK call site and the unit test so the
* remap-vs-passthrough distinction cannot drift. */
int ha_repl_ack_abs_peer_idx(ha_state_t *ha, int ep_is_vote, int peer_idx);
/*
* ha_schedule_demotion_convergence: schedule the one-shot delaymgr task that
* truncates above committed_lsn (Req 1.2), WARN-logs the discarded range
* (Req 1.5), and self-restarts (Req 1.3). Called from raft_cb_become_follower's
* was-PRIMARY branch AFTER the persistence hooks are disabled inline (Req 1.1),
* so all file I/O and the exec run off the become-follower callback thread (D1).
* Returns 0 on success, negative on alloc/schedule failure (suffix is then
* reconciled by rejoin reconciliation at the next join).
*/
int ha_schedule_demotion_convergence(cluster_t *c);
#ifdef XCE_TESTING
/*
* ha_demotion_convergence_fn: the delaymgr task body — exposed under XCE_TESTING
* only so the demotion-convergence suite can reference its contract; the test
* fixture drives the queued task through its stored entry pointer.
*/
int ha_demotion_convergence_fn(void *arg);
#endif
/*
* ha_rejoin_reconcile: converge a rejoining node whose local WAL log runs ahead
* of the cluster's committed boundary (ha-convergence-recovery Req 2.1–2.3,
* design "Rejoin Reconciliation", research D5).
*
* Called from the standby's CtrlHelloReply intake (cluster.c) after the existing
* last_known_committed_lsn store, with primary_committed_lsn = the boundary the
* primary reported.
*
* Contract (task 3.2 GREEN implements; task 3.1 ships a RED no-op stub):
* - local_tip = persist.current_lsn - 1.
* - If local_tip > primary_committed_lsn AND the rejoin_reconciled once-guard
* is clear: cutoff = max(primary_committed_lsn, local committed_lsn) — the
* CLAMP that never discards the locally known committed prefix (Req 2.2) —
* persist_truncate_wal_above(cutoff), WARN the discarded range, set the
* once-guard, then ha_self_restart (deferred to delaymgr if the intake runs
* on iomgr, same rule as demotion). Returns the restart result.
* - Else (tip <= boundary): return 0 with NO side effects (Req 2.3).
* - A second call in the same process lifetime takes no restart (Req 2.1 —
* defensive once-guard; "once" is also structural since post-restart the tip
* equals the boundary).
*
* Generalizes the RDHUP truncation pattern (ha_on_primary_ctrl_fd_rdhup), which
* remains in place unchanged for the failover path.
*/
int ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn);
/*
* cluster_apply_ctrlhello_reply: standby-side CtrlHelloReply intake body
* (ha-convergence-recovery task 3.1). Stores the primary-reported committed
* boundary + config tag; called from cluster.c's myreq_handler on every
* CtrlHelloReply. Defined in ha_convergence.c (a unit-linkable TU) so the
* rejoin intake-integration test can drive the real path. Task 3.2 adds the
* ha_rejoin_reconcile trigger after the store.
*/
void cluster_apply_ctrlhello_reply(cluster_t *c, uint64_t committed_lsn,
uint64_t config_tag);
/*
* ha_walack_on_ack: the post-ACK catch-up decision (ha-convergence-recovery
* task 4.2, Req 5.1, 5.2, 7.1, design D6). Called from the walAck branch of
* repl_ack_decode_and_dispatch (ha.c) AFTER repl_on_ack, and from the
* ha_walack_continuation_for_test shim so production and tests exercise the
* same logic. Defined in config_repl.c (alongside ha_stream_wal_to_standby and
* the continuation task body).
*
* target = max(committed_lsn, last_written) (same derivation as ha.c:2771-2776)
* if acked_lsn < target → schedule ONE catch-up continuation that streams
* [acked_lsn+1, target] on delaymgr (never inline on
* iomgr), UNLESS c->ha.catchup_inflight[peer_idx] is
* already set (per-standby single-flight, Req 5.2).
* if acked_lsn >= target → repl_complete_catchup(&c->persist.repl, ep) to
* re-include the standby in live replication (Req 7.1).
*
* c — cluster_t* (real persist + repl registry; never NULL).
* ep — repl registry endpoint key for the standby (endpoint_t* /
* repl_ep_t); identifies the standby slot for streaming + parity.
* peer_idx — 0-based standby slot index (into c->ha.catchup_inflight[]).
* acked_lsn— the LSN the standby just acknowledged.
*
* Returns 0 on the parity / single-flight-suppressed paths and on a successfully
* scheduled continuation; negative on a scheduling/allocation failure. Never
* performs file I/O itself (streaming runs in the deferred continuation task).
*/
int ha_walack_on_ack(cluster_t *c, void *ep, int peer_idx, uint64_t acked_lsn);
/*
* ha_log_quorum_transition: emit an operator-visible WARN on each write-quorum
* loss/restore EDGE (ha-convergence-recovery task 5.x, Req 8.1, 8.2, 8.3).
*
* Evaluates the current write-quorum state and compares it against the
* ha_state_t.quorum_lost edge-detector flag. On a TRANSITION (and only on a
* transition), it atomically flips the flag via CAS and emits exactly one WARN:
* - loss: "HA quorum lost: connected=%d required=%d" (Req 8.1)
* - restore: "HA quorum restored: connected=%d required=%d" (Req 8.2)
* The flag makes the logging an edge detector, NOT a latch: repeated calls while
* the state is unchanged emit nothing (Req 8.3 — no log storms across repeated
* dispatch-gate rejections), but a later loss/restore cycle logs again. The CAS
* guarantees once-per-transition under concurrency.
*
* Call sites (task 5.2): the standby_endpoints mutation points (promotion
* registration, RDHUP removal, on_standby_connected) plus the ha_dispatch_gate
* quorum-rejection branch as a backstop. Observability only — never gates an
* operation outcome.
*
* Defined in ha_convergence.c (a unit-linkable TU) so the quorum-transition
* tests can drive it directly against a real persist/repl fixture.
*
* RED scaffold (task 5.1): the current definition is a NO-OP — it performs no
* edge detection, flips no flag, and emits no WARN — so the quorum-transition
* tests FAIL until task 5.2 lands the real edge-detection logic and wires the
* call sites.
*/
void ha_log_quorum_transition(ha_state_t *ha);
/* ── Redirect helpers used by join.c ──────────────────────── */
/*
* ha_should_redirect: returns 1 if this controller should redirect join
* requests to the primary.
* Returns 0 if: (a) role is PRIMARY, or (b) primary_host is empty /
* primary_port is 0 (no primary known yet — client should retry).
*/
int ha_should_redirect(const ha_state_t *ha);
/*
* ha_no_primary_known: returns 1 if role is STANDBY and primary_host is
* empty or primary_port is 0. Caller should reply E_XCE_NOMASTER so the
* client knows to retry later rather than treating the error as permanent.
*/
int ha_no_primary_known(const ha_state_t *ha);
/*
* ha_is_primary: returns 1 if ha->role is CTRL_PRIMARY, 0 otherwise.
* Lock-free. Safe to call from any thread.
*/
int ha_is_primary(const ha_state_t *ha);
/* ha_check_priority_stepdown() deleted — replaced by quorum election (Req 2.1) */
/* start_heartbeat_listener() / stop_heartbeat_listener() deleted — never called (election-iomgr-refactor) */
/*
* ha_dispatch_gate: role-based operation filter for the vtaskmgr dispatch loop.
* Returns:
* 0 — operation is permitted for the current role
* -E_XCE_NOT_READY — role is CTRL_INIT or CTRL_ELECTING and opcode is not
* OP_requestVote or OP_probeConfigTag
* -E_XCE_STANDBY — role is CTRL_STANDBY and opcode is a primary-only op
* CTRL_PRIMARY always returns 0.
*/
int ha_dispatch_gate(ha_state_t *ha, int opcode);
/*
* ha_config_promotion_fenced: 1 when this member is config serve-fenced and
* therefore NON-PROMOTABLE — its latest committed WAL_CONFIG_MARKER references
* config bytes it does not hold/match, so it must not become authoritative
* CTRL_PRIMARY (ha-term-safety task 4.3, R7.9). 0 otherwise. The pre-promotion
* gate in raft_cb_become_leader consults this; a fenced member stays ALIVE and
* non-promotable until the correct bytes arrive out-of-band and re-verify clears
* the fence (controller-never-self-terminate). Pure atomic read (no I/O).
*/
int ha_config_promotion_fenced(const ha_state_t *ha);
/*
* ha_apply_wal_catchup_entry: apply a single WAL catch-up entry to local
* persistent storage and advance ha->applied_lsn on success.
*
* Called from cluster.c OPCode_walCatchupEntry dispatch handler (Task 2.1).
* On success (rc == 0): applied_lsn advances to entry->lsn+1 and a
* WalAckReply {term, lsn, success=1} is sent back to the primary via
* xce_reply(ep, c->myhost, &reply) (Task 3.1, Req 3.1).
* On error (rc < 0): logs ERROR, returns error; no ACK sent (Task 3.2).
* applied_lsn is in-memory only — not persisted to disk.
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK is skipped.
* Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
*/
int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
wal_catchup_entry_t *entry,
struct endpoint_t *ep, cluster_t *c);
/* ha_nop_reply_state_report — populate a heartbeat reply with stateReport.
* Fills *sr from ha->applied_lsn (atomic read) and myconf->config_version_tag,
* then sets reply_out->data_which = stateReportMsg and
* reply_out->data.state_report = sr.
* Caller must keep *sr live until xce_reply() serialises the reply. */
void ha_nop_reply_state_report(ha_state_t *ha, myconf_t *myconf,
state_report_t *sr, xce_reply_t *reply_out);
/* ── Task 4.1 (ha-repl-hardening): safe commit point computation ─────────────
*
* ha_compute_safe_committed_lsn: determine safe_committed_lsn and safe_config_tag
* from vote replies collected in ctx after winning an election.
*
* Collects LSNs from all granted voters (self + peers with vote_granted=1),
* sorts descending, and returns the majority-threshold element.
*
* single-node (n_peers=0): returns ctx->self_lsn and ctx->self_tag directly —
* self is the only majority; returning 0 would truncate all WAL on re-election.
*
* Requirements: 2.3 (ha-repl-hardening spec)
* Task 6.3: single-node fix (self_lsn/self_tag fields added to ha_elect_ctx_t)
*/
void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
uint64_t *safe_lsn_out,
uint64_t *safe_tag_out);
/* ── bug-1060: Leader Completeness guard for promotion-time WAL truncation ────
*
* ha_promotion_truncate_floor: clamp the post-election WAL truncation cutoff so
* a newly-elected leader NEVER truncates entries it holds durably.
*
* Returns max(safe_lsn, durable_tip). safe_lsn is the majority-threshold commit
* point from the vote replies (ha_compute_safe_committed_lsn); durable_tip is the
* leader's own durable WAL tip (persist_durable_floor). Because the cutoff is
* never below the leader's durable tip, persist_truncate_wal_above is a no-op on
* the leader's own durable log — preserving every entry committed by a prior
* leader (Raft §5.4 Leader Completeness). safe_lsn only governs removal of a
* divergent, non-durable tail STRICTLY ABOVE the leader's durable tip.
*
* Root cause it fixes: truncating to the bare majority element destroyed
* committed entries the new leader had acked as a standby (granted voters
* {self=47, peer=44} → safe_lsn=44 cut the leader's own committed 45..47),
* producing divergent WALs and a permanent prevote deadlock. */
uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip);
/* ── raft-drive-async task 1.1: peer-scaled liveness reservation ─────────────
*
* drive_async_liveness_reserved: PURE derivation of VTASK_LIVENESS_RESERVED —
* the number of vtask-pool workers kept free for liveness-critical handling
* (heartbeat/vote/ack) so they are never starved by blocking offloaded tasks.
*
* reserved = clamp(n_peers - 1, 1, n_vcpus - 1)
* = max(1, min(n_peers - 1, n_vcpus - 1))
*
* - n_peers is the TOTAL cluster size including self (matches ha->n_peers).
* Per-endpoint serialization (endpoint_block_tasks) caps in-flight work at
* one per peer connection, so worst-case concurrent liveness work is the
* n_peers-1 OTHER peers (one ack/vote each) — reserving n_peers-1 means no
* peer's liveness message queues behind a busy reserved worker (a LATENCY
* optimization).
* - Floor 1 is the hard CORRECTNESS invariant (deadlock-freedom): one
* guaranteed-free worker + liveness-never-gated + sub-ms cycling is
* deadlock-free regardless of peer count. Applied LAST so it always wins.
* - Ceiling n_vcpus-1 keeps >=1 REQ_BLOCK (commit) slot, so the commit cap
* (n_vcpus - reserved) is never 0.
*
* On an under-provisioned box (n_vcpus < n_peers) the ceiling clamps the
* result below n_peers-1; the result is still correct via fast cycling.
* The CALLER (controller init) is responsible for emitting a loud WARN in
* that case so the under-provisioning is operator-visible — this helper is
* pure and never logs. Controllers NEVER self-terminate on this condition. */
int drive_async_liveness_reserved(int n_peers, int n_vcpus);
/* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ──────────
*
* ha_promotion_gate_decision: PURE pre-promotion eligibility decision (Req 2.2,
* 2.5, 2.6). Maps the promoting member's recovery/durability state to the Boot
* Policy Decision Table outcome by building recover_policy_inputs_t and
* delegating to persist_recover_decision (DRY — the 3.2/3.3 decision table is
* the single source of truth; this gate does not re-derive it). Note the
* FENCE_UNRECOVERABLE terminal is a LOUD PERMANENT NON-promotable fence that
* STAYS ALIVE and keeps retrying catch-up — it is NEVER process exit
* (never-self-exit invariant, 2026-06-18):
* - recovery_truncated → the task-3.1 structural-corruption flag.
* - durable_floor → the local CLAMPED durable position (read AFTER the
* promotion-path persist_truncate_wal_above so the
* 2.2 truncate-clamp is reflected).
* - safe_hwm → the cluster committed high-water mark from the voter
* replies (ha_compute_safe_committed_lsn), NOT a stale
* ctrlhello.
* - peer_reachable → a healthy peer is reachable to catch up from.
* - catch_up_succeeded → catch-up reached a committed-consistent state.
* A member is "behind the HWM" iff durable_floor < safe_hwm. Returns
* RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}; a hollow node (truncated OR behind)
* is never ELIGIBLE until it catches up, regardless of quorum size (Req 2.5).
*/
recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
uint64_t durable_floor,
uint64_t safe_hwm,
bool peer_reachable,
bool catch_up_succeeded);
/* ha_evaluate_promotion_fence: live pre-promotion gate over ha->in_catch_up
* (Req 2.2, 2.6). Evaluates ha_promotion_gate_decision and applies the fence:
* - ELIGIBLE → clears the in_catch_up fence and returns ELIGIBLE (promotion
* may proceed; a member that has caught up past the committed
* boundary regains eligibility — Req 2.6).
* - FENCE → sets the in_catch_up fence (refuse promotion; catch up from a
* healthy peer) and returns FENCE.
* - FENCE_UNRECOVERABLE → routes through
* persist_recover_loud_fence_if_unrecoverable (logs LOUDLY at
* ERROR + sets the sticky persist recovery_unrecoverable flag),
* also sets the in_catch_up fence, and returns
* FENCE_UNRECOVERABLE. The process STAYS ALIVE, non-promotable,
* retrying catch-up — it NEVER self-exits (Req 2.4).
* Reuses the existing in_catch_up fence — does NOT invent a parallel fence. */
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool recovery_truncated,
uint64_t durable_floor,
uint64_t safe_hwm,
bool peer_reachable,
bool catch_up_succeeded);
/* ── ha-commit-durability task 4.4: catch-up re-alert window ─────────────────
* (never-self-exit rework 2026-06-18.)
*
* When a recovery-truncated / behind member wins an election with no reachable
* peer AT THAT INSTANT (a transient no-peer snapshot during aggressive
* failover), it FENCES (sets in_catch_up, does NOT become primary) and arms a
* delaymgr re-alert timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS. When the
* timer fires, if the member is STILL recovery-truncated/behind AND no healthy
* peer is reachable AND no catch-up progress was made, it RE-EMITS the loud
* operator alert (persist_recover_loud_fence_if_unrecoverable) and RE-ARMS —
* the process STAYS ALIVE and keeps retrying catch-up INDEFINITELY. It NEVER
* terminates: a controller must never self-exit as a designed response to an
* operational condition. Catch-up that brings durable_floor >= HWM releases
* the fence + clears the sticky flag and stops the re-alert (Req 2.6); shutdown
* cancels it too.
*
* ha_promotion_fence_window_ms: the re-alert interval in milliseconds.
* Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via
* ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */
uint32_t ha_promotion_fence_window_ms(void);
/* ha_cancel_promotion_fence_timer: cancel a pending re-alert timer (idempotent;
* no-op when none is armed). Called when the member catches up (fence released)
* or on shutdown, so a stopped member stops re-alerting. It never gated a
* process exit — the timer never terminates the process. */
void ha_cancel_promotion_fence_timer(ha_state_t *ha);
#ifdef XCE_TESTING
/* Test seam: override the re-alert window so the timer fires deterministically
* fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS default. XCE_TESTING
* only — compiled out of production. */
void ha_set_promotion_fence_window_ms(uint32_t ms);
#endif
/* ── Task 3.2: Unified replication decision enum and evaluator ────────────── */
/*
* ha_repl_action_t: result of ha_evaluate_repl_action().
*
* Encodes the single replication action the primary should take for a standby
* on the current heartbeat cycle, based solely on the standby's reported
* (config_tag, applied_lsn) vs the primary's committed state.
*
* Decision tree (design §Unified Heartbeat-Driven Protocol):
* standby_tag < committed_tag → PUSH_CONFIG
* standby_lsn < oldest_wal_lsn → SEND_SNAPSHOT
* standby_lsn < committed_lsn → STREAM_WAL
* tag >= committed AND lsn >= committed → CONVERGED
*
* Requirements: 2.2, 2.3, 2.4 (ha-repl-hardening spec)
*/
typedef enum {
HA_REPL_PUSH_CONFIG, /* standby config tag is behind — push config first */
HA_REPL_SEND_SNAPSHOT, /* standby LSN predates oldest WAL — transfer snapshot */
HA_REPL_STREAM_WAL, /* standby LSN is within WAL window — stream entries */
HA_REPL_CONVERGED, /* standby tag and LSN are both current */
} ha_repl_action_t;
/*
* ha_evaluate_repl_action: pure decision function — no side effects.
*
* Evaluates the standby's reported (standby_tag, standby_lsn) against the
* primary's committed state (committed_tag, committed_lsn, oldest_wal_lsn)
* and returns the single action the primary should take this heartbeat cycle.
*
* Arguments:
* standby_tag — config_tag last reported by this standby
* standby_lsn — applied_lsn last reported by this standby
* committed_tag — primary's committed config_tag
* committed_lsn — primary's committed WAL LSN
* oldest_wal_lsn — oldest retained WAL LSN (entries before this are gone)
*
* Requirements: 2.2, 2.3, 2.4 (ha-repl-hardening spec)
* Boundary: Primary Heartbeat Handler (ha.c)
*/
ha_repl_action_t ha_evaluate_repl_action(uint64_t standby_tag,
uint64_t standby_lsn,
uint64_t committed_tag,
uint64_t committed_lsn,
uint64_t oldest_wal_lsn);
/*
* ha_process_standby_state: update per-standby convergence tracking.
*
* Called on each heartbeat reply from a standby while the primary is in
* CTRL_PRIMARY. Stores the standby's reported
* (config_tag, applied_lsn) and evaluates whether the standby has converged
* with the primary's committed state.
*
* Convergence criteria (Req 1.3):
* reported_tag >= committed_tag AND reported_lsn >= committed_lsn
*
* Logs INFO on the first transition from not-converged → converged (Req 7.2).
* Regression (converged → not-converged) clears standby_converged[peer_idx]
* and re-evaluates (Req 2.5).
*
* peer_idx: 0-based index into ha->standby_reported_* arrays; must be in
* [0, MAX_HA_CANDIDATES).
*
* Requirements: 1.2, 2.2, 2.5 (ha-repl-hardening spec)
* Boundary: Primary Heartbeat Handler (ha.c)
*/
void ha_process_standby_state(ha_state_t *ha, int peer_idx,
uint64_t reported_tag, uint64_t reported_lsn,
uint64_t committed_tag, uint64_t committed_lsn);
/* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */
/*
* ha_handle_snapshot_chunk: receive one snapshot transfer chunk on the standby.
*
* Each chunk's payload bytes are written (or appended) to snap.part in snap_dir.
* When chunk->is_last is non-zero:
* 1. Verify total bytes written == chunk->total_size.
* 2. Atomically rename snap.part → snap-{snapshot_lsn:016x}.bin.
* 3. Load the snapshot via persist_load_snapshot().
* 4. Set ha->applied_lsn = p->checkpoint_lsn (in-memory only).
*
* On mid-transfer reconnect (caller resets state), snap.part is discarded.
* Returns 0 on success, negative errno on error.
*
* Requirements: 4.2, 4.3, 4.4 (ha-repl-hardening spec)
* Boundary: Standby Dispatch (cluster.c, ha.c)
*/
int ha_handle_snapshot_chunk(ha_state_t *ha, persist_t *p,
const char *snap_dir,
snapshot_transfer_t *chunk,
const uint8_t *payload_bytes,
size_t payload_len);
/* ── HA Config Bootstrap & Term Durability (ha-config-bootstrap spec) ── */
/*
* ha_persist_term: write ha-term.bin atomically to {workdir}/ha-term.bin.
* Uses write + rename for durability (existing pattern in the codebase).
* Must be called before any vote is granted or term is incremented.
* If workdir is NULL, the file write is skipped (safe for unit tests).
*/
void ha_persist_term(ha_state_t *ha, const char *workdir);
/*
* ha_init_raft: initialize the embedded raft state machine with cluster topology.
*
* Must be called ONCE at controller startup after config is loaded and
* ha_restore_term() has run. Subsequent elections (initial and re-elections
* after primary loss) reuse the same raft state — only raft_start_election()
* and raft_on_leader_lost() are called for each election cycle.
*
* Calling raft_init() more than once would reset raft's internal state
* (term, voted_for, role) and could lose durable election progress.
*
* n_peers: total cluster size including self (candidates_count from config).
*/
void ha_init_raft(ha_state_t *ha, cluster_t *c, int n_peers);
/*
* ha_restore_term: read {workdir}/ha-term.bin at startup and restore
* ha->term and ha->voted_for_* into ha_state_t.
*
* On missing file (ENOENT): no-op — term stays 0, voted_for_name stays empty.
* Returns 0 (safe first-boot path). Req 9.3.
* On CRC/magic mismatch: log WARN, zero ha->term and ha->voted_for_* for
* safety and return 0. Req 9.2.
* On valid file: atomically restore ha->term; copy voted_for_term
* and voted_for_name. Returns 0. Req 9.2.
* On unexpected I/O error: return -errno.
*
* Must be called after ha_init() and before any election or vote processing.
* If workdir is NULL, the function is a no-op (safe for unit tests).
* Requirements: 9.1, 9.2, 9.3.
*/
int ha_restore_term(ha_state_t *ha, const char *workdir);
/*
* handle_request_vote: process an incoming OP_requestVote RPC.
* Applies term/voted_for/freshness gate; persists state via ha_persist_term().
* Returns 1 = vote granted, 0 = vote denied, <0 = error.
* Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1
*
* TEST-ONLY (XCE_TESTING): production vote dispatch is cluster.c →
* raft_on_vote_request. This is the legacy entry point for test-only
* role-gate unit tests (test_ha_vote_gate.c, test_ha_election.c).
*/
#ifdef XCE_TESTING
int handle_request_vote(ha_state_t *ha,
uint64_t term,
const char *candidate_id,
uint64_t config_tag,
uint64_t last_lsn,
uint64_t my_config_tag,
uint64_t my_last_lsn,
int in_catch_up_flag);
#endif
#ifdef XCE_TESTING
/*
* ha_cb_persist_entry_for_test (ha-term-safety task 2.1): invoke the WIRED
* on_persist_entry callback (s_real_cbs.on_persist_entry) directly so a unit
* test exercises the real adapter body — term stamping + WAL write at the given
* 1-based index == LSN — without driving raft_log_append (task 3.1). `ha` must
* have raft_c set to a cluster whose persist module is started. The `entry`
* blob is the adapter envelope (raft_entry_blob_t prefix + payload). Returns
* the callback rc (0 on durable persist; non-zero on persist failure, in which
* case nothing is ACKed). TEST-ONLY.
*/
int ha_cb_persist_entry_for_test(ha_state_t *ha, uint64_t index, uint64_t term,
const void *entry, size_t len);
/*
* ha_cb_get_entry_term_for_test (ha-term-safety task 2.2): invoke the WIRED
* on_get_entry_term callback directly so a unit test exercises the real
* entry-term accessor (persist_entry_term at the given 1-based index == LSN).
* Returns the recorded term (legacy-0 mapped to WAL_LEGACY_TERM), or 0 when the
* LSN is absent. TEST-ONLY.
*/
uint64_t ha_cb_get_entry_term_for_test(ha_state_t *ha, uint64_t index);
/*
* ha_cb_truncate_from_for_test (ha-term-safety task 2.2): invoke the WIRED
* on_truncate_from callback directly so a unit test exercises the real conflict
* truncation (delete local entries at LSN >= index). Returns the truncate rc
* (0 on success; non-zero propagates so the library/leader retries while the
* member stays non-promotable). TEST-ONLY.
*/
int ha_cb_truncate_from_for_test(ha_state_t *ha, uint64_t index);
/*
* ha_cb_send_append_entries_for_test (ha-term-safety task 2.3): invoke the WIRED
* on_send_append_entries callback directly so a unit test exercises the real
* AppendEntries send body — peer→endpoint mapping, prev-tuple + leader_commit
* carry, and (for n_entries==1) sourcing the single entry at prev_log_index+1
* from the leader's own WAL. Combine with repl_send_ae_capture_hook to assert
* the built envelope content without network I/O. Returns the callback rc.
* TEST-ONLY.
*/
int ha_cb_send_append_entries_for_test(ha_state_t *ha, int peer_idx,
uint64_t prev_log_index,
uint64_t prev_log_term,
uint64_t leader_commit, int n_entries);
/*
* ha_cb_commit_advanced_for_test (ha-term-safety task 2.4): invoke the WIRED
* on_commit_advanced callback directly so a unit test exercises the real
* commit-source trigger — feeding the raft library's current-term-gated
* commit_index into the cluster's synchronous commit barrier as the commit
* source. TEST-ONLY.
*/
void ha_cb_commit_advanced_for_test(ha_state_t *ha, uint64_t commit_index);
/*
* ha_arm_own_term_gate_for_test (ha-term-safety task 3.1): drive the static
* own-term gate arm (the promotion WAL_NOOP, now routed through
* ha_primary_log_append) so a unit test can assert the no-op records the live
* term via the library. TEST-ONLY.
*/
void ha_arm_own_term_gate_for_test(cluster_t *c);
#endif
/*
* ha_vote_self_freshness: compute the voter's effective freshness floor.
*
* Election safety: a voter must deny candidates that are outdated against the
* strongest knowledge it has of the cluster's COMMITTED state — not merely its
* own local config file / applied LSN. The floor is the per-field max of the
* voter's local state and its learned-committed state:
*
* *out_tag = max(local_config_tag, atomic committed_config_tag)
* *out_lsn = max(local_applied_lsn, atomic last_known_committed_lsn)
*
* Uses acquire loads of the atomic learned-committed fields; never reads
* last_known_committed_config_tag (a heartbeat-learned config tag, not the
* vote freshness floor). Pure aside from the atomic reads — no state mutation.
*/
void ha_vote_self_freshness(ha_state_t *ha,
uint64_t local_config_tag,
uint64_t local_applied_lsn,
uint64_t *out_tag,
uint64_t *out_lsn);
/*
* ha_candidate_advertise_lsn: the LSN a candidate advertises in a RequestVote /
* PreVote payload (ha-commit-durability task 4.2).
*
* Returns the BARE durable WAL tip persist_durable_floor(p) — NOT the folded
* max(durable_floor, last_known_committed_lsn) ha_vote_self_freshness computes
* for the voter floor. A candidate may only claim a log position it can
* durably back, so a node restarted with a truncated log cannot over-advertise
* a stale volatile learned-committed value and win (design §1a/§5a). The
* candidate's TAG axis still uses ha_vote_self_freshness (config epoch is
* orthogonal to log durability). NULL-safe (→ 0).
*/
uint64_t ha_candidate_advertise_lsn(persist_t *p);
/* ha_sync_applied_lsn_from_recovery (bug-848) — after local WAL recovery on
* (re)start, set ha->applied_lsn to the position the member actually recovered
* (persist_durable_floor(p)), instead of the ha_init default of 0.
*
* Why: a restarted standby advertises ha->applied_lsn to the primary in its
* state report; the primary then streams the catch-up range [applied_lsn+1 ..
* committed]. If a node that already holds [1..N] on disk reports 0, the
* primary re-streams the WHOLE log and the receiver O_TRUNC-overwrites already-
* consistent records (instead of Raft incremental catch-up of only the missing
* suffix). This mirrors the value the election/PreVote path already trusts
* (ha_candidate_advertise_lsn == persist_durable_floor), so the catch-up and
* vote-freshness advertisements finally agree. NULL-safe; never lowers an
* already-higher applied_lsn (monotonic). */
void ha_sync_applied_lsn_from_recovery(ha_state_t *ha, persist_t *p);
/* ha_sync_raft_log_from_recovery (bug-1042) — after local WAL recovery on a
* (re)start, raise the raft LIBRARY's log_len / commit_index / term to the
* recovered position. ha_init_raft (before the election) sources the library's
* initial_log_len from persist_durable_floor, but on a standby-rejoin boot
* persist_recover runs LATER (cluster_init_phase2), so the library would
* otherwise stay at log_len=0 and reject every incoming AppendEntries (logOk
* fails for prevLogIndex>=1), wedging the rejoined member as a permanently non-
* converging standby. Call this right after persist_recover. Monotonic + NULL-
* safe; clamps commit_index<=log_len. */
void ha_sync_raft_log_from_recovery(ha_state_t *ha, persist_t *p);
/*
* ha_elect_evaluate_and_signal: evaluate accumulated ha_elect_ctx_t votes.
* Sets ctx->result (1=primary, 0=standby) and posts ctx->done semaphore.
* Applies quorum check, fresher-peer yield gate, and config_tag=0 fatal gate.
* Requirements: 2.1, 2.7, 2.8
*/
ctrl_role_t ha_elect_evaluate_and_signal(ha_state_t *ha, ha_elect_ctx_t *ctx);
/*
* handle_request_vote_reply: process an incoming reply to an OP_requestVote RPC.
*
* Called by each per-peer vote-request thread once the peer replies (or times out).
* Updates the entry in ctx->votes[peer_idx] with the reply fields.
* Increments ctx->votes_granted if vote_granted is non-zero.
* Increments ctx->n_replied unconditionally.
* Updates ctx->max_resp_tag / ctx->max_resp_lsn to track the freshest peer seen.
* When quorum is reached (votes_granted >= quorum) OR all peers have replied
* (n_replied >= n_peers), calls ha_elect_evaluate_and_signal() to post ctx->done.
*
* peer_idx: 0-based index into ctx->votes[] identifying the replying peer.
* term: term the peer reported in its reply (used for fencing if > ha->term).
* vote_granted: non-zero if the peer granted the vote, 0 if denied.
* peer_config_tag: peer's current config_tag at reply time.
* peer_lsn: peer's current WAL LSN at reply time.
*
* Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1
*/
void handle_request_vote_reply(ha_state_t *ha,
ha_elect_ctx_t *ctx,
int peer_idx,
uint64_t term,
int vote_granted,
uint64_t peer_config_tag,
uint64_t peer_lsn);
/*
* ha_record_vote_reply: record-only vote-reply seam (bug-828).
*
* Records ctx->votes[peer_idx] (granted flag + peer_config_tag + peer_lsn) under
* ctx->lock with the same idempotency guard as handle_request_vote_reply, and
* updates the vote tally / freshest-responder tracking — but does NOT call
* ha_elect_evaluate_and_signal (never posts ctx->done, never runs the legacy
* promotion side-effects). The PRODUCTION vote_reply_dispatch path calls this so
* the granting voter's advertised LSN reaches ha_compute_safe_committed_lsn via
* ctx->votes[], while the raft library remains the sole production promotion
* driver (raft_on_vote_reply → on_become_leader) — no double-promotion.
*
* peer_idx: 0-based index into ctx->votes[] (same convention as
* handle_request_vote_reply; the absolute→local mapping is done by the caller).
*/
void ha_record_vote_reply(ha_state_t *ha,
ha_elect_ctx_t *ctx,
int peer_idx,
uint64_t term,
int vote_granted,
uint64_t peer_config_tag,
uint64_t peer_lsn);
/*
* xce_probe_config_tag: single-RTT OP_probeConfigTag RPC to one peer.
* Fills *tag_out with the peer's current config_tag on success.
* Returns 0 on success, -errno on connection/protocol error.
*/
int xce_probe_config_tag(const char *host, uint16_t port, uint64_t *tag_out);
/*
* ha_wait_for_heartbeat_or_timeout: wait in CTRL_INIT for an incoming heartbeat.
*
* Blocks on sem_timedwait(&ha->heartbeat_seen, &deadline) where
* deadline = now(CLOCK_REALTIME) + election_timeout_ms + xorshift64_jitter
*
* Returns CTRL_STANDBY — heartbeat arrived; ha_do_demote() was called
* (role is now CTRL_STANDBY).
* Returns CTRL_ELECTING — deadline expired; caller must drive the election.
*
* The heartbeat handler posts heartbeat_seen only when role == CTRL_INIT.
* OPCode_heartbeat is allowed through ha_dispatch_gate() in CTRL_INIT state
* so that the primary's periodic heartbeat can reach the handler.
*
* Requirements: 1.1 (heartbeat → CTRL_STANDBY), 1.2 (timeout → CTRL_ELECTING).
* Boundary: Controller FSM (ha.c).
*/
ctrl_role_t ha_wait_for_heartbeat_or_timeout(ha_state_t *ha);
/* ── Task 9.4: EPOLLRDHUP Failover and Heartbeat Timeout (Req 6.2–6.4, 6.8) ─
* ha_on_primary_ctrl_fd_rdhup: EPOLLRDHUP event handler.
* Closes primary_ctrl_fd, transitions to CTRL_ELECTING, truncates WAL to
* last_known_committed_lsn, fires ha_start_election_async (via cluster when non-NULL).
* Pass c == NULL to skip persist/election (unit test mode).
*/
void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c);
/*
* ha_arm_heartbeat_timeout_vtask: start/re-arm the heartbeat watchdog thread.
* Fires ha_on_primary_ctrl_fd_rdhup after election_timeout_ms + rand_jitter of
* silence. Calling ha_on_heartbeat_received() resets the deadline.
*/
void ha_arm_heartbeat_timeout_vtask(ha_state_t *ha, cluster_t *c);
/* ha_discover_primary: step down from CTRL_ELECTING to CTRL_STANDBY when a
* live primary message (NOP, config push, ctrlhello, WAL, snapshot) arrives.
* Safe to call from any state — no-ops unless role == CTRL_ELECTING. */
void ha_discover_primary(ha_state_t *ha, cluster_t *c);
/* ha_on_heartbeat_received: reset the heartbeat deadline (Req 6.3). */
void ha_on_heartbeat_received(ha_state_t *ha);
/*
* ha_observe_heartbeat_committed: reconcile-free monotonic update of a standby's
* learned committed position from a primary heartbeat
* (ha-heartbeat-committed-propagation, Req 2.1, 2.2, 2.3, 3.1, 3.3).
*
* Advances ha->last_known_committed_lsn to committed_lsn ONLY when it is
* strictly greater than the current value (monotonic); when it advances, the
* carried committed_config_tag is stored together with it (they move as a pair —
* the tag never regresses independently). No-op when committed_lsn is 0 or
* <= the current learned LSN.
*
* This is the reconcile-free sibling of cluster_apply_ctrlhello_reply: it MUST
* NOT call ha_rejoin_reconcile, truncate the WAL, or modify ANY other state
* (produced/persisted committed, role, term, applied, etc.). The value is
* advisory — for observability and vote-floor freshness only. The caller (the
* OP_nop receive path) applies the term-guard before invoking this; the helper
* itself is term-agnostic and single-receiver-thread safe.
*/
void ha_observe_heartbeat_committed(cluster_t *c, uint64_t committed_lsn,
uint64_t committed_config_tag);
/*
* ha_consume_heartbeat_committed: the standby-side OP_nop consume branch
* (ha-heartbeat-committed-propagation, Req 1.2, 3.2, 3.4, 4.2).
*
* Safe to call unconditionally from the OP_nop handler with the decoded
* heartbeatmsg body. It applies the two safety gates before delegating to the
* monotonic ha_observe_heartbeat_committed:
* - non-primary gate (3.4): a primary uses its own authoritative committed and
* never consumes an inbound value;
* - term-guard (3.2): a heartbeat whose advertised term is behind this node's
* term is rejected, fencing a stale/partitioned primary from inflating the
* vote-freshness floor.
* NULL body, a failed gate, or a zero committed value updates nothing (4.2);
* the helper never touches liveness or any other OP_nop behaviour (4.1 — the
* caller invokes this AFTER the unchanged liveness/reply path).
*/
void ha_consume_heartbeat_committed(cluster_t *c, const heartbeat_msg_t *hb);
/*
* ha_consume_append_entries (ha-term-safety task 3.2): route ONE incoming
* heartbeat (entry->n_entries==0) or replication (entry->n_entries>0, one entry)
* AppendEntries through the raft library (raft_on_append_entries_request) so
* UpdateTerm (G5), logOk (G2), and the (term,lsn) conflict-truncate (G3) happen
* via the wired callbacks — replacing the term-blind term handling in
* ha_consume_heartbeat_committed and the committed-boundary truncate in
* ha_rejoin_reconcile. `entry` carries the AppendEntries metadata (prev tuple +
* leader_commit) on its top-level fields. `leader_idx` is the sending leader's
* raft peer index. Returns 0 on every path (never crashes the receive thread);
* the AE result (term adoption / append / truncate / reject) is reflected in the
* library + WAL state. The ACK reply that carries success/match_index is task 3.3.
*/
int ha_consume_append_entries(cluster_t *c, wal_catchup_entry_t *entry,
int leader_idx);
/* ha_heartbeat_is_fresh: 1 if a primary message arrived within
* heartbeat_timeout_ms (leader-stickiness predicate). Exposed for PreVote
* denial-reason logging in cluster.c; pure read, no state change. */
int ha_heartbeat_is_fresh(ha_state_t *ha);
/* ha_disarm_heartbeat_timeout_vtask: stop the heartbeat watchdog thread. */
void ha_disarm_heartbeat_timeout_vtask(ha_state_t *ha);
/*
* ha_start_primary_heartbeat: arm a repeating delaymgr task that sends
* OPCode_nop to every standby in ha->standby_endpoints every
* heartbeat_interval_ms. Keeps the standby-side watchdog from timing out
* when the cluster is idle (no WAL entries, no config pushes).
* Must be called after ha->role is set to CTRL_PRIMARY.
* No-op when c == NULL (test mode).
*/
void ha_start_primary_heartbeat(cluster_t *c);
/*
* ha_set_reconnect_offload_for_test / ha_reconnect_offload_enabled:
* Feature-flag accessors (raft-drive-async Task 2.1). The offload flag gates
* whether primary_heartbeat_fn submits the peer reconnect to the slow lane (ON,
* the production default — the delaymgr thread never blocks on connect) or runs
* the legacy inline blocking connect on the delaymgr thread (OFF, retained only
* so the TDD RED test can demonstrate the pre-change stall). These accessors
* are compiled in ALL builds; the setter is used by tests to flip the flag.
*/
void ha_set_reconnect_offload_for_test(int on);
int ha_reconnect_offload_enabled(void);
/*
* ha_set_snapshot_offload_for_test / ha_snapshot_offload_enabled:
* TDD kill-switch for raft-drive-async task 3.2 snapshot slow-lane offload.
* OFF (0, PRODUCTION DEFAULT): snapshot send runs INLINE on the caller
* (io_worker/drive) thread — byte-for-byte the legacy path. The offload
* lands DORMANT because it borrows a standby endpoint that is not yet
* pool-lifetime-safe (bug-1197); so bug-1198 stays open by design until
* Task 3.4 flips this ON together with the peer-control dispatch flag.
* ON (1): snapshot send runs on a slow-lane worker, off the drive thread.
*/
void ha_set_snapshot_offload_for_test(int on);
int ha_snapshot_offload_enabled(void);
/*
* ha_set_configpush_offload_for_test / ha_configpush_offload_enabled:
* TDD kill-switch for raft-drive-async task 3.3 requester-less config-push
* slow-lane offload.
* OFF (0, PRODUCTION DEFAULT): the config push runs INLINE on the caller
* (io_worker/drive) thread — byte-for-byte the legacy path (the
* requester-less push_config_transport_only fan-out + its blocking
* wait_all_with_timeout up to CONF_PUSH_TIMEOUT_MS). The offload lands
* DORMANT because the lane task borrows standby endpoints it does not own
* (bug-1197 family) and reads c->ha.standby_endpoints; moving the push to a
* lane thread while the io_worker may RDHUP-free those endpoints is a
* cross-thread UAF. Flag-ON is therefore only safe once Task 3.4
* establishes endpoint-lifetime discipline, and flipping it ON is owned by
* Task 3.4 (the single gate that flips the snapshot + peerctrl flags too).
* ON (1): the config push runs on a slow-lane worker, off the drive thread.
* NOTE: the admin-upgrade path (push_config_to_standbys, which has a waiting
* requester / barrier) is NOT affected by this flag.
*/
void ha_set_configpush_offload_for_test(int on);
int ha_configpush_offload_enabled(void);
/*
* ha_set_peerctrl_dispatch_for_test / ha_peerctrl_dispatch_enabled:
* Feature-flag accessors (raft-drive-async Task 3.1). The dispatch flag gates
* whether the registered peer-control handler (peer_ctrl_msg_handler) runs its
* routing body INLINE on the iomgr io_worker thread (OFF — the production
* default for this commit; behavior is byte-for-byte identical to before the
* task) or hands the decoded packet to the vtask pool via vtask_enq_classified
* and returns promptly (ON — the target behavior, validated by the task 3.4
* deadlock gate before it becomes the default). These accessors are compiled
* in ALL builds; the setter is used by tests to flip RED<->GREEN.
*
* DEFAULT: OFF (0). Production behavior is UNCHANGED by this commit.
*/
void ha_set_peerctrl_dispatch_for_test(int on);
int ha_peerctrl_dispatch_enabled(void);
/*
* ha_peerctrl_body_probe: test-only observation seam (raft-drive-async Task
* 3.1). When non-NULL it is invoked at the TOP of the peer-control routing
* body (peer_ctrl_msg_handler_inline) with the endpoint, BEFORE any role
* routing. Tests use it to (a) count body invocations, (b) capture the thread
* the body ran on (proving inline-vs-pool execution), and (c) optionally park
* the body to prove per-endpoint arrival-order serialization. NULL in
* production (zero cost) — a plain global like ha_nop_req_hook. */
extern void (*ha_peerctrl_body_probe)(endpoint_t *ep);
/*
* ha_vep_teardown_probe / ha_vote_ep_teardown_for_test (bug-1197): test seams
* for the deferred vote-ep teardown. ha_vep_teardown_probe (NULL in production)
* fires with the real ep at the top of the teardown vtask body, before
* endpoint_fini, so a test can assert teardown ran strictly after the last live
* same-ep use. ha_vote_ep_teardown_for_test drives the flag-ON per-ep async
* teardown branch (mark + iomgr_unmonitor + enqueue) for a single ep without
* staging a full election. Both compiled only under XCE_TESTING.
*/
extern void (*ha_vep_teardown_probe)(endpoint_t *ep);
void ha_vote_ep_teardown_for_test(ha_state_t *ha, endpoint_t *ep);
/*
* ha_peerctrl_predispatch_probe (bug-1197): test seam fired inside the dispatch
* path AFTER the tearing_down check passes but BEFORE the pool enqueue (within
* the ep->dispatch_inflight bracket). A test installs it to park a dispatch in
* exactly the drained window and prove endpoint_mark_tearing_down blocks until
* the enqueue completes. NULL in production; compiled only under XCE_TESTING.
*/
extern void (*ha_peerctrl_predispatch_probe)(endpoint_t *ep);
/* ── Task 2.4: Non-blocking async election initiator ────────────────────── */
/*
* ha_start_election_async: initiate a quorum election without blocking.
*
* Steps:
* 1. Store n_peers in ha->n_peers.
* 2. For single-node (n_peers == 1): call ha_elect_evaluate_and_signal directly
* with a self-only ha_elect_ctx_t — shortcircuit, no iomgr/delaymgr needed.
* 3. For multi-node: for each peer in peers[], allocate vote_ep_ctx_t, init
* endpoint with client_ssl_ctx, set on_vote_connected + vote_ep_err_handler,
* call endpoint_connect + iomgr_monitor; skip peer on connect failure.
* Allocate election_timeout_ctx_t (cancelled=0) and schedule
* delay_task_ms(delaymgr, election_timeout_ms, 1, election_timeout_fn, ctx).
* 4. Returns immediately; election result delivered via ha_elect_evaluate_and_signal.
*
* Observable: returns in < 5ms; endpoints registered in iomgr; watchdog in delaymgr.
* Requirements: 1.3 (non-blocking), 2.1 (single-node shortcircuit), 2.4 (async vote)
* Boundary: ha.c (election orchestration)
*/
void ha_start_election_async(ha_state_t *ha, cluster_t *c,
int n_peers, xce_candidate_t **peers,
uint64_t my_config_tag, uint64_t my_lsn,
const char *my_name);
/*
* ha_start_prevote_async: start a side-effect-free PreVote round before any
* real election (ha-prevote task 4.1).
*
* Snapshots self tag/lsn, calls raft_begin_pre_vote (self-grant + per-peer
* preVoteRequest send), and arms a round timeout bounded by election_timeout_ms
* (Req 5.4). The node stays CTRL_STANDBY and mutates NO persistent state
* (no term / voted_for / role change) for the duration of the round. Replies
* are fed to the reply handler (pre_vote_reply_dispatch); on a majority WIN the
* unchanged real-election sequence runs (truncate + CTRL_ELECTING +
* ha_start_election_async), on a LOSS / round-timeout the node stays
* CTRL_STANDBY and re-arms the liveness watchdog.
*
* The failover triggers are rewired to call this in task 4.2; until then the
* round is inert in production. c may be NULL in unit-test mode.
* Requirements: 1.1, 1.4, 1.5, 5.4, 7.3.
*/
void ha_start_prevote_async(ha_state_t *ha, cluster_t *c);
/*
* ha_begin_failover: single failover entry point (ha-prevote task 4.2).
*
* Called by both failover triggers (ha_on_primary_ctrl_fd_rdhup and, through
* it, the liveness-timeout poll) AFTER connection-teardown bookkeeping and the
* config-less voter-only guard. Starts a side-effect-free PreVote round
* (ha_start_prevote_async); does NOT truncate the WAL or transition to
* CTRL_ELECTING — those destructive steps are deferred to PreVote success.
* c may be NULL in unit-test mode. Requirements: 1.1, 4.3, 4.4.
*/
void ha_begin_failover(ha_state_t *ha, cluster_t *c);
/*
* ha_election_stagger_ms: per-node desync delay (ms) for a FAILOVER re-election.
*
* Mirrors the startup stagger (myapp.c): my_priority * election_priority_step_ms
* plus a fresh random jitter in [0, election_jitter_max_ms]. Applied before the
* PreVote start on the failover path so two survivors that lose the primary at
* the same instant do not campaign in lock-step and split the vote forever
* (Raft §5.2 randomized election timeout; bug-606). Re-randomized each call —
* the jitter_seed advances — so repeated collisions still converge.
*/
uint32_t ha_election_stagger_ms(ha_state_t *ha);
/*
* ha_jittered_election_delay: the per-CAMPAIGN election-timeout watchdog delay,
* randomized to the canonical Raft interval [base, 2*base] (Ongaro §3.4) and
* re-drawn on every campaign.
*
* task 5.7: the campaign watchdog (raft_start_election → on_schedule_timeout →
* raft_cb_schedule_timeout) was previously armed with a FIXED election_timeout_ms.
* Under cold-connection churn a survivor's single needed grant can land a hair
* AFTER a constant deadline on EVERY campaign — a resonance that times out,
* re-escalates the term, and then drops the now-stale grant (DropStaleResponse)
* forever (unbounded term climb; the final round-4 chaotic-soak stall).
* Randomizing the watchdog per campaign breaks the resonance: some campaign's
* deadline falls AFTER its grant RTT, the grant lands in-window, quorum is
* reached. Independent of election_jitter_max_ms (which tunes only the
* sub-second startup/stagger jitter) so it still spreads when jmax is small/0.
* Never returns below `base`, so genuine-loss failover is never sped up past the
* configured election_timeout_ms floor. base==0 returns 0 (no modulo-by-zero).
*/
uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base);
/*
* ha_dispatch_repl_action_for_standby: evaluate the replication decision for
* one standby and execute the resulting action.
*
* Reads ha->standby_reported_tag[peer_idx] and ha->standby_reported_lsn[peer_idx]
* (stored by ha_process_standby_state on the previous call), calls
* ha_evaluate_repl_action, and acts on the result:
* - HA_REPL_PUSH_CONFIG: In production builds, calls push_config_to_standbys(c, …).
* In XCE_TESTING builds, calls ha_repl_push_config_for_standby_hook(c, peer_idx).
* - HA_REPL_STREAM_WAL: calls ha_stream_wal_to_standby(ep, wal_path, standby_lsn,
* committed_lsn) to batch-stream WAL entries via xce_request() (non-blocking).
* In XCE_TESTING builds, calls ha_repl_stream_wal_hook if non-NULL.
* - HA_REPL_SEND_SNAPSHOT: calls ha_send_snapshot_to_standby(ep, snap_path, lsn).
* In XCE_TESTING builds, calls ha_repl_send_snapshot_hook(peer_idx) if non-NULL.
* - HA_REPL_CONVERGED: no-op.
*
* One action per heartbeat cycle per standby (design §Unified Heartbeat-Driven
* Protocol). CONVERGED is a no-op.
*
* ep — outbound endpoint to the standby (used for STREAM_WAL; may be
* NULL in tests that only exercise PUSH_CONFIG or CONVERGED paths)
* peer_idx — 0-based index into ha->standby_reported_* arrays
* committed_tag — primary's committed config_tag (from c->myconf or test fixture)
* committed_lsn — primary's committed WAL LSN (from c->persist or test fixture)
* oldest_wal_lsn— oldest retained WAL LSN (0 = all WAL retained, skip snapshot)
*
* Returns 0 on success; negative on internal error.
* Requirements: 1.4, 5.1–5.5 (ha-repl-hardening spec)
*/
int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
cluster_t *c,
endpoint_t *ep,
int peer_idx,
uint64_t committed_tag,
uint64_t committed_lsn,
uint64_t oldest_wal_lsn);
/*
* ha_send_ctrlhello_to_all_standbys: iterate ha->standby_endpoints and send
* an OP_hello CtrlHelloRequest to each connected standby.
* This triggers the existing catch-up replication path in handle_ctrlhello
* (Req 7.2). Defined in cluster.c; called from ha_elect_evaluate_and_signal
* in the post-election path.
*/
void ha_send_ctrlhello_to_all_standbys(cluster_t *c);
/* ── P3-6: Test-only failover shim ────────────────────────────── */
#ifdef XCE_TESTING
/*
* ha_phase2_hook: test-only seam for cluster_init_phase2.
* When non-NULL, ha_elect_evaluate_and_signal() calls this INSTEAD OF the
* real cluster_init_phase2() so unit tests can count phase2 calls without
* requiring a real cluster_t. Pattern mirrors xce_vote_request_hook.
*/
extern int (*ha_phase2_hook)(void *c);
/*
* ha_send_ctrlhello_hook: test-only seam for ha_send_ctrlhello_to_all_standbys.
* When non-NULL, ha_elect_evaluate_and_signal() calls this in the post-election
* path instead of the real function (cluster.c not linked in test binary).
* Requirements: 7.2.
*/
extern int (*ha_send_ctrlhello_hook)(void *c);
/*
* ha_push_config_hook: test-only seam for push_config_to_standbys.
* When non-NULL, ha_elect_evaluate_and_signal() calls this in the post-election
* path instead of the real function. Requirements: 7.1.
*/
extern int (*ha_push_config_hook)(void *c);
/*
* ha_heartbeat_req_hook: test-only seam for the per-standby HA heartbeat send
* in primary_heartbeat_fn (Task 3.1). When non-NULL, primary_heartbeat_fn
* calls this INSTEAD OF xce_ha_heartbeat_req, passing the SAME three values it
* stamps onto the wire — the primary's CURRENT own term / committed_lsn /
* committed_config_tag. A test installs it to capture those args and assert
* the heartbeat carries the primary's own committed state (Req 1.1, 3.4).
*
* Precedence (XCE_TESTING only): ha_heartbeat_req_hook (if set) takes priority,
* else the legacy ha_nop_req_hook (if set), else the real xce_ha_heartbeat_req.
* Keeping ha_nop_req_hook as a fallback preserves the existing liveness tests.
*/
extern int (*ha_heartbeat_req_hook)(endpoint_t *ep, uint64_t term,
uint64_t committed_lsn,
uint64_t committed_config_tag);
/*
* ha_reconnect_connect_hook: test-only seam for the BLOCKING half of the
* offloaded peer reconnect (raft-drive-async Task 2.1). The reconnect splits
* into a non-blocking prep (alloc endpoint, set callbacks) on the delaymgr
* tick and a blocking connect+monitor that runs on a slow-lane worker. When
* non-NULL, the lane worker calls this INSTEAD OF the real bounded
* endpoint_connect + iomgr_monitor, so a test can drive the reconnect path
* deterministically (e.g. park the worker to simulate an unreachable peer)
* without a real socket.
*
* Signature: int hook(cluster_t *c, int peer_idx)
* returns 0 to simulate a successful connect, <0 to simulate failure.
* The lane worker clears reconnect_inflight[peer_idx] after this returns
* regardless of outcome, so dedup never wedges (Req 2.5).
*/
extern int (*ha_reconnect_connect_hook)(cluster_t *c, int peer_idx);
/*
* ha_repl_push_config_for_standby_hook: test-only seam for the per-standby
* config push triggered by the heartbeat-reply decision loop (Task 3.3).
*
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
* push_config_to_standbys() when the decision is HA_REPL_PUSH_CONFIG, so unit
* tests can verify the wiring without requiring a real cluster_t or network.
*
* Signature: int hook(void *c, int peer_idx)
* c — cluster_t* (may be NULL in test mode)
* peer_idx — 0-based index of the standby that triggered the push
* Requirements: 1.4 (ha-repl-hardening)
*/
extern int (*ha_repl_push_config_for_standby_hook)(void *c, int peer_idx);
/*
* ha_repl_stream_wal_hook: test-only seam for ha_stream_wal_to_standby.
*
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
* ha_stream_wal_to_standby() when the decision is HA_REPL_STREAM_WAL, so unit
* tests can count batch-send invocations without requiring a real endpoint or
* WAL directory.
*
* Signature: int hook(void *ep, const char *wal_path,
* uint64_t standby_lsn, uint64_t committed_lsn)
* ep — endpoint_t* (may be NULL in test mode)
* wal_path — WAL directory path
* standby_lsn — standby's last applied LSN
* committed_lsn— primary's committed LSN
* Requirements: 5.1–5.5 (ha-repl-hardening)
*/
extern int (*ha_repl_stream_wal_hook)(void *ep, const char *wal_path,
uint64_t standby_lsn,
uint64_t committed_lsn);
/*
* ha_repl_send_snapshot_hook: test-only seam for ha_send_snapshot_to_standby.
*
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
* ha_send_snapshot_to_standby() when the decision is HA_REPL_SEND_SNAPSHOT,
* so unit tests can verify the wiring without requiring a real endpoint or
* snapshot directory.
*
* Signature: int hook(int peer_idx)
* peer_idx — 0-based index of the standby that triggered the snapshot
* Requirements: 4.1–4.5 (ha-repl-hardening)
*/
extern int (*ha_repl_send_snapshot_hook)(int peer_idx);
/*
* ha_tcp_connect_hook: test-only seam for the raw TCP connect inside
* ha_find_and_connect_to_primary. When non-NULL, the function calls this
* INSTEAD OF the real getaddrinfo/connect path, allowing unit tests to inject
* socketpair fds without real network I/O.
* Signature: int hook(const char *host, int port)
* Returns fd >= 0 on success, or -errno on failure.
* Requirements: task 9.2.
*/
extern int (*ha_tcp_connect_hook)(const char *host, int port);
/* Test-only shim: demote then re-promote; returns new term.
* Exercises the manual failover state machine without a live cluster_t.
* Calls ha_do_demote() then ha_do_promote(); validates role transitions
* and term monotonicity. See design doc P3-6 and src/docs/persistence-and-ha-design.md.
*/
uint64_t ha_trigger_manual_failover_for_test(ha_state_t *ha);
/*
* ha_rdhup_election_hook: test-only seam for ha_start_election_async in the
* RDHUP failover path.
*
* When non-NULL, ha_on_primary_ctrl_fd_rdhup() calls this INSTEAD OF the real
* ha_start_election_async() so unit tests can verify the RDHUP path triggers
* async re-election without requiring a live iomgr/delaymgr.
*
* The hook receives the same arguments that would be passed to ha_start_election_async.
* Requirements: 6.3, 6.5, 9.3.
*/
extern void (*ha_rdhup_election_hook)(ha_state_t *ha, cluster_t *c,
int n_peers, xce_candidate_t **peers,
uint64_t my_config_tag, uint64_t my_lsn,
const char *my_name);
/*
* vote_ep_ctx_set_cluster_for_test: set the cluster_t pointer inside the
* vote_ep_ctx_t stored at ep->data. Used by unit tests that need the
* promoted-endpoint dispatch path to reach ha_promote_to_primary(c) with a
* live cluster pointer.
* The vote_ep_ctx_t struct is internal to ha.c; this helper avoids exposing it.
* Requirements: task 7.3 (ha-peer-connection-reuse).
*/
void vote_ep_ctx_set_cluster_for_test(struct endpoint_t *ep, cluster_t *c);
/*
* ha_restart_hook: test-only seam for ha_self_restart's process re-execution
* (ha-convergence-recovery Req 1.6).
*
* When non-NULL, ha_self_restart() invokes this INSTEAD OF execv() so unit
* tests can observe the eventual restart deterministically (the demotion
* convergence task and rejoin reconciliation both end in ha_self_restart).
* The hook's return value is ha_self_restart's return value on the
* test-success path; a NON-ZERO return models an exec failure and routes
* ha_self_restart to the FATAL contract via ha_fatal_hook (Req 1.4).
*
* Signature: int hook(cluster_t *c)
* Returns 0 to model a successful restart; non-zero to model exec failure.
*/
extern int (*ha_restart_hook)(cluster_t *c);
/*
* ha_fatal_hook: test-only seam for the FATAL (abort) contract of
* ha_self_restart (ha-convergence-recovery Req 1.4).
*
* When non-NULL, ha_self_restart() invokes this INSTEAD OF abort() when the
* restart cannot be performed, so a test binary can assert the FATAL path was
* taken without the process dying. ha_self_restart returns after invoking it
* (test builds only — production aborts).
*
* Signature: void hook(const char *msg)
*/
extern void (*ha_fatal_hook)(const char *msg);
/*
* raft_cb_become_follower_with_cluster_for_test: drive the follower path with
* a LIVE cluster pointer (ha-convergence-recovery task 2.1 demotion-sequence
* tests). Unlike raft_cb_become_follower_for_test (which forces raft_c = NULL
* to suppress production calls), this shim sets ha->raft_c = c first so the
* cluster-dependent demotion-convergence branch (task 2.2: hook disable +
* scheduled truncate/restart) is exercised against a real persist/hooksys/
* delaymgr fixture. Mirrors the real-cluster seam ha_promote_to_primary(c)
* used by the promotion-side lifecycle tests.
*/
void raft_cb_become_follower_with_cluster_for_test(cluster_t *c, uint64_t term);
/*
* raft_cb_become_leader_with_cluster_for_test: drive the leader promotion path
* with a LIVE cluster pointer (fix-promotion-standby-registration). Unlike
* raft_cb_become_leader_for_test (which forces raft_c = NULL and so skips the
* c-dependent standby-registration branch entirely), this shim sets
* ha->raft_c = c first so the promotion-time repl_add_standby loop — which
* registers each surviving vote endpoint (votes[i].ep != NULL) carrying a
* peer pointer (votes[i].peer != NULL) into the WAL replicator — is exercised
* against a real persist/repl/commit fixture. Symmetric with
* raft_cb_become_follower_with_cluster_for_test.
*/
void raft_cb_become_leader_with_cluster_for_test(cluster_t *c, uint64_t term);
/*
* ha_walack_continuation_for_test: test-only entry point for the ACK-driven
* catch-up continuation logic (ha-convergence-recovery task 4.x, Req 5.1, 5.2,
* 7.1).
*
* The production trigger lives in the walAck branch of
* repl_ack_decode_and_dispatch (ha.c), which decodes a real Cap'n Proto
* WalAckReply off the wire — too heavy to drive from a unit test. This shim
* exposes the SAME post-ACK decision the walAck branch performs after
* repl_on_ack, so the continuation / single-flight / parity behaviors can be
* exercised directly against a unit fixture:
*
* target = max(committed_lsn, last_written) (same derivation as ha.c:2771-2776)
* if acked_lsn < target → schedule ONE continuation that streams
* [acked_lsn+1, target] via the catch-up streaming
* path, UNLESS ha.catchup_inflight[peer_idx] is set
* (single-flight, Req 5.2); the continuation must not
* dispatch any heartbeat/stateReport action (Req 5.1).
* if acked_lsn >= target → repl_complete_catchup(&c->persist.repl, ep), which
* clears the standby's in_catch_up (Req 7.1).
*
* c — cluster_t* (real persist + repl registry; never NULL in tests)
* ep — repl registry endpoint key for the standby (the same pointer
* passed to repl_add_standby); identifies the standby slot.
* peer_idx — 0-based standby slot index (into ha.catchup_inflight[]).
* acked_lsn— the LSN the standby just acknowledged.
*
* Returns 0 on the no-op / parity paths and on a successfully scheduled
* continuation; negative on a scheduling failure.
*
* RED scaffold (task 4.1): ha_convergence.c currently records the ACK only and
* neither schedules a continuation nor calls repl_complete_catchup — task 4.2
* implements the real decision. The continuation / single-flight / parity tests
* therefore FAIL until 4.2 lands.
*/
int ha_walack_continuation_for_test(cluster_t *c, void *ep, int peer_idx,
uint64_t acked_lsn);
/*
* ha_quorum_transition_log_hook: test-only emission-count seam for
* ha_log_quorum_transition (ha-convergence-recovery task 5.x, Req 8.1-8.3).
*
* There is NO log-capture harness in this testsuite, so the quorum-transition
* tests use a counter/edge-observable instead (the task permits "log-capture OR
* counter-hook pattern"). When non-NULL, ha_log_quorum_transition invokes this
* hook EXACTLY ONCE on each transition it WARNs about, passing the new state and
* the connected/required counts that go into the WARN line. Tests count
* invocations to assert "logged once per transition" (Req 8.3) and inspect the
* arguments to assert the loss line carries the right counts (Req 8.1).
*
* Signature: void hook(int lost, int connected, int required)
* lost — 1 on a quorum-LOSS edge, 0 on a quorum-RESTORE edge.
* connected — connected standby count reported in the WARN line.
* required — required standby count (quorum threshold) in the WARN line.
*
* RED scaffold (task 5.1): ha_log_quorum_transition is a no-op stub that never
* invokes this hook, so the transition tests' emission-count assertions FAIL
* until task 5.2 implements the real edge detection.
*/
extern void (*ha_quorum_transition_log_hook)(int lost, int connected,
int required);
#endif
#endif /* HA_H */
Reference in a new issue View git blame Copy permalink