wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity
myxce/src/controller/ha.h
Rongsong Shen 59d9df246c fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort 
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
2026-06-18 11:10:19 +00:00

1250 lines
64 KiB
C
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#ifndef HA_H
#define HA_H
/*
* XCE Controller HA — role management and election (P1).
*
* P1-1 audit: the existing create_ssl_context() + iomgr_set_ssl_ctx() path
* in cluster.c (line 559, set at line 583) covers all controller-to-controller SSL.
* HA listeners (heartbeat, election probe, replication accept) reuse the same
* ssl_ctx already installed on c->iomgr.
* No new TLS infrastructure is needed — XCE runs on a trusted management LAN.
* Peer identity is validated by host:port match against the --peers candidate list.
* See design doc Part 5.9 and memory: Persistence & HA Trusted LAN Decision.
*
* P1-2: full struct definitions and API.
* endpoint_t* (primary_ep) is intentionally omitted here — including netio.h
* would pull in heavy OpenSSL headers and create circular dependency risks.
* The watchdog (P1-3) will include netio.h directly where it needs endpoint_t.
*/
#include <pthread.h>
#include <semaphore.h>
#include <stdbool.h>
#include <stdatomic.h>
#include <stdint.h>
#include <time.h>
#include "config.h" /* XCE_MAXNAME_LEN, MAX_STANDBYS, etc. */
#include "myconf.h" /* myconf_t, config_version_tag */
#include "persist.h" /* persist_t, apply_wal_entry */
#include "typemap.h" /* wal_catchup_entry_t, state_report_t, xce_reply_t */
#include "xce/cluster.h" /* xce_candidate_t */
#include "zarray.h"
#include "raft/raft.h" /* raft_state_t, raft_init */
/* Forward declaration — avoids circular include with cluster_def.h */
typedef struct cluster_s cluster_t;
/* ── Inline PRNG for election jitter ───────────────────────── */
static inline uint64_t xorshift64_next(uint64_t *state)
{
uint64_t x = *state;
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
*state = x;
return x;
}
/*
* ha_seed_jitter: seed the xorshift64 PRNG from /dev/urandom (8 bytes).
* Falls back to CLOCK_MONOTONIC ^ priority ^ pid if urandom is unavailable.
* Not inline — contains syscalls (open/read/close/clock_gettime/getpid).
*/
void ha_seed_jitter(uint64_t *seed, int my_priority);
/* ── Role enum ─────────────────────────────────────────────── */
typedef enum {
CTRL_PRIMARY = 0,
CTRL_STANDBY = 1,
CTRL_INIT = 2, /* Startup: listener open, awaiting heartbeat or election timeout */
CTRL_ELECTING = 3 /* Running quorum election via OP_requestVote */
} ctrl_role_t;
/* ── Election types ────────────────────────────────────────── */
/*
* ha_vote_entry_t: per-peer state accumulated by the election candidate.
* One entry per peer in ha_elect_ctx_t.votes[] (flexible array).
*/
/* Forward declaration to avoid pulling netio.h into ha.h */
struct endpoint_t;
typedef struct {
xce_candidate_t *peer; /* Peer being solicited */
uint64_t my_term; /* Term sent in OP_requestVote */
uint64_t my_tag; /* config_tag sent in request */
uint64_t my_lsn; /* WAL LSN sent in request */
int vote_granted; /* 1 if peer granted vote */
int replied; /* 1 if peer replied (granted or denied) */
uint64_t peer_tag; /* peer's config_tag from reply */
uint64_t peer_lsn; /* peer's WAL LSN from reply */
struct endpoint_t *ep; /* Outgoing vote endpoint; NULL if connect failed or not yet connected */
} ha_vote_entry_t;
/*
* ha_elect_ctx_t: shared context for one quorum election round.
* Allocated as: calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))
* Stored in ha_state_t.elect_ctx (void * _Atomic) during election; NULL otherwise.
*/
typedef struct {
int n_peers; /* Total peer count (excluding self) */
int votes_granted; /* Votes received; starts at 1 (self-vote) */
int n_replied; /* Number of peers that replied */
uint64_t max_resp_tag; /* Highest config_tag seen in any reply */
uint64_t max_resp_lsn; /* Highest WAL LSN seen among max_resp_tag peers */
int result; /* 1=primary, 0=standby, <0=error */
int done_posted; /* Guard: non-zero once sem_post(&done) has been called */
pthread_mutex_t lock; /* Protects votes_granted/n_replied/max_resp_tag/max_resp_lsn/result */
sem_t done; /* Startup thread blocks; dispatch posts when done */
/* Fix 4 (HIGH): pointer back to the election_timeout_ctx_t so that task 4.1
* (quorum-reached path) can set timeout_ctx->cancelled = 1 to cancel the
* watchdog. Set by ha_start_election_async after scheduling the timeout. */
void *timeout_ctx; /* election_timeout_ctx_t*; NULL until watchdog scheduled */
/* Task 4.1: cluster_t* so ha_elect_evaluate_and_signal can call
* cluster_init_phase2(c).
* NULL in unit tests (c=NULL bypasses phase2). */
cluster_t *c;
/* Task 6.3: candidate's own LSN/tag, stored unconditionally so that
* ha_compute_safe_committed_lsn can use them even for single-node elections
* where votes[] is empty (n_peers=0). */
uint64_t self_lsn; /* Candidate's own WAL LSN at election start */
uint64_t self_tag; /* Candidate's own config_tag at election start */
ha_vote_entry_t votes[]; /* C99 flexible array: n_peers-1 entries */
} ha_elect_ctx_t;
/* ── HA configuration ──────────────────────────────────────── */
/* Default values used by cluster_init(). Tuned for a LAN with ≤5ms RTT. */
#define HA_DEFAULT_HEARTBEAT_INTERVAL_MS 3000
#define HA_DEFAULT_HEARTBEAT_TIMEOUT_MS 10000
#define HA_DEFAULT_ELECTION_TIMEOUT_MS 5000
#define HA_DEFAULT_PROBE_CONNECT_TIMEOUT_MS 500
/* Log the first connect failure per peer, then every Nth attempt. */
#define HA_CONNECT_LOG_THROTTLE 10
#define HA_DEFAULT_MAX_PROMOTION_RETRIES 5
#define HA_DEFAULT_ELECTION_JITTER_MAX_MS 200
#define HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500
typedef struct {
uint32_t heartbeat_interval_ms; /* Primary heartbeat send interval (min 2000 ms) */
uint32_t heartbeat_timeout_ms; /* Standby timeout before promoting (>= interval) */
uint32_t election_timeout_ms; /* Election probe shared deadline (min 500 ms) */
uint32_t probe_connect_timeout_ms; /* Per-probe TCP connect timeout */
uint32_t max_promotion_retries; /* Max consecutive failed promotion attempts */
uint32_t election_jitter_max_ms; /* random jitter added to election timeout (default 200) */
uint32_t election_priority_step_ms; /* extra ms per priority level (default 500) */
} ha_config_t;
/* probe_ctx_t deleted — only used by am_i_primary() (election-iomgr-refactor) */
/* ── HA runtime state (embedded in cluster_t as `ha_state_t ha` in P1-5) ── */
typedef struct {
_Atomic uint64_t term; /* Election term — fencing token */
_Atomic int32_t role; /* Cast to ctrl_role_t at access sites */
int my_priority; /* Index in --peers list (lower = higher priority) */
_Atomic uint64_t applied_lsn; /* Standby: last applied WAL LSN */
char primary_host[XCE_MAXNAME_LEN]; /* Hostname of current primary */
uint32_t primary_port; /* Port of current primary */
/* last_heartbeat deleted — heartbeat freshness now derives from last_activity */
_Atomic int promotion_failed;
_Atomic int promotion_retry_count;
_Atomic int in_catch_up; /* Non-zero while processing historical WAL entries */
/* failover_requested deleted — watchdog replaced by ha_arm_heartbeat_timeout_vtask */
int hooks_registered; /* Idempotency guard for xce_add_hook calls */
uint32_t last_received_crc; /* Chained CRC of last received replicated entry */
ha_config_t config;
pthread_mutex_t lock; /* Protects primary_host/port + voted_for_term/name */
/* watchdog_stop / watchdog_thread deleted — replaced by ha_arm_heartbeat_timeout_vtask */
void *ssl_ctx; /* SSL_CTX* cast; avoids pulling openssl headers here */
uint64_t jitter_seed; /* xorshift64 PRNG state for election jitter */
zarray_t *standby_endpoints; /* connected standby controller endpoints for config push */
/* ── New fields added by ha-config-bootstrap spec ──────── */
uint64_t voted_for_term; /* Req 9: term of last granted vote */
char voted_for_name[XCE_MAXNAME_LEN]; /* Req 9: candidate of last granted vote */
int primary_ctrl_fd; /* Req 6: persistent channel fd (-1 = none) */
bool defer_config_fetch; /* Req 3: tag=0 node defers fetch to post-election */
void * _Atomic elect_ctx; /* Req 2: ha_elect_ctx_t* during election; NULL otherwise */
_Atomic uint64_t last_known_committed_lsn; /* Req 5: from primary heartbeats */
sem_t heartbeat_seen; /* Req 1: posted by heartbeat handler during CTRL_INIT */
uint64_t last_known_committed_config_tag; /* Req 8: from primary heartbeats */
_Atomic uint64_t committed_config_tag; /* Req 8.1: set on quorum ACK */
/* ── New fields added by election-iomgr-refactor spec (Task 2.1) ── */
void *client_ssl_ctx; /* SSL_CTX* (TLS_client_method); created in myapp_run */
int phase2_done; /* 1 after cluster_init_phase2 runs */
int n_peers; /* total cluster size (including self), matches raft convention */
/* ── New fields added by raft-library spec (Task 6.1) ──────── */
raft_state_t raft; /* Raft consensus state (wired in task 6.2) */
/* ── New fields added by raft-library spec (Task 6.2) ──────── */
cluster_t *raft_c; /* cluster_t* set by ha_start_election_async; used by raft callbacks */
void *raft_election_timeout_ctx; /* election_timeout_ctx_t* for cancel (task 6.2) */
/* ── New fields added by ha-heartbeat-liveness spec (Task 1.1) ── */
time_t last_activity; /* CLOCK_MONOTONIC seconds of last primary message; 0 = none yet */
/* ── New fields added by ha-repl-hardening spec (Task 1.4) ──────── */
uint64_t standby_reported_tag[MAX_HA_CANDIDATES]; /* Last config_tag reported by each standby */
uint64_t standby_reported_lsn[MAX_HA_CANDIDATES]; /* Last applied_lsn reported by each standby */
bool standby_converged[MAX_HA_CANDIDATES]; /* Whether standby[i] has converged with primary */
int peer_connect_fail_count[MAX_HA_CANDIDATES]; /* throttle log spam on repeated connect failure */
/* ── New field added by ha-commit-engine spec (Task 4.2) ──────── */
replicator_t *repl; /* Back-pointer to the standby registry (persist_t.repl).
* When set, ha_has_write_quorum counts registry truth via
* repl_active_standby_count (ep != NULL && !in_catch_up);
* NULL → legacy standby_endpoints->count fallback.
* (Req 3.1-3.3, 6.1, 6.2; wired at init by task 5.3.) */
/* ── New field added by ha-convergence-recovery spec (Task 3.x) ──────── */
_Atomic int rejoin_reconciled; /* Defensive once-guard for rejoin
* reconciliation (Req 2.1): set after the
* first truncate+self-restart so a second
* CtrlHelloReply in the same process
* lifetime cannot re-fire the restart.
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-convergence-recovery spec (Task 4.x) ──────── */
_Atomic int catchup_inflight[MAX_STANDBYS]; /* Per-standby single-flight
* guard for ACK-driven catch-up
* continuation (Req 5.2): set when a
* continuation is scheduled for a
* standby slot, cleared when it
* completes; a duplicate walAck while
* the flag is set schedules nothing.
* Slot identity (the repl registry slot
* index) is stable across schedule/run.
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-convergence-recovery spec (Task 5.x) ──────── */
_Atomic int quorum_lost; /* Quorum-transition edge detector (Req
* 8.1-8.3): 0 = quorum held (or unknown
* at init), 1 = quorum lost. Flipped by
* ha_log_quorum_transition via CAS so the
* loss/restore WARN is emitted exactly
* once per edge (no log storms across
* repeated dispatch-gate rejections).
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-commit-durability spec (Task 4.4) ──────── */
void * _Atomic promotion_fence_ctx; /* promotion_fence_ctx_t* for the
* bounded-window catch-up timer (Req
* 2.2/2.4): a recovery-truncated/behind
* member that wins an election with no
* reachable peer is NOT FATAL'd at the
* promotion instant; it is fenced
* (in_catch_up) and this one-shot timer
* is armed for DEFAULT_RECOVERY_CATCHUP_-
* WINDOW_MS. The 3.3 FATAL is reached
* ONLY from that timer iff still
* hollow + no peer + no catch-up
* progress when it fires; cleared/
* cancelled on recovery or shutdown so a
* caught-up or stopped member never
* FATALs. NULL at init / when no fence
* timer is pending. */
} ha_state_t;
/* ── Public API ────────────────────────────────────────────── */
/*
* ha_init: zero-initialise *ha, validate cfg, store config and my_priority.
* Returns 0 on success, -EINVAL on bad config.
* Initial role is always CTRL_STANDBY.
*/
int ha_init(ha_state_t *ha, const ha_config_t *cfg, int my_priority);
/*
* ha_fini: destroy mutex. Watchdog thread join added in P1-3.
*/
void ha_fini(ha_state_t *ha);
/* am_i_primary() deleted — replaced by ha_start_election_async (election-iomgr-refactor) */
/*
* ha_promote_to_primary: stub — atomically sets role to PRIMARY.
* Full promotion sequence (hooks, WAL flush, peer notify) added in P1-3a.
*/
int ha_promote_to_primary(cluster_t *c);
/*
* ha_demote_to_standby: stub — atomically sets role to STANDBY.
* Full demotion sequence added in P1-3a.
*/
int ha_demote_to_standby(cluster_t *c);
/* ── Internal promotion/demotion (P1-3a) ───────────────────── */
/*
* ha_do_promote: increment term, set role=PRIMARY, reset last_received_crc.
* No cluster_t dependency — fully testable standalone.
* Callers: ha_promote_to_primary() (P1-5), watchdog failover path.
*/
void ha_do_promote(ha_state_t *ha);
/*
* ha_do_demote: set role=STANDBY, reset hooks_registered idempotency guard.
* Hook disable requires cluster_t and is done in ha_demote_to_standby() (P1-5).
*/
void ha_do_demote(ha_state_t *ha);
/* ── Demotion convergence + self-restart primitive ─────────────
* (ha-convergence-recovery Requirement 1: a demoted ex-primary discards its
* uncommitted, potentially divergent state and rejoins from committed state
* only via truncation + process self-restart through boot recovery.)
*/
/*
* controller_saved_argv: accessor for the NULL-terminated argv captured by
* main() before argp_parse, used by ha_self_restart to re-exec the controller
* (ha-convergence-recovery Req 1.3, design "Self-Restart Primitive").
* Returns NULL if argv was never saved (e.g. unit-test binaries).
*/
char **controller_saved_argv(void);
/*
* ha_self_restart: the truncate-and-restart primitive's restart half.
*
* Production contract (Req 1.3, 1.4): flush logs, execv(controller_saved_argv())
* to rebuild in-memory state through boot recovery from the persisted committed
* state only; if exec cannot be performed, terminate FATALLY (abort) rather than
* continue running with potentially divergent state — NEVER returns in
* production.
*
* Test contract (Req 1.6): under XCE_TESTING, invoke ha_restart_hook instead of
* execv and return its result (0 on the success path); a failing ha_restart_hook
* routes to ha_fatal_hook in place of abort so tests can assert the FATAL path
* without dying.
*
* Preconditions: persistence hooks already disabled; truncation completed
* before this call. Invariant: no new WAL entry is appended by this node
* between hook-disable and exec.
*/
int ha_self_restart(cluster_t *c);
/*
* ha_schedule_demotion_convergence: schedule the one-shot delaymgr task that
* truncates above committed_lsn (Req 1.2), WARN-logs the discarded range
* (Req 1.5), and self-restarts (Req 1.3). Called from raft_cb_become_follower's
* was-PRIMARY branch AFTER the persistence hooks are disabled inline (Req 1.1),
* so all file I/O and the exec run off the become-follower callback thread (D1).
* Returns 0 on success, negative on alloc/schedule failure (suffix is then
* reconciled by rejoin reconciliation at the next join).
*/
int ha_schedule_demotion_convergence(cluster_t *c);
#ifdef XCE_TESTING
/*
* ha_demotion_convergence_fn: the delaymgr task body — exposed under XCE_TESTING
* only so the demotion-convergence suite can reference its contract; the test
* fixture drives the queued task through its stored entry pointer.
*/
int ha_demotion_convergence_fn(void *arg);
#endif
/*
* ha_rejoin_reconcile: converge a rejoining node whose local WAL log runs ahead
* of the cluster's committed boundary (ha-convergence-recovery Req 2.1–2.3,
* design "Rejoin Reconciliation", research D5).
*
* Called from the standby's CtrlHelloReply intake (cluster.c) after the existing
* last_known_committed_lsn store, with primary_committed_lsn = the boundary the
* primary reported.
*
* Contract (task 3.2 GREEN implements; task 3.1 ships a RED no-op stub):
* - local_tip = persist.current_lsn - 1.
* - If local_tip > primary_committed_lsn AND the rejoin_reconciled once-guard
* is clear: cutoff = max(primary_committed_lsn, local committed_lsn) — the
* CLAMP that never discards the locally known committed prefix (Req 2.2) —
* persist_truncate_wal_above(cutoff), WARN the discarded range, set the
* once-guard, then ha_self_restart (deferred to delaymgr if the intake runs
* on iomgr, same rule as demotion). Returns the restart result.
* - Else (tip <= boundary): return 0 with NO side effects (Req 2.3).
* - A second call in the same process lifetime takes no restart (Req 2.1 —
* defensive once-guard; "once" is also structural since post-restart the tip
* equals the boundary).
*
* Generalizes the RDHUP truncation pattern (ha_on_primary_ctrl_fd_rdhup), which
* remains in place unchanged for the failover path.
*/
int ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn);
/*
* cluster_apply_ctrlhello_reply: standby-side CtrlHelloReply intake body
* (ha-convergence-recovery task 3.1). Stores the primary-reported committed
* boundary + config tag; called from cluster.c's myreq_handler on every
* CtrlHelloReply. Defined in ha_convergence.c (a unit-linkable TU) so the
* rejoin intake-integration test can drive the real path. Task 3.2 adds the
* ha_rejoin_reconcile trigger after the store.
*/
void cluster_apply_ctrlhello_reply(cluster_t *c, uint64_t committed_lsn,
uint64_t config_tag);
/*
* ha_walack_on_ack: the post-ACK catch-up decision (ha-convergence-recovery
* task 4.2, Req 5.1, 5.2, 7.1, design D6). Called from the walAck branch of
* repl_ack_decode_and_dispatch (ha.c) AFTER repl_on_ack, and from the
* ha_walack_continuation_for_test shim so production and tests exercise the
* same logic. Defined in config_repl.c (alongside ha_stream_wal_to_standby and
* the continuation task body).
*
* target = max(committed_lsn, last_written) (same derivation as ha.c:2771-2776)
* if acked_lsn < target → schedule ONE catch-up continuation that streams
* [acked_lsn+1, target] on delaymgr (never inline on
* iomgr), UNLESS c->ha.catchup_inflight[peer_idx] is
* already set (per-standby single-flight, Req 5.2).
* if acked_lsn >= target → repl_complete_catchup(&c->persist.repl, ep) to
* re-include the standby in live replication (Req 7.1).
*
* c — cluster_t* (real persist + repl registry; never NULL).
* ep — repl registry endpoint key for the standby (endpoint_t* /
* repl_ep_t); identifies the standby slot for streaming + parity.
* peer_idx — 0-based standby slot index (into c->ha.catchup_inflight[]).
* acked_lsn— the LSN the standby just acknowledged.
*
* Returns 0 on the parity / single-flight-suppressed paths and on a successfully
* scheduled continuation; negative on a scheduling/allocation failure. Never
* performs file I/O itself (streaming runs in the deferred continuation task).
*/
int ha_walack_on_ack(cluster_t *c, void *ep, int peer_idx, uint64_t acked_lsn);
/*
* ha_log_quorum_transition: emit an operator-visible WARN on each write-quorum
* loss/restore EDGE (ha-convergence-recovery task 5.x, Req 8.1, 8.2, 8.3).
*
* Evaluates the current write-quorum state and compares it against the
* ha_state_t.quorum_lost edge-detector flag. On a TRANSITION (and only on a
* transition), it atomically flips the flag via CAS and emits exactly one WARN:
* - loss: "HA quorum lost: connected=%d required=%d" (Req 8.1)
* - restore: "HA quorum restored: connected=%d required=%d" (Req 8.2)
* The flag makes the logging an edge detector, NOT a latch: repeated calls while
* the state is unchanged emit nothing (Req 8.3 — no log storms across repeated
* dispatch-gate rejections), but a later loss/restore cycle logs again. The CAS
* guarantees once-per-transition under concurrency.
*
* Call sites (task 5.2): the standby_endpoints mutation points (promotion
* registration, RDHUP removal, on_standby_connected) plus the ha_dispatch_gate
* quorum-rejection branch as a backstop. Observability only — never gates an
* operation outcome.
*
* Defined in ha_convergence.c (a unit-linkable TU) so the quorum-transition
* tests can drive it directly against a real persist/repl fixture.
*
* RED scaffold (task 5.1): the current definition is a NO-OP — it performs no
* edge detection, flips no flag, and emits no WARN — so the quorum-transition
* tests FAIL until task 5.2 lands the real edge-detection logic and wires the
* call sites.
*/
void ha_log_quorum_transition(ha_state_t *ha);
/* ── Redirect helpers used by join.c ──────────────────────── */
/*
* ha_should_redirect: returns 1 if this controller should redirect join
* requests to the primary.
* Returns 0 if: (a) role is PRIMARY, or (b) primary_host is empty /
* primary_port is 0 (no primary known yet — client should retry).
*/
int ha_should_redirect(const ha_state_t *ha);
/*
* ha_no_primary_known: returns 1 if role is STANDBY and primary_host is
* empty or primary_port is 0. Caller should reply E_XCE_NOMASTER so the
* client knows to retry later rather than treating the error as permanent.
*/
int ha_no_primary_known(const ha_state_t *ha);
/*
* ha_is_primary: returns 1 if ha->role is CTRL_PRIMARY, 0 otherwise.
* Lock-free. Safe to call from any thread.
*/
int ha_is_primary(const ha_state_t *ha);
/* ha_check_priority_stepdown() deleted — replaced by quorum election (Req 2.1) */
/* start_heartbeat_listener() / stop_heartbeat_listener() deleted — never called (election-iomgr-refactor) */
/*
* ha_dispatch_gate: role-based operation filter for the vtaskmgr dispatch loop.
* Returns:
* 0 — operation is permitted for the current role
* -E_XCE_NOT_READY — role is CTRL_INIT or CTRL_ELECTING and opcode is not
* OP_requestVote or OP_probeConfigTag
* -E_XCE_STANDBY — role is CTRL_STANDBY and opcode is a primary-only op
* CTRL_PRIMARY always returns 0.
*/
int ha_dispatch_gate(ha_state_t *ha, int opcode);
/*
* ha_apply_wal_catchup_entry: apply a single WAL catch-up entry to local
* persistent storage and advance ha->applied_lsn on success.
*
* Called from cluster.c OPCode_walCatchupEntry dispatch handler (Task 2.1).
* On success (rc == 0): applied_lsn advances to entry->lsn+1 and a
* WalAckReply {term, lsn, success=1} is sent back to the primary via
* xce_reply(ep, c->myhost, &reply) (Task 3.1, Req 3.1).
* On error (rc < 0): logs ERROR, returns error; no ACK sent (Task 3.2).
* applied_lsn is in-memory only — not persisted to disk.
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK is skipped.
* Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
*/
int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
wal_catchup_entry_t *entry,
struct endpoint_t *ep, cluster_t *c);
/* ha_nop_reply_state_report — populate a heartbeat reply with stateReport.
* Fills *sr from ha->applied_lsn (atomic read) and myconf->config_version_tag,
* then sets reply_out->data_which = stateReportMsg and
* reply_out->data.state_report = sr.
* Caller must keep *sr live until xce_reply() serialises the reply. */
void ha_nop_reply_state_report(ha_state_t *ha, myconf_t *myconf,
state_report_t *sr, xce_reply_t *reply_out);
/* ── Task 4.1 (ha-repl-hardening): safe commit point computation ─────────────
*
* ha_compute_safe_committed_lsn: determine safe_committed_lsn and safe_config_tag
* from vote replies collected in ctx after winning an election.
*
* Collects LSNs from all granted voters (self + peers with vote_granted=1),
* sorts descending, and returns the majority-threshold element.
*
* single-node (n_peers=0): returns ctx->self_lsn and ctx->self_tag directly —
* self is the only majority; returning 0 would truncate all WAL on re-election.
*
* Requirements: 2.3 (ha-repl-hardening spec)
* Task 6.3: single-node fix (self_lsn/self_tag fields added to ha_elect_ctx_t)
*/
void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
uint64_t *safe_lsn_out,
uint64_t *safe_tag_out);
/* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ──────────
*
* ha_promotion_gate_decision: PURE pre-promotion eligibility decision (Req 2.2,
* 2.5, 2.6). Maps the promoting member's recovery/durability state to the Boot
* Policy Decision Table outcome by building recover_policy_inputs_t and
* delegating to persist_recover_decision (DRY — the 3.2/3.3 decision table is
* the single source of truth; this gate does not re-derive it). Note the
* FENCE_UNRECOVERABLE terminal is a LOUD PERMANENT NON-promotable fence that
* STAYS ALIVE and keeps retrying catch-up — it is NEVER process exit
* (never-self-exit invariant, 2026-06-18):
* - recovery_truncated → the task-3.1 structural-corruption flag.
* - durable_floor → the local CLAMPED durable position (read AFTER the
* promotion-path persist_truncate_wal_above so the
* 2.2 truncate-clamp is reflected).
* - safe_hwm → the cluster committed high-water mark from the voter
* replies (ha_compute_safe_committed_lsn), NOT a stale
* ctrlhello.
* - peer_reachable → a healthy peer is reachable to catch up from.
* - catch_up_succeeded → catch-up reached a committed-consistent state.
* A member is "behind the HWM" iff durable_floor < safe_hwm. Returns
* RECOVER_DECISION_{ELIGIBLE,FENCE,FATAL}; a hollow node (truncated OR behind)
* is never ELIGIBLE until it catches up, regardless of quorum size (Req 2.5).
*/
recover_decision_t ha_promotion_gate_decision(bool recovery_truncated,
uint64_t durable_floor,
uint64_t safe_hwm,
bool peer_reachable,
bool catch_up_succeeded);
/* ha_evaluate_promotion_fence: live pre-promotion gate over ha->in_catch_up
* (Req 2.2, 2.6). Evaluates ha_promotion_gate_decision and applies the fence:
* - ELIGIBLE → clears the in_catch_up fence and returns ELIGIBLE (promotion
* may proceed; a member that has caught up past the committed
* boundary regains eligibility — Req 2.6).
* - FENCE → sets the in_catch_up fence (refuse promotion; catch up from a
* healthy peer) and returns FENCE.
* - FENCE_UNRECOVERABLE → routes through
* persist_recover_loud_fence_if_unrecoverable (logs LOUDLY at
* ERROR + sets the sticky persist recovery_unrecoverable flag),
* also sets the in_catch_up fence, and returns
* FENCE_UNRECOVERABLE. The process STAYS ALIVE, non-promotable,
* retrying catch-up — it NEVER self-exits (Req 2.4).
* Reuses the existing in_catch_up fence — does NOT invent a parallel fence. */
recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha,
bool recovery_truncated,
uint64_t durable_floor,
uint64_t safe_hwm,
bool peer_reachable,
bool catch_up_succeeded);
/* ── ha-commit-durability task 4.4: catch-up re-alert window ─────────────────
* (never-self-exit rework 2026-06-18.)
*
* When a recovery-truncated / behind member wins an election with no reachable
* peer AT THAT INSTANT (a transient no-peer snapshot during aggressive
* failover), it FENCES (sets in_catch_up, does NOT become primary) and arms a
* delaymgr re-alert timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS. When the
* timer fires, if the member is STILL recovery-truncated/behind AND no healthy
* peer is reachable AND no catch-up progress was made, it RE-EMITS the loud
* operator alert (persist_recover_loud_fence_if_unrecoverable) and RE-ARMS —
* the process STAYS ALIVE and keeps retrying catch-up INDEFINITELY. It NEVER
* terminates: a controller must never self-exit as a designed response to an
* operational condition. Catch-up that brings durable_floor >= HWM releases
* the fence + clears the sticky flag and stops the re-alert (Req 2.6); shutdown
* cancels it too.
*
* ha_promotion_fence_window_ms: the re-alert interval in milliseconds.
* Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via
* ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */
uint32_t ha_promotion_fence_window_ms(void);
/* ha_cancel_promotion_fence_timer: cancel a pending re-alert timer (idempotent;
* no-op when none is armed). Called when the member catches up (fence released)
* or on shutdown, so a stopped member stops re-alerting. It never gated a
* process exit — the timer never terminates the process. */
void ha_cancel_promotion_fence_timer(ha_state_t *ha);
#ifdef XCE_TESTING
/* Test seam: override the re-alert window so the timer fires deterministically
* fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS default. XCE_TESTING
* only — compiled out of production. */
void ha_set_promotion_fence_window_ms(uint32_t ms);
#endif
/* ── Task 3.2: Unified replication decision enum and evaluator ────────────── */
/*
* ha_repl_action_t: result of ha_evaluate_repl_action().
*
* Encodes the single replication action the primary should take for a standby
* on the current heartbeat cycle, based solely on the standby's reported
* (config_tag, applied_lsn) vs the primary's committed state.
*
* Decision tree (design §Unified Heartbeat-Driven Protocol):
* standby_tag < committed_tag → PUSH_CONFIG
* standby_lsn < oldest_wal_lsn → SEND_SNAPSHOT
* standby_lsn < committed_lsn → STREAM_WAL
* tag >= committed AND lsn >= committed → CONVERGED
*
* Requirements: 2.2, 2.3, 2.4 (ha-repl-hardening spec)
*/
typedef enum {
HA_REPL_PUSH_CONFIG, /* standby config tag is behind — push config first */
HA_REPL_SEND_SNAPSHOT, /* standby LSN predates oldest WAL — transfer snapshot */
HA_REPL_STREAM_WAL, /* standby LSN is within WAL window — stream entries */
HA_REPL_CONVERGED, /* standby tag and LSN are both current */
} ha_repl_action_t;
/*
* ha_evaluate_repl_action: pure decision function — no side effects.
*
* Evaluates the standby's reported (standby_tag, standby_lsn) against the
* primary's committed state (committed_tag, committed_lsn, oldest_wal_lsn)
* and returns the single action the primary should take this heartbeat cycle.
*
* Arguments:
* standby_tag — config_tag last reported by this standby
* standby_lsn — applied_lsn last reported by this standby
* committed_tag — primary's committed config_tag
* committed_lsn — primary's committed WAL LSN
* oldest_wal_lsn — oldest retained WAL LSN (entries before this are gone)
*
* Requirements: 2.2, 2.3, 2.4 (ha-repl-hardening spec)
* Boundary: Primary Heartbeat Handler (ha.c)
*/
ha_repl_action_t ha_evaluate_repl_action(uint64_t standby_tag,
uint64_t standby_lsn,
uint64_t committed_tag,
uint64_t committed_lsn,
uint64_t oldest_wal_lsn);
/*
* ha_process_standby_state: update per-standby convergence tracking.
*
* Called on each heartbeat reply from a standby while the primary is in
* CTRL_PRIMARY. Stores the standby's reported
* (config_tag, applied_lsn) and evaluates whether the standby has converged
* with the primary's committed state.
*
* Convergence criteria (Req 1.3):
* reported_tag >= committed_tag AND reported_lsn >= committed_lsn
*
* Logs INFO on the first transition from not-converged → converged (Req 7.2).
* Regression (converged → not-converged) clears standby_converged[peer_idx]
* and re-evaluates (Req 2.5).
*
* peer_idx: 0-based index into ha->standby_reported_* arrays; must be in
* [0, MAX_HA_CANDIDATES).
*
* Requirements: 1.2, 2.2, 2.5 (ha-repl-hardening spec)
* Boundary: Primary Heartbeat Handler (ha.c)
*/
void ha_process_standby_state(ha_state_t *ha, int peer_idx,
uint64_t reported_tag, uint64_t reported_lsn,
uint64_t committed_tag, uint64_t committed_lsn);
/* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */
/*
* ha_handle_snapshot_chunk: receive one snapshot transfer chunk on the standby.
*
* Each chunk's payload bytes are written (or appended) to snap.part in snap_dir.
* When chunk->is_last is non-zero:
* 1. Verify total bytes written == chunk->total_size.
* 2. Atomically rename snap.part → snap-{snapshot_lsn:016x}.bin.
* 3. Load the snapshot via persist_load_snapshot().
* 4. Set ha->applied_lsn = p->checkpoint_lsn (in-memory only).
*
* On mid-transfer reconnect (caller resets state), snap.part is discarded.
* Returns 0 on success, negative errno on error.
*
* Requirements: 4.2, 4.3, 4.4 (ha-repl-hardening spec)
* Boundary: Standby Dispatch (cluster.c, ha.c)
*/
int ha_handle_snapshot_chunk(ha_state_t *ha, persist_t *p,
const char *snap_dir,
snapshot_transfer_t *chunk,
const uint8_t *payload_bytes,
size_t payload_len);
/* ── HA Config Bootstrap & Term Durability (ha-config-bootstrap spec) ── */
/*
* ha_persist_term: write ha-term.bin atomically to {workdir}/ha-term.bin.
* Uses write + rename for durability (existing pattern in the codebase).
* Must be called before any vote is granted or term is incremented.
* If workdir is NULL, the file write is skipped (safe for unit tests).
*/
void ha_persist_term(ha_state_t *ha, const char *workdir);
/*
* ha_init_raft: initialize the embedded raft state machine with cluster topology.
*
* Must be called ONCE at controller startup after config is loaded and
* ha_restore_term() has run. Subsequent elections (initial and re-elections
* after primary loss) reuse the same raft state — only raft_start_election()
* and raft_on_leader_lost() are called for each election cycle.
*
* Calling raft_init() more than once would reset raft's internal state
* (term, voted_for, role) and could lose durable election progress.
*
* n_peers: total cluster size including self (candidates_count from config).
*/
void ha_init_raft(ha_state_t *ha, cluster_t *c, int n_peers);
/*
* ha_restore_term: read {workdir}/ha-term.bin at startup and restore
* ha->term and ha->voted_for_* into ha_state_t.
*
* On missing file (ENOENT): no-op — term stays 0, voted_for_name stays empty.
* Returns 0 (safe first-boot path). Req 9.3.
* On CRC/magic mismatch: log WARN, zero ha->term and ha->voted_for_* for
* safety and return 0. Req 9.2.
* On valid file: atomically restore ha->term; copy voted_for_term
* and voted_for_name. Returns 0. Req 9.2.
* On unexpected I/O error: return -errno.
*
* Must be called after ha_init() and before any election or vote processing.
* If workdir is NULL, the function is a no-op (safe for unit tests).
* Requirements: 9.1, 9.2, 9.3.
*/
int ha_restore_term(ha_state_t *ha, const char *workdir);
/*
* handle_request_vote: process an incoming OP_requestVote RPC.
* Applies term/voted_for/freshness gate; persists state via ha_persist_term().
* Returns 1 = vote granted, 0 = vote denied, <0 = error.
* Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1
*
* TEST-ONLY (XCE_TESTING): production vote dispatch is cluster.c →
* raft_on_vote_request. This is the legacy entry point for test-only
* role-gate unit tests (test_ha_vote_gate.c, test_ha_election.c).
*/
#ifdef XCE_TESTING
int handle_request_vote(ha_state_t *ha,
uint64_t term,
const char *candidate_id,
uint64_t config_tag,
uint64_t last_lsn,
uint64_t my_config_tag,
uint64_t my_last_lsn,
int in_catch_up_flag);
#endif
/*
* ha_vote_self_freshness: compute the voter's effective freshness floor.
*
* Election safety: a voter must deny candidates that are outdated against the
* strongest knowledge it has of the cluster's COMMITTED state — not merely its
* own local config file / applied LSN. The floor is the per-field max of the
* voter's local state and its learned-committed state:
*
* *out_tag = max(local_config_tag, atomic committed_config_tag)
* *out_lsn = max(local_applied_lsn, atomic last_known_committed_lsn)
*
* Uses acquire loads of the atomic learned-committed fields; never reads the
* non-atomic last_known_committed_config_tag (data-race safe). Pure aside from
* the atomic reads — no state mutation.
*/
void ha_vote_self_freshness(ha_state_t *ha,
uint64_t local_config_tag,
uint64_t local_applied_lsn,
uint64_t *out_tag,
uint64_t *out_lsn);
/*
* ha_candidate_advertise_lsn: the LSN a candidate advertises in a RequestVote /
* PreVote payload (ha-commit-durability task 4.2).
*
* Returns the BARE durable WAL tip persist_durable_floor(p) — NOT the folded
* max(durable_floor, last_known_committed_lsn) ha_vote_self_freshness computes
* for the voter floor. A candidate may only claim a log position it can
* durably back, so a node restarted with a truncated log cannot over-advertise
* a stale volatile learned-committed value and win (design §1a/§5a). The
* candidate's TAG axis still uses ha_vote_self_freshness (config epoch is
* orthogonal to log durability). NULL-safe (→ 0).
*/
uint64_t ha_candidate_advertise_lsn(persist_t *p);
/*
* ha_elect_evaluate_and_signal: evaluate accumulated ha_elect_ctx_t votes.
* Sets ctx->result (1=primary, 0=standby) and posts ctx->done semaphore.
* Applies quorum check, fresher-peer yield gate, and config_tag=0 fatal gate.
* Requirements: 2.1, 2.7, 2.8
*/
ctrl_role_t ha_elect_evaluate_and_signal(ha_state_t *ha, ha_elect_ctx_t *ctx);
/*
* handle_request_vote_reply: process an incoming reply to an OP_requestVote RPC.
*
* Called by each per-peer vote-request thread once the peer replies (or times out).
* Updates the entry in ctx->votes[peer_idx] with the reply fields.
* Increments ctx->votes_granted if vote_granted is non-zero.
* Increments ctx->n_replied unconditionally.
* Updates ctx->max_resp_tag / ctx->max_resp_lsn to track the freshest peer seen.
* When quorum is reached (votes_granted >= quorum) OR all peers have replied
* (n_replied >= n_peers), calls ha_elect_evaluate_and_signal() to post ctx->done.
*
* peer_idx: 0-based index into ctx->votes[] identifying the replying peer.
* term: term the peer reported in its reply (used for fencing if > ha->term).
* vote_granted: non-zero if the peer granted the vote, 0 if denied.
* peer_config_tag: peer's current config_tag at reply time.
* peer_lsn: peer's current WAL LSN at reply time.
*
* Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1
*/
void handle_request_vote_reply(ha_state_t *ha,
ha_elect_ctx_t *ctx,
int peer_idx,
uint64_t term,
int vote_granted,
uint64_t peer_config_tag,
uint64_t peer_lsn);
/*
* xce_probe_config_tag: single-RTT OP_probeConfigTag RPC to one peer.
* Fills *tag_out with the peer's current config_tag on success.
* Returns 0 on success, -errno on connection/protocol error.
*/
int xce_probe_config_tag(const char *host, uint16_t port, uint64_t *tag_out);
/*
* ha_wait_for_heartbeat_or_timeout: wait in CTRL_INIT for an incoming heartbeat.
*
* Blocks on sem_timedwait(&ha->heartbeat_seen, &deadline) where
* deadline = now(CLOCK_REALTIME) + election_timeout_ms + xorshift64_jitter
*
* Returns CTRL_STANDBY — heartbeat arrived; ha_do_demote() was called
* (role is now CTRL_STANDBY).
* Returns CTRL_ELECTING — deadline expired; caller must drive the election.
*
* The heartbeat handler posts heartbeat_seen only when role == CTRL_INIT.
* OPCode_heartbeat is allowed through ha_dispatch_gate() in CTRL_INIT state
* so that the primary's periodic heartbeat can reach the handler.
*
* Requirements: 1.1 (heartbeat → CTRL_STANDBY), 1.2 (timeout → CTRL_ELECTING).
* Boundary: Controller FSM (ha.c).
*/
ctrl_role_t ha_wait_for_heartbeat_or_timeout(ha_state_t *ha);
/* ── Task 9.4: EPOLLRDHUP Failover and Heartbeat Timeout (Req 6.2–6.4, 6.8) ─
* ha_on_primary_ctrl_fd_rdhup: EPOLLRDHUP event handler.
* Closes primary_ctrl_fd, transitions to CTRL_ELECTING, truncates WAL to
* last_known_committed_lsn, fires ha_start_election_async (via cluster when non-NULL).
* Pass c == NULL to skip persist/election (unit test mode).
*/
void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c);
/*
* ha_arm_heartbeat_timeout_vtask: start/re-arm the heartbeat watchdog thread.
* Fires ha_on_primary_ctrl_fd_rdhup after election_timeout_ms + rand_jitter of
* silence. Calling ha_on_heartbeat_received() resets the deadline.
*/
void ha_arm_heartbeat_timeout_vtask(ha_state_t *ha, cluster_t *c);
/* ha_discover_primary: step down from CTRL_ELECTING to CTRL_STANDBY when a
* live primary message (NOP, config push, ctrlhello, WAL, snapshot) arrives.
* Safe to call from any state — no-ops unless role == CTRL_ELECTING. */
void ha_discover_primary(ha_state_t *ha, cluster_t *c);
/* ha_on_heartbeat_received: reset the heartbeat deadline (Req 6.3). */
void ha_on_heartbeat_received(ha_state_t *ha);
/* ha_heartbeat_is_fresh: 1 if a primary message arrived within
* heartbeat_timeout_ms (leader-stickiness predicate). Exposed for PreVote
* denial-reason logging in cluster.c; pure read, no state change. */
int ha_heartbeat_is_fresh(ha_state_t *ha);
/* ha_disarm_heartbeat_timeout_vtask: stop the heartbeat watchdog thread. */
void ha_disarm_heartbeat_timeout_vtask(ha_state_t *ha);
/*
* ha_start_primary_heartbeat: arm a repeating delaymgr task that sends
* OPCode_nop to every standby in ha->standby_endpoints every
* heartbeat_interval_ms. Keeps the standby-side watchdog from timing out
* when the cluster is idle (no WAL entries, no config pushes).
* Must be called after ha->role is set to CTRL_PRIMARY.
* No-op when c == NULL (test mode).
*/
void ha_start_primary_heartbeat(cluster_t *c);
/* ── Task 2.4: Non-blocking async election initiator ────────────────────── */
/*
* ha_start_election_async: initiate a quorum election without blocking.
*
* Steps:
* 1. Store n_peers in ha->n_peers.
* 2. For single-node (n_peers == 1): call ha_elect_evaluate_and_signal directly
* with a self-only ha_elect_ctx_t — shortcircuit, no iomgr/delaymgr needed.
* 3. For multi-node: for each peer in peers[], allocate vote_ep_ctx_t, init
* endpoint with client_ssl_ctx, set on_vote_connected + vote_ep_err_handler,
* call endpoint_connect + iomgr_monitor; skip peer on connect failure.
* Allocate election_timeout_ctx_t (cancelled=0) and schedule
* delay_task_ms(delaymgr, election_timeout_ms, 1, election_timeout_fn, ctx).
* 4. Returns immediately; election result delivered via ha_elect_evaluate_and_signal.
*
* Observable: returns in < 5ms; endpoints registered in iomgr; watchdog in delaymgr.
* Requirements: 1.3 (non-blocking), 2.1 (single-node shortcircuit), 2.4 (async vote)
* Boundary: ha.c (election orchestration)
*/
void ha_start_election_async(ha_state_t *ha, cluster_t *c,
int n_peers, xce_candidate_t **peers,
uint64_t my_config_tag, uint64_t my_lsn,
const char *my_name);
/*
* ha_start_prevote_async: start a side-effect-free PreVote round before any
* real election (ha-prevote task 4.1).
*
* Snapshots self tag/lsn, calls raft_begin_pre_vote (self-grant + per-peer
* preVoteRequest send), and arms a round timeout bounded by election_timeout_ms
* (Req 5.4). The node stays CTRL_STANDBY and mutates NO persistent state
* (no term / voted_for / role change) for the duration of the round. Replies
* are fed to the reply handler (pre_vote_reply_dispatch); on a majority WIN the
* unchanged real-election sequence runs (truncate + CTRL_ELECTING +
* ha_start_election_async), on a LOSS / round-timeout the node stays
* CTRL_STANDBY and re-arms the liveness watchdog.
*
* The failover triggers are rewired to call this in task 4.2; until then the
* round is inert in production. c may be NULL in unit-test mode.
* Requirements: 1.1, 1.4, 1.5, 5.4, 7.3.
*/
void ha_start_prevote_async(ha_state_t *ha, cluster_t *c);
/*
* ha_begin_failover: single failover entry point (ha-prevote task 4.2).
*
* Called by both failover triggers (ha_on_primary_ctrl_fd_rdhup and, through
* it, the liveness-timeout poll) AFTER connection-teardown bookkeeping and the
* config-less voter-only guard. Starts a side-effect-free PreVote round
* (ha_start_prevote_async); does NOT truncate the WAL or transition to
* CTRL_ELECTING — those destructive steps are deferred to PreVote success.
* c may be NULL in unit-test mode. Requirements: 1.1, 4.3, 4.4.
*/
void ha_begin_failover(ha_state_t *ha, cluster_t *c);
/*
* ha_election_stagger_ms: per-node desync delay (ms) for a FAILOVER re-election.
*
* Mirrors the startup stagger (myapp.c): my_priority * election_priority_step_ms
* plus a fresh random jitter in [0, election_jitter_max_ms]. Applied before the
* PreVote start on the failover path so two survivors that lose the primary at
* the same instant do not campaign in lock-step and split the vote forever
* (Raft §5.2 randomized election timeout; bug-606). Re-randomized each call —
* the jitter_seed advances — so repeated collisions still converge.
*/
uint32_t ha_election_stagger_ms(ha_state_t *ha);
/*
* ha_dispatch_repl_action_for_standby: evaluate the replication decision for
* one standby and execute the resulting action.
*
* Reads ha->standby_reported_tag[peer_idx] and ha->standby_reported_lsn[peer_idx]
* (stored by ha_process_standby_state on the previous call), calls
* ha_evaluate_repl_action, and acts on the result:
* - HA_REPL_PUSH_CONFIG: In production builds, calls push_config_to_standbys(c, …).
* In XCE_TESTING builds, calls ha_repl_push_config_for_standby_hook(c, peer_idx).
* - HA_REPL_STREAM_WAL: calls ha_stream_wal_to_standby(ep, wal_path, standby_lsn,
* committed_lsn) to batch-stream WAL entries via xce_request() (non-blocking).
* In XCE_TESTING builds, calls ha_repl_stream_wal_hook if non-NULL.
* - HA_REPL_SEND_SNAPSHOT: calls ha_send_snapshot_to_standby(ep, snap_path, lsn).
* In XCE_TESTING builds, calls ha_repl_send_snapshot_hook(peer_idx) if non-NULL.
* - HA_REPL_CONVERGED: no-op.
*
* One action per heartbeat cycle per standby (design §Unified Heartbeat-Driven
* Protocol). CONVERGED is a no-op.
*
* ep — outbound endpoint to the standby (used for STREAM_WAL; may be
* NULL in tests that only exercise PUSH_CONFIG or CONVERGED paths)
* peer_idx — 0-based index into ha->standby_reported_* arrays
* committed_tag — primary's committed config_tag (from c->myconf or test fixture)
* committed_lsn — primary's committed WAL LSN (from c->persist or test fixture)
* oldest_wal_lsn— oldest retained WAL LSN (0 = all WAL retained, skip snapshot)
*
* Returns 0 on success; negative on internal error.
* Requirements: 1.4, 5.1–5.5 (ha-repl-hardening spec)
*/
int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
cluster_t *c,
endpoint_t *ep,
int peer_idx,
uint64_t committed_tag,
uint64_t committed_lsn,
uint64_t oldest_wal_lsn);
/*
* ha_send_ctrlhello_to_all_standbys: iterate ha->standby_endpoints and send
* an OP_hello CtrlHelloRequest to each connected standby.
* This triggers the existing catch-up replication path in handle_ctrlhello
* (Req 7.2). Defined in cluster.c; called from ha_elect_evaluate_and_signal
* in the post-election path.
*/
void ha_send_ctrlhello_to_all_standbys(cluster_t *c);
/* ── P3-6: Test-only failover shim ────────────────────────────── */
#ifdef XCE_TESTING
/*
* ha_phase2_hook: test-only seam for cluster_init_phase2.
* When non-NULL, ha_elect_evaluate_and_signal() calls this INSTEAD OF the
* real cluster_init_phase2() so unit tests can count phase2 calls without
* requiring a real cluster_t. Pattern mirrors xce_vote_request_hook.
*/
extern int (*ha_phase2_hook)(void *c);
/*
* ha_send_ctrlhello_hook: test-only seam for ha_send_ctrlhello_to_all_standbys.
* When non-NULL, ha_elect_evaluate_and_signal() calls this in the post-election
* path instead of the real function (cluster.c not linked in test binary).
* Requirements: 7.2.
*/
extern int (*ha_send_ctrlhello_hook)(void *c);
/*
* ha_push_config_hook: test-only seam for push_config_to_standbys.
* When non-NULL, ha_elect_evaluate_and_signal() calls this in the post-election
* path instead of the real function. Requirements: 7.1.
*/
extern int (*ha_push_config_hook)(void *c);
/*
* ha_repl_push_config_for_standby_hook: test-only seam for the per-standby
* config push triggered by the heartbeat-reply decision loop (Task 3.3).
*
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
* push_config_to_standbys() when the decision is HA_REPL_PUSH_CONFIG, so unit
* tests can verify the wiring without requiring a real cluster_t or network.
*
* Signature: int hook(void *c, int peer_idx)
* c — cluster_t* (may be NULL in test mode)
* peer_idx — 0-based index of the standby that triggered the push
* Requirements: 1.4 (ha-repl-hardening)
*/
extern int (*ha_repl_push_config_for_standby_hook)(void *c, int peer_idx);
/*
* ha_repl_stream_wal_hook: test-only seam for ha_stream_wal_to_standby.
*
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
* ha_stream_wal_to_standby() when the decision is HA_REPL_STREAM_WAL, so unit
* tests can count batch-send invocations without requiring a real endpoint or
* WAL directory.
*
* Signature: int hook(void *ep, const char *wal_path,
* uint64_t standby_lsn, uint64_t committed_lsn)
* ep — endpoint_t* (may be NULL in test mode)
* wal_path — WAL directory path
* standby_lsn — standby's last applied LSN
* committed_lsn— primary's committed LSN
* Requirements: 5.1–5.5 (ha-repl-hardening)
*/
extern int (*ha_repl_stream_wal_hook)(void *ep, const char *wal_path,
uint64_t standby_lsn,
uint64_t committed_lsn);
/*
* ha_repl_send_snapshot_hook: test-only seam for ha_send_snapshot_to_standby.
*
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
* ha_send_snapshot_to_standby() when the decision is HA_REPL_SEND_SNAPSHOT,
* so unit tests can verify the wiring without requiring a real endpoint or
* snapshot directory.
*
* Signature: int hook(int peer_idx)
* peer_idx — 0-based index of the standby that triggered the snapshot
* Requirements: 4.1–4.5 (ha-repl-hardening)
*/
extern int (*ha_repl_send_snapshot_hook)(int peer_idx);
/*
* ha_tcp_connect_hook: test-only seam for the raw TCP connect inside
* ha_find_and_connect_to_primary. When non-NULL, the function calls this
* INSTEAD OF the real getaddrinfo/connect path, allowing unit tests to inject
* socketpair fds without real network I/O.
* Signature: int hook(const char *host, int port)
* Returns fd >= 0 on success, or -errno on failure.
* Requirements: task 9.2.
*/
extern int (*ha_tcp_connect_hook)(const char *host, int port);
/* Test-only shim: demote then re-promote; returns new term.
* Exercises the manual failover state machine without a live cluster_t.
* Calls ha_do_demote() then ha_do_promote(); validates role transitions
* and term monotonicity. See design doc P3-6 and src/docs/persistence-and-ha-design.md.
*/
uint64_t ha_trigger_manual_failover_for_test(ha_state_t *ha);
/*
* ha_rdhup_election_hook: test-only seam for ha_start_election_async in the
* RDHUP failover path.
*
* When non-NULL, ha_on_primary_ctrl_fd_rdhup() calls this INSTEAD OF the real
* ha_start_election_async() so unit tests can verify the RDHUP path triggers
* async re-election without requiring a live iomgr/delaymgr.
*
* The hook receives the same arguments that would be passed to ha_start_election_async.
* Requirements: 6.3, 6.5, 9.3.
*/
extern void (*ha_rdhup_election_hook)(ha_state_t *ha, cluster_t *c,
int n_peers, xce_candidate_t **peers,
uint64_t my_config_tag, uint64_t my_lsn,
const char *my_name);
/*
* vote_ep_ctx_set_cluster_for_test: set the cluster_t pointer inside the
* vote_ep_ctx_t stored at ep->data. Used by unit tests that need the
* promoted-endpoint dispatch path to reach ha_promote_to_primary(c) with a
* live cluster pointer.
* The vote_ep_ctx_t struct is internal to ha.c; this helper avoids exposing it.
* Requirements: task 7.3 (ha-peer-connection-reuse).
*/
void vote_ep_ctx_set_cluster_for_test(struct endpoint_t *ep, cluster_t *c);
/*
* ha_restart_hook: test-only seam for ha_self_restart's process re-execution
* (ha-convergence-recovery Req 1.6).
*
* When non-NULL, ha_self_restart() invokes this INSTEAD OF execv() so unit
* tests can observe the eventual restart deterministically (the demotion
* convergence task and rejoin reconciliation both end in ha_self_restart).
* The hook's return value is ha_self_restart's return value on the
* test-success path; a NON-ZERO return models an exec failure and routes
* ha_self_restart to the FATAL contract via ha_fatal_hook (Req 1.4).
*
* Signature: int hook(cluster_t *c)
* Returns 0 to model a successful restart; non-zero to model exec failure.
*/
extern int (*ha_restart_hook)(cluster_t *c);
/*
* ha_fatal_hook: test-only seam for the FATAL (abort) contract of
* ha_self_restart (ha-convergence-recovery Req 1.4).
*
* When non-NULL, ha_self_restart() invokes this INSTEAD OF abort() when the
* restart cannot be performed, so a test binary can assert the FATAL path was
* taken without the process dying. ha_self_restart returns after invoking it
* (test builds only — production aborts).
*
* Signature: void hook(const char *msg)
*/
extern void (*ha_fatal_hook)(const char *msg);
/*
* raft_cb_become_follower_with_cluster_for_test: drive the follower path with
* a LIVE cluster pointer (ha-convergence-recovery task 2.1 demotion-sequence
* tests). Unlike raft_cb_become_follower_for_test (which forces raft_c = NULL
* to suppress production calls), this shim sets ha->raft_c = c first so the
* cluster-dependent demotion-convergence branch (task 2.2: hook disable +
* scheduled truncate/restart) is exercised against a real persist/hooksys/
* delaymgr fixture. Mirrors the real-cluster seam ha_promote_to_primary(c)
* used by the promotion-side lifecycle tests.
*/
void raft_cb_become_follower_with_cluster_for_test(cluster_t *c, uint64_t term);
/*
* raft_cb_become_leader_with_cluster_for_test: drive the leader promotion path
* with a LIVE cluster pointer (fix-promotion-standby-registration). Unlike
* raft_cb_become_leader_for_test (which forces raft_c = NULL and so skips the
* c-dependent standby-registration branch entirely), this shim sets
* ha->raft_c = c first so the promotion-time repl_add_standby loop — which
* registers each surviving vote endpoint (votes[i].ep != NULL) carrying a
* peer pointer (votes[i].peer != NULL) into the WAL replicator — is exercised
* against a real persist/repl/commit fixture. Symmetric with
* raft_cb_become_follower_with_cluster_for_test.
*/
void raft_cb_become_leader_with_cluster_for_test(cluster_t *c, uint64_t term);
/*
* ha_walack_continuation_for_test: test-only entry point for the ACK-driven
* catch-up continuation logic (ha-convergence-recovery task 4.x, Req 5.1, 5.2,
* 7.1).
*
* The production trigger lives in the walAck branch of
* repl_ack_decode_and_dispatch (ha.c), which decodes a real Cap'n Proto
* WalAckReply off the wire — too heavy to drive from a unit test. This shim
* exposes the SAME post-ACK decision the walAck branch performs after
* repl_on_ack, so the continuation / single-flight / parity behaviors can be
* exercised directly against a unit fixture:
*
* target = max(committed_lsn, last_written) (same derivation as ha.c:2771-2776)
* if acked_lsn < target → schedule ONE continuation that streams
* [acked_lsn+1, target] via the catch-up streaming
* path, UNLESS ha.catchup_inflight[peer_idx] is set
* (single-flight, Req 5.2); the continuation must not
* dispatch any heartbeat/stateReport action (Req 5.1).
* if acked_lsn >= target → repl_complete_catchup(&c->persist.repl, ep), which
* clears the standby's in_catch_up (Req 7.1).
*
* c — cluster_t* (real persist + repl registry; never NULL in tests)
* ep — repl registry endpoint key for the standby (the same pointer
* passed to repl_add_standby); identifies the standby slot.
* peer_idx — 0-based standby slot index (into ha.catchup_inflight[]).
* acked_lsn— the LSN the standby just acknowledged.
*
* Returns 0 on the no-op / parity paths and on a successfully scheduled
* continuation; negative on a scheduling failure.
*
* RED scaffold (task 4.1): ha_convergence.c currently records the ACK only and
* neither schedules a continuation nor calls repl_complete_catchup — task 4.2
* implements the real decision. The continuation / single-flight / parity tests
* therefore FAIL until 4.2 lands.
*/
int ha_walack_continuation_for_test(cluster_t *c, void *ep, int peer_idx,
uint64_t acked_lsn);
/*
* ha_quorum_transition_log_hook: test-only emission-count seam for
* ha_log_quorum_transition (ha-convergence-recovery task 5.x, Req 8.1-8.3).
*
* There is NO log-capture harness in this testsuite, so the quorum-transition
* tests use a counter/edge-observable instead (the task permits "log-capture OR
* counter-hook pattern"). When non-NULL, ha_log_quorum_transition invokes this
* hook EXACTLY ONCE on each transition it WARNs about, passing the new state and
* the connected/required counts that go into the WARN line. Tests count
* invocations to assert "logged once per transition" (Req 8.3) and inspect the
* arguments to assert the loss line carries the right counts (Req 8.1).
*
* Signature: void hook(int lost, int connected, int required)
* lost — 1 on a quorum-LOSS edge, 0 on a quorum-RESTORE edge.
* connected — connected standby count reported in the WARN line.
* required — required standby count (quorum threshold) in the WARN line.
*
* RED scaffold (task 5.1): ha_log_quorum_transition is a no-op stub that never
* invokes this hook, so the transition tests' emission-count assertions FAIL
* until task 5.2 implements the real edge detection.
*/
extern void (*ha_quorum_transition_log_hook)(int lost, int connected,
int required);
#endif
#endif /* HA_H */
View git blame Copy permalink