1208 lines
64 KiB
Markdown
1208 lines
64 KiB
Markdown
# Brief: ha-election-raft-config
|
||
|
||
## Problem
|
||
|
||
The current LSF-style priority-based election has three critical safety gaps:
|
||
|
||
1. **Config freshness not guaranteed**: A node with missing or stale config can be elected primary and serve incorrect cluster state to workers/schedulers.
|
||
|
||
2. **False self-elections during concurrent startup**: When pc13v1 starts while pc13v (the priority-0 primary) is still initializing, pc13v's peer listener port is not open yet. pc13v1's election probe gets ECONNREFUSED, falsely concludes the primary is absent, and self-elects. Both nodes declare themselves primary.
|
||
|
||
3. **No bootstrap path for genuine failover**: When a standby is promoted during real failover (original primary down), it has no mechanism to pull config from peer standbys if its local `conf/current` is missing.
|
||
|
||
Reproduction from logs:
|
||
- pc13v starts at 12:30:03, pc13v1 starts at 12:30:22 (19s later)
|
||
- pc13v1 probes pc13v:9990 during election, times out within 3046ms
|
||
- pc13v is healthy but still in `load_primary_config()` or early phase2 — listener not open
|
||
- pc13v1 wins: "no higher-priority candidate responded"
|
||
- pc13v1 calls `load_primary_config()` → "Primary has no config at '/tmp/xce/pc13v1/conf/current' (rc=-2)" → fatal exit
|
||
|
||
## Current State
|
||
|
||
- **Election** (ha.c:243–306): LSF-style priority-based. Priority-0 always wins unconditionally; priority-N probes [0..N-1] and self-elects on timeout. No quorum. No config-version check. Logically correct for simple cases but unsafe for config freshness.
|
||
- **Listener timing** (myapp.c:21–96): `bind()+listen()` happens inside `cluster_init_phase2()` AFTER election and config load. Creates a 2–3 second window where the primary is unreachable to probes.
|
||
- **Config versioning** (config_repl.c): SHA-256 tagged, atomic writes, retention policy, staleness detection (`conf_stale` flag). Complete and correct.
|
||
- **Config fetch** (config_repl.c:891–965): standby→primary pull framework exists; `do_one_config_fetch()` returns `-ENOSYS` (stub, Task 11 on branch).
|
||
- **Bootstrap after failover**: not implemented. Design §14.4 deferred quorum consensus.
|
||
- **Simultaneous primaries**: `ha_check_priority_stepdown()` (ha.c:322) has a term=0 exact check that fails to resolve races when both nodes increment term before heartbeat exchange.
|
||
|
||
## Availability Trade-off (CP chosen)
|
||
|
||
This design chooses **Consistency over Availability** (CP in CAP terms):
|
||
|
||
- A cluster of N nodes requires a **majority (≥(N+1)/2)** for both **election** and **write commits**.
|
||
- **Election**: A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored.
|
||
- **Writes (WAL replication)**: Primary waits for majority standby ACKs before confirming a write (job submit, queue change, state update). In-flight writes that cannot reach majority are rejected. This adds **~1 LAN RTT per write** (~0.1–5ms depending on network).
|
||
- **Zero data loss on failover**: A promoted standby is guaranteed to have all committed WAL entries because majority replication ensures durability before write ACK.
|
||
- **Operational implication**: Plan for N≥3 nodes. A 2-node cluster has no meaningful majority; it is equivalent to single-node with a cold standby. For HPC, zero data loss (no lost job submissions, no stale queue state) outweighs the write-latency cost.
|
||
|
||
## Desired Outcome
|
||
|
||
1. **Raft-style quorum election (CP)**: A candidate must obtain ACKs from a strict majority of peers (≥(N+1)/2) before becoming primary. A minority partition cannot elect a primary. The cluster is unavailable until the majority partition is restored.
|
||
|
||
2. **Config freshness encoded in votes**: Each vote reply includes the peer's config tag and WAL LSN. The candidate with the freshest config (highest tag, then highest LSN) among quorum members wins tiebreaks.
|
||
|
||
3. **No-config nodes bootstrap after election (deferred fetch)**: A node with `config_tag=0`
|
||
never initiates candidacy. It probes peers to determine whether config exists anywhere,
|
||
but **defers the actual fetch until the primary is known** — the election proves who
|
||
has the freshest state, so bootstrap fetches from the proven winner rather than
|
||
speculatively guessing. Three sub-cases:
|
||
- `tag=0`, peers have config → set `defer_config_fetch`, participate as voter only, fetch from winner after election
|
||
- `tag=0`, all peers also `tag=0` → first-boot cluster, proceed to election as-is
|
||
- `tag=0`, no peer reachable → retry: probe loop with exponential backoff (500ms→10s);
|
||
log WARN every attempt; block indefinitely until a peer responds or admin kills
|
||
|
||
4. **Listener before election**: Peer listener socket opens before election so concurrent-startup probes reach an open port. False self-elections during overlapping startup are eliminated.
|
||
|
||
5. **Config freshness preserved through all paths**: A node always has a valid config
|
||
before entering phase 2 — either from local disk (normal case) or fetched from the
|
||
proven election winner after the election resolves (no-config case, deferred fetch in
|
||
Change C). A quorum winner therefore always has `config_tag > 0` when peers do, and
|
||
serves correct cluster state immediately after promotion with no additional bootstrap step.
|
||
|
||
6. **Most up-to-date primary**: Among all reachable nodes, only the one with the
|
||
highest config_tag (then highest LSN) can become primary. A candidate that wins
|
||
the vote count but sees a fresher responding peer yields back to standby —
|
||
the fresher peer runs its own election and wins. Non-responding (partitioned or
|
||
crashed) peers never block election.
|
||
|
||
7. **No simultaneous primaries**: Only one node can obtain a quorum at any given term. Term-based fencing prevents split-brain on partition heal. `voted_for` tracking (persisted to `ha-term.bin`) enforces one vote per term per node across restarts.
|
||
|
||
8. **Majority-commit WAL replication (CP writes)**: Primary waits for majority (≥(N+1)/2) standby ACKs before marking a WAL entry committed and confirming the write to the caller. In-flight writes that cannot reach majority within a timeout are rejected. Zero data loss on failover: promoted standby has all committed entries.
|
||
|
||
9. **Majority-commit config push (CP config)**: Config upgrades (`OP_upgradeConfig`) use the
|
||
same `repl_broadcast()` + `repl_wait_majority_ack()` infrastructure as WAL replication —
|
||
the primary waits for ≥(N+1)/2 standby ACKs before confirming a config change to the
|
||
operator. Operator changes cannot be silently lost on failover.
|
||
|
||
10. **Hard SSL/cert gate**: Controller refuses to boot if TLS cert/key files are missing or unreadable. Operator-friendly error message with file path.
|
||
|
||
11. **Observable via logs**: Election results, vote tallies, quorum counts, config tag comparisons, bootstrap operations, WAL commit quorum status, config push quorum status, and cert validation logged at INFO or WARN severity.
|
||
|
||
## Approach
|
||
|
||
**Replace LSF priority election with Raft-style quorum voting AND add majority-commit WAL replication for full CP durability. Config upgrades use the same majority-commit infrastructure for uniform CP semantics.**
|
||
|
||
Eight focused changes:
|
||
|
||
### Change A — Move listener, iomgr, and vtaskmgr before election
|
||
|
||
Extract `bind()+listen()`, `iomgr_init()`, and `vtaskmgr_init()` from
|
||
`cluster_init_phase2()` so the full connection framework is running before election.
|
||
`vtaskmgr_init()` uses the new `--vcpus N` CLI parameter (Change D item 1) instead of
|
||
`cluster_config->vcpus`, removing the last config dependency on this startup block.
|
||
|
||
New startup order in `myapp.c`:
|
||
```
|
||
validate_peer_count() ← Change D: odd-N check before any init
|
||
validate_certs() ← Change D: cert pre-flight
|
||
cluster_init_phase1() ← ha_init() reads ha-term.bin: restores term + voted_for
|
||
← conf_read_tag() reads config_version_tag from disk (0 if absent)
|
||
← Node now has durable identity before network is up
|
||
create_ssl_context()
|
||
iomgr_init() + iomgr_set_ssl_ctx()
|
||
cluster_open_listener(port) ← bind + listen; register fd with iomgr
|
||
vtaskmgr_init(vcpus)
|
||
iomgr_start()
|
||
vtaskmgr_start()
|
||
atomic_store(ha.role, CTRL_INIT) ← node now answers OP_requestVote + OP_probeConfigTag
|
||
bootstrap_config_if_needed() ← Change C: if config_tag==0, probe peers; defers fetch until primary known
|
||
ha_wait_for_heartbeat_or_timeout() ← Raft: sem_timedwait(election_timeout + rand_jitter)
|
||
← heartbeat received → CTRL_STANDBY, skip election
|
||
← timeout fires → CTRL_ELECTING → am_i_primary_quorum()
|
||
[primary or standby path]
|
||
load_primary_config() / ha_connect_to_primary()
|
||
cluster_init_phase2_rest() ← kvmgr, xstring, jobmgr, delaymgr, apiserver, etc.
|
||
cluster_run()
|
||
```
|
||
|
||
With iomgr and vtaskmgr running before election, incoming `OP_requestVote` messages
|
||
are handled by the normal dispatch switch — no special pre-framework path needed.
|
||
Outgoing vote requests use the existing `connect_ssl_raw()` + `xce_encode()/xce_decode()`
|
||
pattern (same as config fetch), keeping per-peer threads lightweight.
|
||
|
||
**Full 4-state controller FSM (`ha_role_t`)**:
|
||
|
||
The controller lifecycle is a single FSM stored in `c->ha.role` (atomic):
|
||
```
|
||
CTRL_INIT ──── timeout + jitter ──→ CTRL_ELECTING ──── quorum won ──→ CTRL_PRIMARY
|
||
└─── heartbeat from │ │
|
||
existing primary ─→ └─── quorum lost ─→ CTRL_STANDBY ────┘
|
||
│
|
||
EPOLLRDHUP / heartbeat timeout
|
||
│
|
||
CTRL_ELECTING
|
||
```
|
||
|
||
**Dispatch gate** (top of vtaskmgr switch, before any `case`):
|
||
```c
|
||
switch (atomic_load(&c->ha.role)) {
|
||
case CTRL_INIT:
|
||
case CTRL_ELECTING:
|
||
// Only vote + config-probe pass; all others get NOT_READY and retry (Change G)
|
||
if (msg->op != OP_requestVote && msg->op != OP_probeConfigTag) {
|
||
xce_send_error(task, XCE_ERR_NOT_READY, "controller not ready");
|
||
return;
|
||
}
|
||
break;
|
||
case CTRL_STANDBY:
|
||
// Primary-only ops rejected with XCE_ERR_STANDBY + redirect_host/port (Change G)
|
||
break;
|
||
case CTRL_PRIMARY:
|
||
break; // full dispatch
|
||
}
|
||
```
|
||
|
||
**State transition triggers**:
|
||
- `CTRL_INIT → CTRL_ELECTING`: `ha_wait_for_heartbeat_or_timeout()` times out — no
|
||
heartbeat received within `election_timeout + rand_jitter`. Different jitter per node
|
||
(xorshift64 from `jitter_seed`) means nodes fire at staggered times, so the first
|
||
candidate typically wins unopposed. This is Raft's primary split-vote prevention.
|
||
- `CTRL_INIT → CTRL_STANDBY`: heartbeat or `OP_ctrlHello` received from an existing
|
||
primary during the wait window — node skips election entirely.
|
||
- `CTRL_ELECTING → CTRL_PRIMARY`: `am_i_primary_quorum()` returns 1.
|
||
- `CTRL_ELECTING → CTRL_STANDBY`: `am_i_primary_quorum()` returns 0.
|
||
- `CTRL_STANDBY → CTRL_ELECTING` (fast): `EPOLLRDHUP` on `primary_ctrl_fd`.
|
||
- `CTRL_STANDBY → CTRL_ELECTING` (fallback): heartbeat timeout vtask fires — primary
|
||
is alive on TCP but not sending heartbeats. Vtask is reset on each heartbeat received;
|
||
fires after `election_timeout + rand_jitter` of silence.
|
||
|
||
**`ha_wait_for_heartbeat_or_timeout()`** in `myapp.c`:
|
||
```c
|
||
struct timespec dl = now_plus_ms(c->ha.config.election_timeout_ms +
|
||
ha_rand_jitter(&c->ha));
|
||
int hb = (sem_timedwait(&c->ha.heartbeat_seen, &dl) == 0);
|
||
if (hb) {
|
||
atomic_store(&c->ha.role, CTRL_STANDBY);
|
||
} else {
|
||
atomic_store(&c->ha.role, CTRL_ELECTING);
|
||
rc = am_i_primary_quorum(c);
|
||
}
|
||
```
|
||
`ha.heartbeat_seen` is posted by the heartbeat dispatch handler when any heartbeat
|
||
or `OP_ctrlHello` is received while `ha.role == CTRL_INIT`.
|
||
|
||
### Change B — Raft-style quorum voting in `am_i_primary()`
|
||
|
||
Replace the current timeout-based probe with a full request-vote protocol:
|
||
|
||
**New wire message: OP_requestVote** (Cap'n Proto)
|
||
```c
|
||
struct RequestVoteRequest {
|
||
term @0 :UInt64; // candidate's term
|
||
candidate_id @1 :Text; // candidate's name (from --peers)
|
||
config_tag @2 :UInt64; // candidate's config SHA-256 tag (0 if none)
|
||
last_log_lsn @3 :UInt64; // candidate's WAL applied_lsn
|
||
// NOTE: priority removed — Raft config_tag+LSN freshness fully determines winner;
|
||
// jitter breaks split votes. Priority plays no role in quorum election.
|
||
}
|
||
|
||
struct RequestVoteReply {
|
||
term @0 :UInt64; // peer's current term
|
||
vote_granted @1 :Bool; // true = ACK
|
||
peer_config_tag @2 :UInt64; // peer's config tag (for config comparison)
|
||
peer_last_log_lsn @3 :UInt64; // peer's WAL LSN (for log comparison)
|
||
}
|
||
```
|
||
|
||
**Election uses iomgr + vtaskmgr (FSM model)**:
|
||
|
||
Since iomgr and vtaskmgr are running before election (Change A), vote requests are
|
||
sent via iomgr's async connect path and replies arrive through the normal vtaskmgr
|
||
dispatch switch. No election-specific threads are created.
|
||
|
||
**New `ha_vote_entry_t`** struct (ha.h) — per-peer election slice; only meaningful as
|
||
`ctx->votes[i]` inside `ha_elect_ctx_t`:
|
||
```c
|
||
typedef struct {
|
||
xce_candidate_t *peer;
|
||
uint64_t my_term; // term sent in RequestVote
|
||
uint64_t my_tag; // config_tag sent in RequestVote
|
||
uint64_t my_lsn; // WAL LSN sent in RequestVote
|
||
int vote_granted; // 1 = peer granted vote
|
||
int replied; // 1 = peer responded (any reply, including denial)
|
||
uint64_t peer_tag; // peer's config_version_tag from reply
|
||
uint64_t peer_lsn; // peer's applied_lsn from reply
|
||
// NOTE: no pthread_t — FSM uses iomgr async sends + vtaskmgr dispatch;
|
||
// ha_elect_ctx_t.done semaphore blocks am_i_primary_quorum()
|
||
} ha_vote_entry_t;
|
||
```
|
||
|
||
**New `ha_elect_ctx_t`** struct (ha.h) — holds all election state; single allocation via
|
||
C99 flexible array member (one `calloc`, one `free`):
|
||
```c
|
||
typedef struct {
|
||
int n_peers;
|
||
int votes_granted; // includes self-vote (starts at 1)
|
||
int n_replied; // peers that responded (any reply)
|
||
uint64_t max_resp_tag; // freshest config_tag seen across all replies
|
||
uint64_t max_resp_lsn; // freshest lsn seen across all replies
|
||
int result; // set before sem_post: 1=primary, 0=standby, <0=error
|
||
sem_t done; // startup thread blocks here; dispatch posts when done
|
||
ha_vote_entry_t votes[]; // C99 flexible array — n_peers-1 entries
|
||
} ha_elect_ctx_t;
|
||
// Allocate: calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0]))
|
||
// Free: free(ctx) — single allocation, no separate votes free needed
|
||
```
|
||
|
||
**`am_i_primary_quorum()`** (ha.c) — initiates election, blocks until done:
|
||
```c
|
||
int am_i_primary_quorum(cluster_t *c) {
|
||
int n_peers = c->pcfg.n_candidates;
|
||
if (n_peers == 1) return 1;
|
||
|
||
uint64_t my_term = atomic_fetch_add(&c->ha.term, 1) + 1;
|
||
ha_persist_term(c, my_term, c->myconf->name);
|
||
|
||
ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + (n_peers-1)*sizeof(ctx->votes[0]));
|
||
ctx->n_peers = n_peers;
|
||
ctx->votes_granted = 1; // self-vote
|
||
ctx->max_resp_tag = c->myconf->config_version_tag;
|
||
ctx->max_resp_lsn = atomic_load(&c->persist.applied_lsn);
|
||
sem_init(&ctx->done, 0, 0);
|
||
atomic_store(&c->ha.elect_ctx, ctx); // dispatch reads this
|
||
|
||
// Send RequestVote to all peers via iomgr (async non-blocking connects)
|
||
for (int i = 0, j = 0; i < n_peers; i++) {
|
||
if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue;
|
||
ctx->votes[j].peer = &c->pcfg.candidates[i];
|
||
ctx->votes[j].my_term = my_term;
|
||
ctx->votes[j].my_tag = c->myconf->config_version_tag;
|
||
ctx->votes[j].my_lsn = atomic_load(&c->persist.applied_lsn);
|
||
iomgr_async_vote_request(c, &ctx->votes[j]); // non-blocking
|
||
j++;
|
||
}
|
||
|
||
// Arm election timeout vtask — fires ha_elect_timeout() if peers don't reply
|
||
vtaskmgr_schedule(c, c->ha.config.election_timeout_ms, ha_elect_timeout, ctx);
|
||
|
||
sem_wait(&ctx->done); // startup blocks here; dispatch or timeout signals it
|
||
|
||
int result = ctx->result;
|
||
ha_elect_ctx_free(ctx);
|
||
atomic_store(&c->ha.elect_ctx, NULL);
|
||
return result;
|
||
}
|
||
```
|
||
|
||
**`handle_request_vote_reply()`** — called from vtaskmgr dispatch switch on
|
||
`OP_requestVoteReply`; accumulates votes and signals completion:
|
||
```c
|
||
void handle_request_vote_reply(cluster_t *c, RequestVoteReply *reply, int peer_idx) {
|
||
ha_elect_ctx_t *ctx = atomic_load(&c->ha.elect_ctx);
|
||
if (!ctx) return; // no election in progress
|
||
|
||
ha_vote_entry_t *v = &ctx->votes[peer_idx];
|
||
v->replied = 1;
|
||
v->peer_tag = reply->peer_config_tag;
|
||
v->peer_lsn = reply->peer_last_log_lsn;
|
||
if (reply->vote_granted) { v->vote_granted = 1; ctx->votes_granted++; }
|
||
if (v->peer_tag > ctx->max_resp_tag ||
|
||
(v->peer_tag == ctx->max_resp_tag && v->peer_lsn > ctx->max_resp_lsn)) {
|
||
ctx->max_resp_tag = v->peer_tag; ctx->max_resp_lsn = v->peer_lsn;
|
||
}
|
||
ctx->n_replied++;
|
||
|
||
int quorum = (ctx->n_peers + 1) / 2;
|
||
// Signal when quorum achieved or all peers replied (or timed out)
|
||
if (ctx->votes_granted >= quorum || ctx->n_replied == ctx->n_peers - 1)
|
||
ha_elect_evaluate_and_signal(c, ctx);
|
||
}
|
||
```
|
||
|
||
**`ha_elect_timeout()`** vtask — fires when `election_timeout_ms` expires:
|
||
counts unreplied peers as non-respondents, calls `ha_elect_evaluate_and_signal()`.
|
||
|
||
**`ha_elect_evaluate_and_signal()`** — applies gates, sets `ctx->result`, signals:
|
||
- Defense-in-depth assertion (config_tag=0 BUG check)
|
||
- Hard gate B (yield if responding peer has fresher data → `result=0`)
|
||
- Quorum met → `result=1`
|
||
- Quorum lost → schedule jitter vtask, retry once with incremented term;
|
||
if second round also fails → `result=0`; finally `sem_post(&ctx->done)`
|
||
|
||
**Vote handler on peers** (new function in ha.c):
|
||
```c
|
||
void handle_request_vote(cluster_t *c, RequestVoteRequest *req, RequestVoteReply *reply) {
|
||
pthread_mutex_lock(&c->ha.lock);
|
||
uint64_t my_term = atomic_load(&c->ha.term);
|
||
|
||
if (req->term < my_term) { // stale candidate — reject
|
||
reply->term = my_term;
|
||
reply->vote_granted = 0;
|
||
goto done;
|
||
}
|
||
if (req->term > my_term) { // update term from candidate
|
||
atomic_store(&c->ha.term, req->term);
|
||
my_term = req->term;
|
||
// Clear voted_for on new term transition
|
||
c->ha.voted_for_term = 0;
|
||
c->ha.voted_for_name[0] = '\0';
|
||
}
|
||
|
||
// Catch-up gate (Clarification 6): behind nodes must not influence elections
|
||
if (c->ha.in_catch_up) {
|
||
reply->vote_granted = 0;
|
||
goto fill_reply; // still send peer_config_tag/lsn so candidate has full picture
|
||
}
|
||
|
||
uint64_t my_tag = c->myconf->config_version_tag;
|
||
uint64_t my_lsn = atomic_load(&c->persist.applied_lsn);
|
||
int log_ok = (req->config_tag > my_tag) ||
|
||
(req->config_tag == my_tag && req->last_log_lsn >= my_lsn);
|
||
|
||
// One vote per term: grant only if not yet voted, or voted for same candidate
|
||
int already_voted = (c->ha.voted_for_term == my_term &&
|
||
strcmp(c->ha.voted_for_name, req->candidate_id) != 0);
|
||
reply->vote_granted = (log_ok && !already_voted) ? 1 : 0;
|
||
|
||
if (reply->vote_granted) {
|
||
c->ha.voted_for_term = my_term;
|
||
strncpy(c->ha.voted_for_name, req->candidate_id, XCE_MAXNAME_LEN - 1);
|
||
ha_persist_term(c, my_term, req->candidate_id); // durable before reply
|
||
}
|
||
done:
|
||
reply->term = my_term;
|
||
reply->peer_config_tag = my_tag;
|
||
reply->peer_last_log_lsn = my_lsn;
|
||
pthread_mutex_unlock(&c->ha.lock);
|
||
}
|
||
```
|
||
|
||
**Election safety extensions**:
|
||
|
||
1. **Term + `voted_for` persistence** (`ha-term.bin` in `{workdir}/`):
|
||
New `ha_persist_term(cluster_t *c, uint64_t term, const char *voted_for)` writes
|
||
a 4-field struct atomically (write + rename) before any vote is granted or term
|
||
incremented. On startup, `ha_init()` reads `ha-term.bin` and restores `ha.term`,
|
||
`ha.voted_for_term`, `ha.voted_for_name`. Without this, a restarting node resets
|
||
to term=0 and can double-vote in a term it already participated in.
|
||
|
||
2. **New fields in `ha_state_t`** (`ha.h`):
|
||
```c
|
||
uint64_t voted_for_term; // term of last granted vote
|
||
char voted_for_name[XCE_MAXNAME_LEN]; // candidate name of last granted vote
|
||
int primary_ctrl_fd; // persistent control channel fd (Change F; -1 = none)
|
||
bool defer_config_fetch; // Change C: set when tag=0 and peers have config;
|
||
// cleared after post-election fetch completes
|
||
```
|
||
|
||
3. **Retire `ha_check_priority_stepdown()`** (ha.c:322–368):
|
||
This function's guard `if (my_term > 0 || sender_term > 0) return` makes it
|
||
permanently unreachable after the first quorum election (term is always ≥ 1
|
||
thereafter). Change B explicitly removes the call site from the heartbeat handler
|
||
and deletes the function body. Term-based leadership is fully handled by the
|
||
`reply.term > my_term` check in `handle_request_vote()` and the existing
|
||
`ha_do_demote()` path.
|
||
|
||
**Key properties**:
|
||
- **Most up-to-date wins**: A candidate that won quorum still yields if any *responding*
|
||
peer has higher config_tag or (same tag, higher LSN). Non-responding peers never
|
||
block election — only reachable peers with fresher state can cause a yield.
|
||
- **Config freshness hard gate**: Yielding due to fresher peer returns 0 (standby, not
|
||
fatal). Only `config_tag == 0` with peers having `config_tag > 0` is fatal (-ENODATA),
|
||
because that means the node has never been configured at all.
|
||
- **Split vote prevention (primary)**: Each node waits `election_timeout + rand_jitter`
|
||
before candidacy (Change A `ha_wait_for_heartbeat_or_timeout()`). Different jitter
|
||
values stagger candidacy attempts so the first node typically wins unopposed — the
|
||
same xorshift64 `jitter_seed` used elsewhere. This is Raft's fundamental split-vote
|
||
prevention, not just recovery.
|
||
- **Split vote recovery (safety net)**: If a split vote still occurs inside
|
||
`ha_elect_evaluate_and_signal()`, a second attempt is scheduled with `rand_jitter`
|
||
delay and incremented term. If that also fails, node returns 0 (standby). This
|
||
safety net handles the rare case where multiple nodes have identical jitter values.
|
||
- **No priority tiebreak**: Priority is not used in Raft quorum election. When
|
||
config_tag and LSN are identical across all peers, the jitter-based pre-election
|
||
timeout is the tiebreaker. The `ha.my_priority` field is preserved in `ha_state_t`
|
||
for backward compatibility but plays no role in election outcome.
|
||
|
||
### Change C — Pre-election config bootstrap (deferred fetch)
|
||
|
||
A node with `config_tag=0` is a config-seeker, not a candidate. Before calling
|
||
`am_i_primary_quorum()`, `bootstrap_config_if_needed()` determines whether config
|
||
exists among peers, but **defers the actual fetch until the primary is known** —
|
||
avoiding a speculative download from a peer that may not win the election.
|
||
|
||
**Rationale**: Fetching from the highest-tag *reachable* peer before election can
|
||
waste I/O: a fresher peer may start late, win the election, and push its config
|
||
anyway. Since the election already proves who has the freshest state, bootstrap
|
||
should defer to that proof rather than guessing.
|
||
|
||
New function `bootstrap_config_if_needed()` in config_repl.c (called from myapp.c):
|
||
```c
|
||
void bootstrap_config_if_needed(cluster_t *c) {
|
||
if (c->myconf->config_version_tag != 0) return; // already have config
|
||
|
||
int retry_ms = 500;
|
||
for (;;) {
|
||
uint64_t best_tag = 0;
|
||
bool any_reachable = false;
|
||
|
||
for (int i = 0; i < c->pcfg.n_candidates; i++) {
|
||
if (strcmp(c->pcfg.candidates[i].name, c->myconf->name) == 0) continue;
|
||
uint64_t peer_tag = 0;
|
||
int rc = xce_probe_config_tag(c->pcfg.candidates[i].host,
|
||
c->pcfg.candidates[i].port, &peer_tag);
|
||
if (rc != 0) continue; // peer unreachable — skip
|
||
any_reachable = true;
|
||
if (peer_tag > best_tag) best_tag = peer_tag;
|
||
}
|
||
|
||
if (!any_reachable) {
|
||
WARN("No config and no peer reachable — retrying in %dms (kill to abort)",
|
||
retry_ms);
|
||
usleep((useconds_t)retry_ms * 1000);
|
||
retry_ms = MIN(retry_ms * 2, 10000);
|
||
continue;
|
||
}
|
||
|
||
if (best_tag == 0) {
|
||
INFO("All peers also have no config — first-boot cluster, proceeding");
|
||
return; // all-zero cluster: first boot, proceed to election as-is
|
||
}
|
||
|
||
// Config exists among peers — defer fetch until primary is known.
|
||
// The election will prove who has the freshest state.
|
||
INFO("Config exists among peers (best tag=%016" PRIx64 ") — "
|
||
"deferring fetch until primary is known", best_tag);
|
||
c->ha.defer_config_fetch = true;
|
||
return;
|
||
}
|
||
}
|
||
```
|
||
|
||
The function never returns an error. If no peer is reachable, it retries with
|
||
exponential backoff. If all peers are `tag=0`, it returns for first-boot election.
|
||
If config exists among peers, it sets `defer_config_fetch` and returns — the node
|
||
will fetch from the proven winner after the election resolves.
|
||
|
||
`xce_probe_config_tag()` is a lightweight new RPC (single round-trip) that returns
|
||
the peer's current `config_version_tag` without transferring the full config payload.
|
||
Reuses the existing SSL connection setup pattern (`connect_ssl_raw()`).
|
||
|
||
**Cap'n Proto schema for `OP_probeConfigTag`**:
|
||
```c
|
||
struct ProbeConfigTagRequest {
|
||
name @0 :Text; // requester's node name (for logging on peer side)
|
||
}
|
||
struct ProbeConfigTagReply {
|
||
config_tag @0 :UInt64; // peer's current config_version_tag (0 if no config)
|
||
}
|
||
```
|
||
Answered by all nodes in any FSM state (including `CTRL_INIT`) — the dispatch gate
|
||
explicitly allows `OP_probeConfigTag` through before the controller is ready for
|
||
normal requests (Change A). Note: the actual config transfer (`OP_replConfig`) does
|
||
NOT need to be role-agnostic — with deferred fetch, `do_one_config_fetch()` is called
|
||
post-election when the serving peer is already in `CTRL_STANDBY` or `CTRL_PRIMARY`
|
||
and full dispatch is active. Only the probe op needs the `CTRL_INIT` exemption.
|
||
|
||
Integrated into `myapp.c` (before election):
|
||
```c
|
||
bootstrap_config_if_needed(&c); // returns immediately: tag>0, first-boot, or defer
|
||
|
||
// The tag=0 node participates in the election as a voter but never wins
|
||
// against nodes with tag>0 (log_ok gate ensures this). It votes, discovers
|
||
// the winner via voted_for, and fetches from the proven primary.
|
||
rc = am_i_primary_quorum(&c);
|
||
if (rc < 0) return rc;
|
||
|
||
if (c.ha.defer_config_fetch && rc != 1) {
|
||
// Fetch from the proven winner (known via voted_for or heartbeat source).
|
||
// Guard rc != 1: a winning node has tag>0 so defer_config_fetch is never
|
||
// set for it, but the explicit guard prevents double load_primary_config()
|
||
// if logic shifts during implementation.
|
||
do_one_config_fetch(c->ha.primary_host, c->ha.primary_port, 0);
|
||
load_primary_config(&c);
|
||
c->ha.defer_config_fetch = false;
|
||
}
|
||
|
||
if (rc == 1) {
|
||
rc = load_primary_config(&c); // load into memory (already on disk)
|
||
if (rc != 0) return rc;
|
||
atomic_store(&c->ha.role, CTRL_PRIMARY);
|
||
}
|
||
```
|
||
|
||
### Change D — Startup pre-flight checks + ASAN leak fix
|
||
|
||
`main.c`, `src/controller/cluster.c`, `lib/netio.c`, `ha.c`
|
||
|
||
1. **`--vcpus N` CLI parameter** (`main.c`, `src/controller/cluster.c`):
|
||
Add a `--vcpus N` flag parsed in `main.c` alongside the existing `--peers` and `--name`
|
||
flags. Default: `8` when not specified. Values `< 1` are rejected at startup with
|
||
`"ERROR: --vcpus must be ≥ 1"` and exit `-EINVAL`. Remove the
|
||
`vcpus` field from `cluster_config_t` (currently read from the config file). Pass the
|
||
parsed value directly to `vtaskmgr_init()` — this removes `vtaskmgr_init()`'s
|
||
dependency on `cluster_config`, enabling it to run before election (Change A).
|
||
|
||
**Removed from cluster config schema**: the `vcpus:` key in the config file is no
|
||
longer read. Existing configs that include it receive a logged deprecation warning
|
||
(`"WARN: 'vcpus' in config file is deprecated — use --vcpus N instead"`) but the
|
||
controller does not fail.
|
||
|
||
**Rationale**: CPU count is a per-host capacity decision, not a cluster-wide policy.
|
||
Different controller nodes in the same cluster may have different hardware. Keeping
|
||
vcpus in the shared config forces all nodes to use the same value and couples an
|
||
infrastructure tunable to the cluster state config file.
|
||
|
||
2. **Peer count validation in `main.c`** (before `cluster_init_phase1()`):
|
||
The `--peers` list must contain exactly 1 node (single-node) or an **odd number ≥ 3**
|
||
(HA mode). Even numbers cannot form a strict majority and are rejected at startup:
|
||
```c
|
||
int n = pcfg.n_candidates;
|
||
if (n != 1 && n % 2 == 0) {
|
||
fprintf(stderr,
|
||
"ERROR: --peers must list an odd number of controllers (1, 3, 5, ...).\n"
|
||
" %d controllers cannot form a majority. Use 1 for single-node\n"
|
||
" or add/remove one controller to reach an odd count.\n", n);
|
||
return -EINVAL;
|
||
}
|
||
```
|
||
This removes the need for the `n_peers == 2` special case inside
|
||
`am_i_primary_quorum()` — the quorum formula `(N+1)/2` is correct for all
|
||
validated (odd) values of N.
|
||
|
||
3. **Cert pre-flight in `main.c`** (before `myapp_run()`): explicit `access()` check on
|
||
`{workdir}/certs/xce.key` and `{workdir}/certs/xce.crt`. On failure: print
|
||
operator-friendly `"ERROR: TLS cert not found: <path>"` and exit `-ENOENT`.
|
||
Controller never starts without valid certs — same hard-prerequisite model as config.
|
||
|
||
4. **Fix ASAN leak in `ha_fini()` (ha.c:451)**: add `SSL_CTX_free(ha->ssl_ctx)` call.
|
||
This closes the ~5320-byte direct+indirect leak shown in the pc13v1 ASAN output.
|
||
|
||
### Change E — Majority-commit WAL replication
|
||
|
||
`src/controller/wal.c` (or `persist.c`), `src/controller/ha.c`, `src/controller/cluster.c`
|
||
|
||
**Principle**: Primary does not ACK a write to its caller until ≥(N+1)/2 standbys have
|
||
confirmed they wrote the WAL entry to disk. This guarantees any promoted standby has
|
||
all committed entries — zero data loss on failover.
|
||
|
||
**New wire message: OP_walAck** (Cap'n Proto):
|
||
```c
|
||
struct WalAckRequest {
|
||
term @0 :UInt64; // primary's current term (for stale-detection)
|
||
lsn @1 :UInt64; // WAL entry LSN being ACKed
|
||
}
|
||
struct WalAckReply {
|
||
term @0 :UInt64; // standby's term (primary steps down if reply.term > mine)
|
||
lsn @1 :UInt64; // highest LSN standby has durably written
|
||
success @2 :Bool;
|
||
}
|
||
```
|
||
|
||
**Primary-side commit protocol** (`persist.c` / `wal.c`):
|
||
1. Write WAL entry to local disk (existing)
|
||
2. Fan-out `OP_walAck` to all connected standbys in parallel (async send)
|
||
3. Wait for ≥(N+1)/2 − 1 standby ACKs (own write counts as 1) within
|
||
`commit_timeout_ms` (configurable, default 5000ms)
|
||
4. On quorum ACK: mark entry committed, return success to caller
|
||
5. On timeout without quorum: return `-ETIMEDOUT` to caller; entry is
|
||
written locally but not committed — next primary will re-evaluate on
|
||
WAL replay
|
||
|
||
**Standby-side WAL write** (existing replication path extended):
|
||
- On receiving replicated WAL entry: write to disk, fsync, send `OP_walAck` reply
|
||
- No change to standby application logic
|
||
|
||
**Backpressure**: Primary maintains a per-standby `pending_ack_count`. If a standby
|
||
falls more than `max_pending_acks` (default 64) entries behind, it is removed from the
|
||
active replication set and reconnects via existing catch-up path.
|
||
|
||
**Single-node mode**: quorum = 1, commit is immediate (own write = majority). No change.
|
||
|
||
**Clarification 1 — Durability boundary (which entry types block on majority-commit)**:
|
||
|
||
Not all WAL entry types have equal durability requirements. The rule:
|
||
|
||
| Entry type | Commit model | Rationale |
|
||
|---|---|---|
|
||
| `WAL_JOB_SUBMIT` | Majority-commit (blocking) | Job loss on failover is visible to user |
|
||
| `WAL_JOB_START` / `WAL_JOB_FINISH` | Majority-commit (blocking) | State transitions must survive failover |
|
||
| `WAL_QUEUE_CHANGE` | Majority-commit (blocking) | Queue config changes must be durable |
|
||
| `WAL_HOST_STATE` | Best-effort async | Reconstructed from worker reconnect on failover |
|
||
| `WAL_CHECKPOINT_REF` | Best-effort async | Checkpoint is self-contained; ref is advisory |
|
||
|
||
The existing `persist_wal_sync_append()` path (used for `WAL_JOB_SUBMIT` today with
|
||
ANY-standby wait) is extended to use majority quorum. The existing `persist_wal_async()`
|
||
path remains fire-and-forget for best-effort types. No new caller API — entry type
|
||
determines which path is taken inside `persist.c`.
|
||
|
||
**Clarification 2 — `committed_lsn` data model**:
|
||
|
||
Two new fields added to `persist_t` (persist.h):
|
||
```c
|
||
_Atomic uint64_t committed_lsn; // highest LSN majority-ACKed; safe to serve/trim
|
||
uint64_t applied_lsn; // existing: highest LSN written locally (may exceed committed)
|
||
```
|
||
|
||
`committed_lsn` is advanced by `repl_wait_majority_ack()` after quorum is reached.
|
||
On promotion, the new primary replays WAL from its checkpoint up to `committed_lsn`
|
||
only — entries above `committed_lsn` are discarded (never committed to majority, so
|
||
no client received success for them). WAL trim guard updated: trim threshold = `committed_lsn`,
|
||
not `applied_lsn`.
|
||
|
||
Catch-up standbys (in_catch_up == 1) are excluded from quorum counting — they are not
|
||
yet in the active replication set. This means a cluster with 2-of-3 nodes catching up
|
||
cannot make progress until at least one catch-up completes. This is the correct CP
|
||
behavior: prefer unavailability over serving uncommitted state.
|
||
|
||
**Clarification 3 — `repl_broadcast()` prerequisite**:
|
||
|
||
`repl_broadcast()` (repl.c:183–196) is currently a stub. Majority-commit requires a
|
||
real broadcast implementation. This is co-developed with Change E — they are the same
|
||
work unit. Change E scope explicitly includes:
|
||
- Implement `repl_broadcast()` with per-standby TCP send via existing iomgr pipeline
|
||
- Implement `repl_wait_majority_ack()` replacing the existing `repl_wait_ack()` (any→majority)
|
||
- Add per-standby send error callback: failed send → decrement active standby count,
|
||
trigger catch-up reconnect (existing path)
|
||
|
||
If `repl_broadcast()` stub lands separately before Change E, Change E still works —
|
||
it replaces the stub. No ordering dependency beyond Task 11 (config fetch).
|
||
|
||
**New function `persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)`** (`persist.c`):
|
||
|
||
Removes all WAL entries with LSN > `cutoff_lsn` from local storage. Called in two
|
||
contexts (Clarifications 4 and 5). Implementation: scan WAL segment files from newest
|
||
to oldest; truncate the file containing `cutoff_lsn` at the byte offset immediately
|
||
after that entry; delete any segment files that start above `cutoff_lsn`. Updates
|
||
`p->applied_lsn = cutoff_lsn` atomically after truncation. Must be idempotent (calling
|
||
twice with same cutoff is safe). New function — does not currently exist in `persist.c`.
|
||
|
||
**Clarification 4 — `committed_lsn` must be piggybacked on heartbeats**:
|
||
|
||
A promoted standby needs to know where to truncate its WAL. Without this it only has
|
||
`applied_lsn`, which may include uncommitted orphan entries written by the old primary
|
||
but never majority-ACKed.
|
||
|
||
- Extend `CtrlHeartbeat` Cap'n Proto struct with `committed_lsn @N :UInt64`
|
||
- Standbys store it in `ha.last_known_committed_lsn` (`_Atomic uint64_t`, init 0) on
|
||
each heartbeat receipt
|
||
- On promotion, immediately after `am_i_primary_quorum()` returns 1:
|
||
```c
|
||
persist_truncate_wal_above(&c->persist, c->ha.last_known_committed_lsn);
|
||
c->persist.committed_lsn = c->ha.last_known_committed_lsn;
|
||
```
|
||
- First-boot failover (no heartbeat ever received): `last_known_committed_lsn = 0`,
|
||
so WAL replay starts from checkpoint only — correct and safe
|
||
|
||
**Clarification 5 — Log divergence on standby re-join**:
|
||
|
||
The existing catch-up path assumes the standby is simply *behind* the primary (a clean
|
||
subset). After a primary crash, a standby may have orphaned WAL entries above the new
|
||
primary's `committed_lsn` — entries the old primary replicated but never majority-ACKed.
|
||
Replaying those entries would corrupt the standby's state.
|
||
|
||
Catch-up handshake extension (Change E scope):
|
||
1. Primary sends its `committed_lsn` in the catch-up initiation reply
|
||
2. Standby calls `persist_truncate_wal_above(&c->persist, primary_committed_lsn)` before
|
||
requesting WAL replay — discards any orphaned entries above that boundary
|
||
3. Standby sets `applied_lsn = primary_committed_lsn` as the new replay base
|
||
4. Normal WAL catch-up (existing mechanism) proceeds from there
|
||
|
||
**Clarification 6 — Catch-up standbys excluded from voting (not just quorum count)**:
|
||
|
||
Clarification 2 excludes `in_catch_up==1` standbys from the majority quorum count.
|
||
They must also be prevented from *granting* votes — a behind-node's stale LSN could
|
||
tip an election to a less-fresh candidate.
|
||
|
||
In `handle_request_vote()`, before the `log_ok` freshness check:
|
||
```c
|
||
if (c->ha.in_catch_up) {
|
||
reply->vote_granted = 0;
|
||
// still populate peer_config_tag/peer_last_log_lsn — candidate needs the full picture
|
||
goto fill_reply;
|
||
}
|
||
```
|
||
|
||
**Clarification 7 — `applied_lsn` in vote request must be post-fsync**:
|
||
|
||
`RequestVoteRequest.last_log_lsn` is sourced from `atomic_load(&c->persist.applied_lsn)`.
|
||
This is correct only if `applied_lsn` is advanced *after* `fsync()` completes in
|
||
`persist.c`. A node that claims an LSN it has not durably written could win an election
|
||
and then fail to replay entries it advertised. Verify in `persist.c` before implementing
|
||
Change B — if `applied_lsn` is advanced pre-fsync anywhere, fix that first.
|
||
|
||
### Change F — Persistent standby↔primary control channel
|
||
|
||
**Files**: `src/controller/ha.c`, `src/controller/myapp.c`, `src/controller/cluster.c`,
|
||
`src/controller/repl.c`
|
||
|
||
**Background**: Standbys currently detect primary failure through `ha_check_heartbeat_nocluster()`
|
||
(ha.c:380–405), which polls a `last_heartbeat` timestamp with a 10s floor. Failover detection
|
||
latency is up to 10 seconds. Additionally, each config fetch and WAL replication message opens a
|
||
new TCP+SSL connection (`connect_ssl_raw()`, config_repl.c:685–775), adding per-message handshake
|
||
latency for WAL streaming (Change E).
|
||
|
||
**Changes**:
|
||
|
||
1. **Standby discovers and connects to primary** — new `ha_find_and_connect_to_primary(cluster_t *c)`
|
||
(called in `myapp.c` standby path after election). Opens a TCP+SSL connection to the primary
|
||
peer and registers the fd with epoll `EPOLLIN | EPOLLRDHUP`. The connection is stored in
|
||
`ha_state_t` as `ha->primary_ctrl_fd`. On connect, the standby sends an extended `OP_ctrlHello`
|
||
(item 5) carrying identity and state. The primary registers the standby, initiates catch-up,
|
||
and begins streaming any missing WAL before the standby enters the active replication set.
|
||
|
||
**Primary discovery algorithm** (how the standby finds the winner's address):
|
||
|
||
Two startup paths lead here with different amounts of prior knowledge:
|
||
|
||
- **Heartbeat path** (`CTRL_INIT → CTRL_STANDBY`): `ha_wait_for_heartbeat_or_timeout()` received
|
||
a heartbeat/`OP_ctrlHello` from an existing primary. The sender's host/port is in the packet.
|
||
Store immediately as `ha.primary_host/port`. No further discovery needed — connect directly.
|
||
|
||
- **Election path** (`CTRL_ELECTING → CTRL_STANDBY`): lost `am_i_primary_quorum()`. Discovery
|
||
uses a three-tier strategy exploiting Raft's structural property that the majority voted for
|
||
the winner:
|
||
|
||
```
|
||
Tier 1 (fast path — O(1) for ≥(N+1)/2 nodes):
|
||
If ha.voted_for_name is set:
|
||
try ha_connect_to_primary(voted_for peer)
|
||
on success: store ha.primary_host/port, done
|
||
on XCE_ERR_NOT_READY: peer still electing — backoff, retry tier 1
|
||
|
||
Tier 2 (redirect — O(1) once any majority member connected):
|
||
Probe remaining peers (random order) with OP_ctrlHello
|
||
A CTRL_STANDBY peer that already knows the primary responds with
|
||
XCE_ERR_STANDBY + redirect_host/port (same field as Change G)
|
||
Follow redirect immediately — connect to that address
|
||
|
||
Tier 3 (blind scan fallback — for split-vote minorities):
|
||
If redirect not received: try remaining peers sequentially
|
||
A CTRL_PRIMARY peer accepts OP_ctrlHello normally → connected
|
||
A CTRL_INIT/CTRL_ELECTING peer returns XCE_ERR_NOT_READY → skip+retry
|
||
Repeat with 100ms → 5s exponential backoff until primary found
|
||
```
|
||
|
||
**Why this converges quickly**: In a 3-node cluster, at least 2 nodes voted for the winner.
|
||
Those 2 connect via Tier 1 immediately. The 3rd probes either of them (Tier 2), gets a
|
||
redirect, and connects. Total: 2 direct + 1 redirect = all standbys connected with at most
|
||
one extra probe per minority node. In a 5-node cluster, at least 3 connect via Tier 1; the
|
||
remaining 2 get redirected.
|
||
|
||
**`ha.primary_host/port` update**: set on successful connect; cleared to empty on
|
||
`EPOLLRDHUP` (primary disconnect). During the election window it is empty; standbys receiving
|
||
`OP_ctrlHello` while `ha.primary_host` is empty reply with `XCE_ERR_STANDBY` and no redirect
|
||
(graceful — probing node falls through to Tier 3).
|
||
|
||
**No new message types**: the redirect uses the existing `XCE_ERR_STANDBY` error reply fields
|
||
already extended in Change G (`redirect_host`, `redirect_port`). The standby dispatch gate
|
||
(already returning `XCE_ERR_STANDBY` for primary-only ops) populates these fields from
|
||
`ha.primary_host/port` when they are known.
|
||
|
||
2. **Standby join synchronization — full state sync on connect** (cluster.c, ha.c, repl.c):
|
||
|
||
A standby with stale WAL cannot safely promote — its in-memory state (jobs, queues,
|
||
resources) would be out of date, and promoted writes would replay entries clients already
|
||
saw as committed. Every standby must be fully current before it enters the quorum.
|
||
|
||
**Extended `OP_ctrlHello` Cap'n Proto schema**:
|
||
```c
|
||
struct CtrlHelloRequest {
|
||
name @0 :Text; // peer name (from --peers)
|
||
key @1 :Data; // cluster auth key: SHA-256(cluster_name + shared_secret)
|
||
// shared_secret read from myconf->cluster_key (new config field)
|
||
// primary rejects connections where HMAC does not match its own
|
||
applied_lsn @2 :UInt64; // standby's current applied_lsn (0 if fresh node)
|
||
config_tag @3 :UInt64; // standby's current config_version_tag (0 if none)
|
||
}
|
||
struct CtrlHelloReply {
|
||
committed_lsn @0 :UInt64; // primary's current committed_lsn
|
||
needs_snapshot @1 :Bool; // true if standby's WAL predates oldest primary segment
|
||
config_tag @2 :UInt64; // primary's current config_version_tag
|
||
}
|
||
```
|
||
|
||
**New wire messages for sync**:
|
||
```c
|
||
struct SnapshotTransfer {
|
||
snapshot_lsn @0 :UInt64; // LSN of this snapshot (= checkpoint_lsn after fresh checkpoint)
|
||
payload @1 :Data; // serialized snapshot blob (from persist_checkpoint())
|
||
}
|
||
struct WalCatchupEntry {
|
||
lsn @0 :UInt64; // WAL entry LSN
|
||
type @1 :UInt8; // WAL entry type (WAL_JOB_SUBMIT etc.)
|
||
payload @2 :Data; // WAL entry payload
|
||
}
|
||
```
|
||
`OP_snapshotTransfer` and `OP_walCatchupEntry` flow on the persistent control channel fd —
|
||
no new connections.
|
||
|
||
**Primary-side `handle_ctrlhello()` sequence** (new handler in cluster.c):
|
||
```
|
||
1. Extract name, applied_lsn, config_tag from CtrlHelloRequest
|
||
2. repl_add_standby(&c->repl, ep, host, port)
|
||
3. needs_snap = repl_needs_snapshot(&c->repl, applied_lsn, oldest_wal_lsn())
|
||
4. Reply: CtrlHelloReply{c->persist.committed_lsn, needs_snap, myconf->config_version_tag}
|
||
5. repl_start_catchup(&c->repl, ep, applied_lsn) // sets standby's in_catch_up=1
|
||
6. If needs_snapshot:
|
||
persist_checkpoint(&c->persist) // flush in-memory state to disk
|
||
send OP_snapshotTransfer(snapshot_lsn, blob)
|
||
applied_lsn = snapshot_lsn // advance standby's effective base
|
||
7. Streaming loop — runs until standby is fully current:
|
||
target_lsn = atomic_load(&c->persist.committed_lsn) // read current live value
|
||
stream WAL entries [applied_lsn+1 → target_lsn] via OP_walCatchupEntry
|
||
wait for OP_walAck with acked_lsn >= target_lsn
|
||
if atomic_load(&c->persist.committed_lsn) > target_lsn:
|
||
applied_lsn = target_lsn; continue loop // new entries committed, catch up again
|
||
// standby is now at current committed_lsn — exit loop
|
||
8. repl_complete_catchup(&c->repl, ep) // in_catch_up=0; standby enters live replication
|
||
9. If standby config_tag != primary config_tag:
|
||
push_config_to_standbys(c, blob, blen, tag)
|
||
```
|
||
|
||
**`repl_broadcast()` skips `in_catch_up==1` standbys**: while the streaming loop (step 7)
|
||
is in progress, `repl_broadcast()` does not send to this standby. The loop itself handles
|
||
delivery of all committed entries — including entries committed after the handshake — by
|
||
re-reading `committed_lsn` on each iteration. The loop exits only when the standby's
|
||
`acked_lsn` equals the *current* `committed_lsn`, at which point `repl_complete_catchup()`
|
||
fires and live broadcast takes over. There is no window where entries can be missed.
|
||
|
||
**Standby-side sequence** (after receiving CtrlHelloReply in ha.c / myapp.c):
|
||
```
|
||
1. Store primary_committed_lsn = reply.committed_lsn
|
||
2. persist_truncate_wal_above(&c->persist, primary_committed_lsn) // Clarification 5
|
||
3. ha.last_known_committed_lsn = primary_committed_lsn // Clarification 4
|
||
4. ha.in_catch_up = 1 // deny votes during sync
|
||
5. If reply.needs_snapshot:
|
||
wait for OP_snapshotTransfer
|
||
write received blob as local snapshot file
|
||
load_snapshot(&c->persist, path, snapshot_lsn)
|
||
applied_lsn = snapshot.snapshot_lsn
|
||
6. Receive OP_walCatchupEntry stream; call apply_wal_entry() for each
|
||
7. Send OP_walAck after each entry (or configurable batch):
|
||
OP_walAck{acked_lsn = last applied LSN}
|
||
8. When primary sends no more OP_walCatchupEntry (catch-up loop ended on primary side):
|
||
ha.in_catch_up = 0 // standby now in live replication; votes granted normally
|
||
```
|
||
|
||
`apply_wal_entry()` is the same WAL replay function used during startup recovery — it
|
||
applies a single WAL entry to the in-memory state (jobs, queues, resources). Must exist
|
||
or be extracted from the startup replay path before this change lands.
|
||
|
||
**`repl_needs_snapshot()` already exists** in repl.c — compares `standby_applied_lsn`
|
||
against `oldest_wal_lsn` (the trim floor). If standby is behind the trim floor,
|
||
`persist_checkpoint()` is called on the primary to produce a fresh snapshot, which is
|
||
then streamed in full. This is the only path where `persist_checkpoint()` is called
|
||
on-demand rather than on schedule.
|
||
|
||
**`push_config_to_standbys()` is declared** in config_repl.h; config sync at join
|
||
handles the case where a standby was offline during a config upgrade.
|
||
|
||
2. **Heartbeat runs on control channel** — primary sends periodic heartbeats on the controller↔controller
|
||
channel (in addition to existing worker heartbeats). Standby updates `last_heartbeat` on receipt,
|
||
but the epoll path is the fast lane.
|
||
|
||
3. **Replace timestamp polling with dual-trigger failover detection** —
|
||
`ha_check_heartbeat_nocluster()` (ha.c:380–405) is retired. Two complementary paths:
|
||
- **Fast path (`EPOLLRDHUP`)**: TCP disconnect detected immediately; `primary_ctrl_fd`
|
||
closed and nulled before `am_i_primary_quorum()` is called. Sub-second latency.
|
||
- **Fallback path (heartbeat timeout vtask)**: armed after each heartbeat is received
|
||
(`vtaskmgr_schedule(election_timeout + rand_jitter, ha_heartbeat_timeout_cb)`); reset
|
||
on every subsequent heartbeat. Fires when primary is silent but TCP is alive — covers
|
||
network partitions that keep TCP open, primary overload, or missed heartbeat bursts.
|
||
On fire: close+null `primary_ctrl_fd`, set `CTRL_ELECTING`, call `am_i_primary_quorum()`.
|
||
|
||
4. **WAL streaming over persistent channel (Change E complement)** — `repl_broadcast()`
|
||
(repl.c:183–196) sends WAL entries on the persistent control channels established by each
|
||
standby rather than opening per-entry connections. No new connection setup per WAL write.
|
||
|
||
**Role flip on promotion**: When a standby wins election and becomes primary, it must
|
||
close and discard its `primary_ctrl_fd` (the old connection to the now-dead primary)
|
||
*before* calling `am_i_primary_quorum()` triggers the role transition. After promotion,
|
||
the new primary accepts `OP_ctrlHello` connections from the remaining standbys — the
|
||
role of `primary_ctrl_fd` inverts from "my connection out" to "their connections in",
|
||
tracked in a per-standby fd table rather than a single field.
|
||
|
||
**Reconnect behavior**: On primary_ctrl_fd disconnect, standby closes and nulls
|
||
`primary_ctrl_fd` first, then calls `am_i_primary_quorum()`.
|
||
If it becomes primary, the old primary_ctrl_fd is discarded. If it remains standby, it reconnects
|
||
with exponential backoff (100ms, 200ms, 400ms, max 5s) until the new primary is reachable.
|
||
|
||
**Single-node mode**: `ha_connect_to_primary()` is a no-op when `n_peers == 1`.
|
||
|
||
**Key properties**:
|
||
- Sub-second failover detection (`EPOLLRDHUP` vs 10s timeout)
|
||
- No per-entry TCP+SSL handshake overhead for WAL replication
|
||
- Single liveness signal: connection alive = peer alive
|
||
- Backward-compatible: standby reconnect on disconnect triggers election (same semantics as today,
|
||
faster latency)
|
||
|
||
### Change G — libxce client-side retry for transient controller errors
|
||
|
||
**Files**: `src/lib/` (libxce — client request/connection layer)
|
||
|
||
**Background**: With `CTRL_ELECTING` gating requests during the election window
|
||
(Change A), and standbys already returning `XCE_ERR_STANDBY` to misdirected
|
||
requests, every daemon (worker, scheduler) needs to handle these transient errors
|
||
gracefully. Without built-in retry in libxce, each daemon would need its own
|
||
ad-hoc loop — fragile and inconsistent.
|
||
|
||
**Changes**:
|
||
|
||
1. **Retry policy in libxce's client request path** (the function wrapping
|
||
`xce_encode()` + send + recv + `xce_decode()`):
|
||
- On `XCE_ERR_NOT_READY`: back off and retry against same endpoint
|
||
- On `XCE_ERR_STANDBY`: **redirect** — the standby includes the current primary's
|
||
`host:port` in the error payload when known (`ha.primary_host/port`). libxce
|
||
reconnects directly to the redirected endpoint instead of blind backoff.
|
||
When primary is unknown (e.g. during startup), falls back to normal retry.
|
||
- On TCP-level `ECONNREFUSED` or `ETIMEDOUT`: reconnect and retry
|
||
(handles the window before `cluster_open_listener()` runs on a starting peer)
|
||
- Non-retriable errors (`XCE_ERR_AUTH`, `XCE_ERR_INVALID`, `XCE_ERR_NOENT`,
|
||
`XCE_ERR_PERM`, etc.): surface immediately to caller
|
||
|
||
**`XCE_ERR_STANDBY` wire extension**: add optional `redirect_host` (Text) and
|
||
`redirect_port` (UInt16) fields to the error reply Cap'n Proto struct. Standbys
|
||
populate these when `ha.primary_host` is non-empty; clients that don't understand
|
||
the new fields simply retry with backoff (backward-compatible).
|
||
|
||
2. **Retry configuration** (sensible defaults, overridable per client):
|
||
```c
|
||
typedef struct {
|
||
int max_retries; // default 20
|
||
int initial_retry_ms; // default 50
|
||
int max_retry_ms; // default 5000
|
||
double backoff_multiplier; // default 2.0 (exponential)
|
||
} xce_retry_config_t;
|
||
```
|
||
Defaults cover a full election round (up to 20 × 50ms–5000ms) without
|
||
requiring callers to configure anything. Daemons that need tighter or looser
|
||
bounds pass a custom `xce_retry_config_t`.
|
||
|
||
3. **Transparent to callers**: existing daemon code makes one `xce_request()` call;
|
||
libxce handles retry internally. No changes needed in worker, scheduler, or
|
||
apiserver code.
|
||
|
||
**Key properties**:
|
||
- Decouples daemon startup order — a worker that starts before the primary is ready
|
||
simply waits for the retry window to expire rather than crashing or looping manually
|
||
- `XCE_ERR_STANDBY` retry enables transparent primary failover: after failover, the
|
||
next request to the old primary (now standby) gets `XCE_ERR_STANDBY`, libxce retries
|
||
against the same host/port and the new primary responds
|
||
- Backoff prevents thundering-herd when 50 workers restart simultaneously after a failover
|
||
|
||
### Change H — Majority-commit config push
|
||
|
||
`src/controller/config_repl.c`, `src/controller/repl.c`
|
||
|
||
**Principle**: Config upgrades (`OP_upgradeConfig`) use the same majority-commit
|
||
infrastructure as WAL replication (Change E). The primary does not ACK a config
|
||
change to the operator until ≥(N+1)/2 standbys have confirmed durable write.
|
||
|
||
**Why this matters**: Without majority-commit, a primary could accept a config change,
|
||
ACK to the operator, then crash. The new primary serves the old config, and the
|
||
operator's change is silently lost. For strict CP, config writes must have the same
|
||
durability guarantee as WAL writes — even though they are orders of magnitude rarer.
|
||
|
||
**Changes**:
|
||
|
||
1. **Reuse Change E infrastructure** — `repl_broadcast()` (sends to all connected
|
||
standbys on persistent channels) and `repl_wait_majority_ack()` (blocks until
|
||
≥(N+1)/2 ACKs or `commit_timeout_ms`) already built for WAL. Both functions
|
||
accept a message-type parameter so they can serve WAL (`OP_walAck`) and config
|
||
(`OP_upgradeConfigAck`) ACKs without duplication. Zero new infrastructure.
|
||
|
||
2. **`push_config_to_standbys()` extended** (config_repl.c):
|
||
```c
|
||
int push_config_to_standbys(cluster_t *c, const char *blob, size_t len, uint64_t tag) {
|
||
// 1. Write locally (existing atomic write+rename — unchanged)
|
||
int rc = config_repl_write_local(c, blob, len, tag);
|
||
if (rc != 0) return rc;
|
||
|
||
// 2. Fan-out OP_upgradeConfig via repl_broadcast (same channel as WAL)
|
||
rc = repl_broadcast(c, OP_upgradeConfig, blob, len);
|
||
if (rc != 0) return rc;
|
||
|
||
// 3. Wait for ≥(N+1)/2 standby ACKs (own write = 1 vote)
|
||
rc = repl_wait_majority_ack(c, c->ha.config.commit_timeout_ms);
|
||
|
||
// 4. On quorum: advance committed_config_tag and return success
|
||
if (rc == 0) {
|
||
atomic_store(&c->myconf->committed_config_tag, tag);
|
||
}
|
||
return rc; // -ETIMEDOUT on failure — operator retries
|
||
}
|
||
```
|
||
|
||
3. **Standby-side config ACK** — on receiving `OP_upgradeConfig`, standby:
|
||
- Writes config blob to disk (existing atomic write path)
|
||
- Loads config into memory
|
||
- Sends `OP_upgradeConfigAck` with `acked_tag = written_tag`
|
||
- ACK semantics identical to `OP_walAck` — durable write complete
|
||
|
||
4. **`OP_upgradeConfigAck` Cap'n Proto**:
|
||
```c
|
||
struct UpgradeConfigAck {
|
||
term @0 :UInt64; // standby's term (primary steps down if ack.term > mine)
|
||
acked_tag @1 :UInt64; // config tag standby has durably written
|
||
success @2 :Bool;
|
||
}
|
||
```
|
||
|
||
5. **`committed_config_tag` field** added to cluster config state (analogous to
|
||
`committed_lsn` for WAL). Advanced only on majority ACK. Stored alongside
|
||
`config_version_tag` in durable state. On promotion, a new primary serves only
|
||
configs up to `committed_config_tag`. **Piggybacked on heartbeats** (same as
|
||
`committed_lsn` — Clarification 4): standbys store the latest value in
|
||
`ha.last_known_committed_config_tag` so a promoted standby knows the safe
|
||
boundary without an extra round-trip. First-boot promotion (no heartbeat
|
||
received): value is 0, standby serves whatever config it loaded from disk —
|
||
correct for first-boot where no upgrade has been committed.
|
||
|
||
**Key properties**:
|
||
- Same `commit_timeout_ms` as WAL (configurable, default 5000ms)
|
||
- Single-node mode: quorum=1, commit is immediate (own write = majority)
|
||
- Config upgrades are operator-initiated and rare — the blocking ACK wait adds
|
||
negligible overhead to cluster operations
|
||
- Catch-up standbys (`in_catch_up==1`) are excluded from quorum count,
|
||
consistent with WAL majority rules
|
||
- If commit times out, operator gets `-ETIMEDOUT` and can retry — libxce retry
|
||
(Change G) handles this transparently for the API server
|
||
|
||
## Scope
|
||
|
||
- **In**:
|
||
- Change A: Extract `cluster_open_listener()`, `iomgr_init()`, and `vtaskmgr_init()`
|
||
from phase2; call all three before election in myapp.c so the full connection
|
||
framework is running during election (incoming `OP_requestVote` handled by normal
|
||
dispatch switch); add `CTRL_ELECTING` role + pre-dispatch gate returning
|
||
`XCE_ERR_NOT_READY` for non-vote requests during election window
|
||
- Change B: Full Raft-style quorum voting in `am_i_primary_quorum()`
|
||
- New `OP_requestVote` wire protocol (Cap'n Proto request + reply)
|
||
- `handle_request_vote()` handler on all peers with `voted_for` enforcement
|
||
- Config tag + LSN freshness comparison in vote grant and candidate yield gate
|
||
- Hard gate A (fatal): refuse primary if config_tag=0 and any peer has tag>0
|
||
- Hard gate B (yield): winner yields if any *responding* peer has fresher data
|
||
- Split vote retry with jitter (one retry after timeout)
|
||
- Term + `voted_for` persistence to `ha-term.bin` (survives restart)
|
||
- New `ha_state_t` fields: `voted_for_term`, `voted_for_name`, `primary_ctrl_fd`
|
||
- Retire `ha_check_priority_stepdown()` (permanently unreachable after first election)
|
||
- Change C: `bootstrap_config_if_needed()` pre-election bootstrap — probes peers
|
||
for config tags, defers fetch until primary is known (avoids speculative download
|
||
from a peer that may not win the election); tag=0 node participates as voter only;
|
||
new `xce_probe_config_tag()` lightweight RPC; role-agnostic
|
||
`handle_config_request()` so peers can serve config to a bootstrapping node
|
||
- Change D: `--vcpus N` CLI parameter + remove `vcpus` from cluster config (deprecation
|
||
warning for existing configs); odd-N peer-count validator at startup (`-EINVAL` on
|
||
even count); cert pre-flight check in `main.c`; `SSL_CTX_free` ASAN fix in `ha.c`
|
||
- Change E: Majority-commit WAL replication — `OP_walAck` protocol, primary
|
||
commit gating, standby ACK path, backpressure, `commit_timeout_ms` config
|
||
- Change F: Persistent standby↔primary control channel — `ha_connect_to_primary()`,
|
||
`EPOLLRDHUP`-based failover detection, heartbeat on control channel, WAL streaming
|
||
over persistent connection
|
||
- Change G: libxce client-side retry — exponential backoff on `XCE_ERR_NOT_READY`,
|
||
`XCE_ERR_STANDBY`, `ECONNREFUSED`, `ETIMEDOUT`; transparent to daemon callers;
|
||
configurable via `xce_retry_config_t` with safe defaults
|
||
- Change H: Majority-commit config push — `OP_upgradeConfig` uses the same
|
||
`repl_broadcast()` + `repl_wait_majority_ack()` infrastructure as WAL replication
|
||
(Change E). Primary waits for ≥(N+1)/2 standby ACKs before confirming a config
|
||
upgrade to the operator. Reuses `commit_timeout_ms`. Zero new infrastructure cost.
|
||
- Unit and integration tests for all eight changes
|
||
- **Out**:
|
||
- Raft snapshot protocol — existing dual-zarray COW is orthogonal and unchanged
|
||
- Joint consensus for cluster membership changes — candidate list is static
|
||
- do_one_config_fetch() stub wiring (Task 11) — prerequisite; must land first or co-developed
|
||
- mTLS between controllers — trusted-LAN model with server-side TLS is sufficient
|
||
|
||
## Boundary Candidates
|
||
|
||
1. **Election protocol** — full replacement of `am_i_primary()` with quorum voting
|
||
2. **Config wire protocol** — new OP_requestVote, OP_probeConfigTag, OP_upgradeConfigAck; existing OP_replConfig used for post-election deferred fetch
|
||
3. **Primary startup path** — bootstrap integration in myapp.c
|
||
4. **Listener lifecycle** — extracted from phase2 to phase1
|
||
|
||
## Out of Boundary
|
||
|
||
- Standby fetch path (`fetch_config_from_primary()`) — untouched
|
||
- Config format, versioning, or storage layout — untouched
|
||
- Raft snapshot protocol — existing dual-zarray COW is orthogonal
|
||
- Joint consensus for cluster membership changes — candidate list is static
|
||
- mTLS — trusted-LAN model with server-side TLS is sufficient
|
||
|
||
## Upstream / Downstream
|
||
|
||
- **Upstream**: Task 11 (`do_one_config_fetch()` stub) on persistent_ha branch — with deferred fetch (Change C), the probe-only path in `bootstrap_config_if_needed()` does not call `do_one_config_fetch()`. Task 11 is hit only at the post-election deferred fetch step in `myapp.c`. If Task 11 is not yet landed, the deferred fetch returns `-ENOSYS` and the bootstrapping standby fails to load config — a fatal but clearly logged error. Both can be co-developed; Task 11 must land before or with Change C for a functional cluster.
|
||
- **Downstream**: Phase 3 hardening (membership changes, leader transfer), ops tooling for election/commit observability, monitoring metrics for WAL replication lag.
|
||
|
||
## Existing Spec Touchpoints
|
||
|
||
- **Adjacent**: none of the active specs (resval-id-decoupling, scheduler-controller-msg-gap, worker-join-refactor) touch config replication, HA election, or cluster init.
|
||
|
||
## Constraints
|
||
|
||
- C (gnu17), no C++, no new external dependencies.
|
||
- ASAN build must pass (`nix build .#xcecore_debug`).
|
||
- **`--peers` must list 1 or an odd number ≥ 3 controllers.** Even counts are rejected
|
||
at startup with `-EINVAL`. This simplifies quorum math: `(N+1)/2` is exact for all
|
||
valid N; no special-case branches needed inside `am_i_primary_quorum()`.
|
||
- **Static membership**: the candidate list is fixed at startup from `--peers` and never
|
||
changes while the cluster is running. No dynamic join/leave of controller peers.
|
||
Worker and scheduler nodes may still join dynamically — only controller peer membership
|
||
is static. This eliminates the need for joint consensus (Raft §6) and locks the quorum
|
||
denominator for the lifetime of the cluster.
|
||
- Single-node mode unaffected: quorum=1 everywhere (election, WAL commit = own write).
|
||
- TDD required: failing test first, then implementation.
|
||
- All new C test cases go in `src/controller/testsuite/` before implementation.
|
||
- `make test` must pass without a live server.
|
||
- `commit_timeout_ms` must be configurable (default 5000ms) to allow tuning per deployment latency.
|
||
- **WAL write stall during standby catch-up (operational constraint)**: In a 3-node cluster,
|
||
quorum for WAL commit is 2 (primary + 1 standby). Between primary winning election and the
|
||
first standby completing `repl_complete_catchup()`, the active replication set has 0 standbys.
|
||
Every `persist_wal_sync_append()` call blocks for `commit_timeout_ms` and returns `-ETIMEDOUT`.
|
||
This is CP-correct behavior — the cluster is unavailable until quorum is formed. Operators
|
||
should expect a stall of up to `commit_timeout_ms` per write during the initial catch-up
|
||
window on first boot or after a restart. Jobs submitted during this window receive a retriable
|
||
error; libxce retry (Change G) handles this transparently for workers and schedulers.
|
||
|
||
## Deployment Reference
|
||
|
||
### Controller node topology
|
||
|
||
| Cluster size | Failure tolerance | WAL ACKs needed | Recommendation |
|
||
|---|---|---|---|
|
||
| 1 node | 0 | self only | dev/test only |
|
||
| 3 nodes | 1 failure | 1 standby ACK | minimum for production |
|
||
| 5 nodes | 2 failures | 2 standby ACKs | recommended for critical HPC |
|
||
| 7 nodes | 3 failures | 3 standby ACKs | rarely justified; adds latency |
|
||
|
||
### Job submission throughput (Raft overhead only)
|
||
|
||
Basis: LAN environment, 5 controller nodes, quorum = 3 (primary + 2 standby ACKs).
|
||
|
||
**Critical path per `WAL_JOB_SUBMIT`:**
|
||
|
||
```
|
||
client → primary: RTT/2 + primary WAL fsync
|
||
primary → 2 standbys: RTT/2 + standby WAL fsync (parallel, overlaps with above)
|
||
2nd ACK → primary: RTT/2
|
||
primary confirms: RTT/2
|
||
─────────────────────────────────────────────────
|
||
Total: 1 LAN RTT + max(fsync_primary, fsync_standby)
|
||
```
|
||
|
||
| Parameter | Optimistic | Conservative |
|
||
|---|---|---|
|
||
| LAN RTT (controller↔controller) | 0.1 ms | 0.5 ms |
|
||
| NVMe fsync latency | 0.05 ms | 0.2 ms |
|
||
| Effective commit latency | ~0.2 ms | ~0.7 ms |
|
||
|
||
**Throughput estimates:**
|
||
|
||
| Submit model | Commit latency | Jobs/sec |
|
||
|---|---|---|
|
||
| Serial single stream | 0.3–1 ms | 1,000–3,300 |
|
||
| 10 concurrent clients | 0.5 ms | ~20,000 |
|
||
| 100 concurrent clients | 0.5 ms | ~200,000 (non-Raft bottlenecks dominate before this) |
|
||
|
||
**Practical HPC ceiling**: 5,000–20,000 jobs/sec before jobmgr queue locking and
|
||
in-memory state mutations become the constraint.
|
||
|
||
**5-node vs 3-node write latency**: negligible. Majority-commit waits for the
|
||
**2nd-fastest** ACK out of 4 standbys (5-node) vs the **1st** out of 2 (3-node).
|
||
Under uniform LAN latency the difference is sub-millisecond. The extra two nodes
|
||
cost nothing in write throughput but double the failure tolerance.
|
||
|
||
**When Raft becomes a bottleneck**: automated pipelines submitting >10,000 short
|
||
jobs/sec. Mitigation: batch multiple job submits into a single WAL entry at the
|
||
API layer before this design makes that necessary.
|