539 lines
34 KiB
Markdown
539 lines
34 KiB
Markdown
# Design Document: election-iomgr-refactor
|
||
|
||
## Overview
|
||
|
||
The XCE controller was originally fully event-driven: `cluster_init` → `cluster_start` →
|
||
`cluster_run` (shutdown-check loop). The HA additions broke this by inserting blocking calls
|
||
before `cluster_run`: `ha_wait_for_heartbeat_or_timeout` and `am_i_primary_quorum` block the
|
||
main thread, causing SSL context confusion, capnp UAF, and mixed-I/O races.
|
||
|
||
This design restores full event-driven operation with no blocking calls after phase1.
|
||
`myapp_run` does setup then immediately enters `cluster_run`. All election I/O flows through
|
||
iomgr callbacks. When the election reaches a terminal state, `ha_elect_evaluate_and_signal`
|
||
calls `cluster_init_phase2` inline from vtask context (phase2 is already thin — managers were
|
||
moved to phase1 in a prior refactor), then atomically sets `ha->role`. The dispatch gate
|
||
(`ha_dispatch_gate`) opens automatically: the next client request sees PRIMARY or STANDBY and
|
||
is handled normally. No semaphores, no wbuf signals, no changes to `cluster_run`.
|
||
|
||
### Goals
|
||
- Restore event-driven model: `myapp_run` → `cluster_run` with zero blocking between them
|
||
- All election I/O through one epoll event loop (iomgr)
|
||
- `ha_elect_evaluate_and_signal` calls thin `cluster_init_phase2` from vtask context; sets role atomically
|
||
- `cluster_run` unchanged — still just the shutdown-check loop
|
||
- `myreq_handler` / `ha_dispatch_gate` is the sole FSM dispatcher — no new state machinery needed
|
||
- Role-based vote gate: PRIMARY and STANDBY-with-live-primary deny votes
|
||
|
||
### Non-Goals
|
||
- Changes to `cluster_run` or `wbuf` signal values
|
||
- Incoming vote RPC handling (already correct in cluster.c)
|
||
- Heartbeat watchdog thread (existing, unchanged)
|
||
- Config replication, RDHUP paths (unchanged)
|
||
- Async DNS resolution (deferred)
|
||
- TLS certificate verification for vote connections
|
||
|
||
## Boundary Commitments
|
||
|
||
### This Spec Owns
|
||
- `vote_request_thread` + `am_i_primary_quorum` blocking loop → replaced by `ha_start_election_async()`
|
||
- `on_connected` callback added to `netio.c`
|
||
- `client_ssl_ctx` in `ha_state_t` + `myapp_run`
|
||
- `vote_reply_msg_handler` for iomgr-native reply handling (Note: this handler was later generalized into `peer_ctrl_msg_handler`, a single state-aware dispatcher that also routes repl acks and PreVote replies — superseded by ha-prevote, see that spec's design.md)
|
||
- Election timeout watchdog thread
|
||
- `ha_elect_evaluate_and_signal` extended to call `cluster_init_phase2` then set role
|
||
- `handle_request_vote` extended with role-based deny gate (Note: `handle_request_vote` is now `#ifdef XCE_TESTING`-only; production vote dispatch is `cluster.c`'s `case OPCode_requestVote:` calling `raft_on_vote_request()` — superseded by raft-adapter-conformance, see that spec's design.md)
|
||
- `myapp_run` restructured: phase1 + SSL + `ha_start_election_async` + `cluster_run` directly
|
||
|
||
### Out of Boundary
|
||
- `cluster_run` internals (unchanged)
|
||
- `case OPCode_requestVote:` in cluster.c (already correct, no changes)
|
||
- `ha_wait_for_heartbeat_or_timeout` (existing; now called internally by `ha_start_election_async`)
|
||
- Config replication SSL
|
||
|
||
### Allowed Dependencies
|
||
- `netio.c` / `endpoint_t` (add `on_connected` callback)
|
||
- `ha_elect_ctx_t` (existing; mutex guards concurrent access)
|
||
- `iomgr_t` / `iomgr_monitor` (existing)
|
||
- `xce_request()` / `endpoint_enq()` (existing)
|
||
- `handle_request_vote_reply()` (existing)
|
||
- `cluster_init_phase2()` (called from vtask context after election; must be vtask-safe)
|
||
|
||
### Revalidation Triggers
|
||
- Changes to `endpoint_t` structure
|
||
- Changes to `cluster_init_phase2` preconditions (anything it accesses that may not be ready)
|
||
- Changes to `ha_dispatch_gate` CTRL_ELECTING allowlist
|
||
|
||
## Architecture
|
||
|
||
### Before: Blocking startup
|
||
```mermaid
|
||
graph TD
|
||
myapp_run --> cluster_init_phase1
|
||
cluster_init_phase1 --> ha_wait_hb_or_timeout
|
||
ha_wait_hb_or_timeout -->|blocks 2s| timeout
|
||
timeout --> am_i_primary_quorum
|
||
am_i_primary_quorum -->|pthreads block on sem| result
|
||
result --> cluster_init_phase2
|
||
cluster_init_phase2 --> cluster_start
|
||
cluster_start --> cluster_run
|
||
```
|
||
|
||
### After: Fully event-driven
|
||
```mermaid
|
||
graph TD
|
||
myapp_run --> cluster_init_phase1
|
||
cluster_init_phase1 --> ssl_setup
|
||
ssl_setup --> cluster_register_listener
|
||
cluster_register_listener --> iomgr_vtask_start
|
||
iomgr_vtask_start --> ha_start_election_async
|
||
ha_start_election_async -->|returns immediately| cluster_run
|
||
|
||
iomgr_worker -->|EPOLLOUT handshake| on_vote_connected
|
||
on_vote_connected --> xce_request_vote
|
||
iomgr_worker -->|EPOLLIN reply| vtask_vote_reply_msg_handler
|
||
vtask_vote_reply_msg_handler --> handle_request_vote_reply
|
||
handle_request_vote_reply --> ha_elect_evaluate_and_signal
|
||
ha_elect_evaluate_and_signal --> cluster_init_phase2_thin
|
||
cluster_init_phase2_thin --> set_ha_role_atomic
|
||
set_ha_role_atomic -->|gate opens| serve_traffic
|
||
|
||
election_watchdog -->|timeout| ha_elect_evaluate_and_signal
|
||
heartbeat_watchdog -->|hb timeout or received| ha_elect_evaluate_and_signal
|
||
```
|
||
|
||
### Key Decisions
|
||
|
||
- **`ha_start_election_async()` is fully non-blocking**: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately. (Note: peer vote-endpoint connection and vote tallying are no longer driven directly by this function — they are now owned by the Raft library (`src/lib/raft/raft.c`): `ha_start_election_async()` calls `raft_start_election()`, which fires `raft_cb_send_vote_request` per peer and `raft_cb_become_leader`/`raft_cb_become_follower` on a terminal result. Superseded by raft-log-replication and raft-commit-safety, see those specs' design.md.)
|
||
- **Two promotion paths in `ha_elect_evaluate_and_signal`**:
|
||
- *Initial startup* (`ha->phase2_done == 0`): calls `cluster_init_phase2` (loads config data, inits structures), sets `ha->phase2_done = 1`. Phase2 runs exactly once per process lifetime — config upgrade always does a full restart.
|
||
- *RDHUP failover* (`ha->phase2_done == 1`): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly. (Note: see CTRL_PRIMARY_COMMITTING note below — this now enters CTRL_PRIMARY directly, no COMMITTING state.)
|
||
- **`is_first_boot` = `ha->phase2_done == 0`**: Not `!c->running` (which is 0 until `cluster_run` sets it, making all elections look like first boot). `phase2_done` is a dedicated bool in `ha_state_t`. (Still accurate — verified current in `ha.c`.)
|
||
- **`CTRL_PRIMARY_COMMITTING`**: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary). **(Note: superseded by ha-single-leader-role, see that spec's design.md — `CTRL_PRIMARY_COMMITTING` was subsequently removed. `ctrl_role_t` now has only 4 values (`CTRL_PRIMARY`, `CTRL_STANDBY`, `CTRL_INIT`, `CTRL_ELECTING`); a winning candidate enters `CTRL_PRIMARY` directly via `raft_cb_become_leader`/`ha_elect_evaluate_and_signal`, with no transitional committing state and no committing-timeout watchdog. Job-traffic admission during startup is now gated by `ha_has_write_quorum()` inside the `CTRL_PRIMARY` case of `ha_dispatch_gate`, not by a separate FSM state. Every other reference to CTRL_PRIMARY_COMMITTING in this document — the sequence diagrams, the `ha_repl_ack_received()`/`ha_dispatch_gate` COMMITTING-case code samples, the Error Handling and Testing Strategy sections below — describes this now-removed mechanism.)**
|
||
- **`cluster_run` unchanged**: Still just `cbuf_get(&wbuf)` for stop signal.
|
||
- **`delaymgr_start` and `persist_start` move to `myapp_run`**: Currently called inside `cluster_start()`. In the new design `cluster_start` is not called from `myapp_run`. These two calls move explicitly into `myapp_run` before `ha_start_election_async`.
|
||
- **`bootstrap_config_if_needed` removed entirely**: Replaced by the freshness gate in voting + primary replication push. No pre-election config probe needed.
|
||
- **`ha_send_ctrlhello_to_all_standbys`**: Iterates `ha->standby_endpoints` (populated when standbys connect and send `OPCode_hello`), sends `CtrlHelloRequest` to each. This triggers the existing catch-up replication in `handle_ctrlhello`. After catch-up, standby sends `OPCode_replCatchUpDone` → `ha_repl_ack_received`.
|
||
- **`committing_timeout_ms`**: Added to `ha_config_t` with default `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000` (10s). Covers the case where standbys are down at startup (single-node degraded mode).
|
||
- **Timeout mechanism — `delay_task_ms` + cancelled flag**: delaymgr is enhanced with one new API:
|
||
- `delay_task_ms(mgr, delay_ms, runcount, fn, arg)` — millisecond-granularity one-shot task using `clock_gettime(CLOCK_MONOTONIC)`. Precision bounded by `mgr->precision_ms`.
|
||
- **Cancellation via state flag**: each context struct (`election_timeout_ctx_t`, `committing_timeout_ctx_t`) carries an `_Atomic int cancelled` field. Setting it to 1 makes the callback a no-op when it fires. This is O(1), lock-free, and race-free — even if the worker dequeued the task and is mid-execution, the flag check at callback entry handles it correctly. No ordq scan or ID tracking needed.
|
||
```c
|
||
typedef struct {
|
||
_Atomic int cancelled;
|
||
ha_state_t *ha;
|
||
cluster_t *c;
|
||
ha_elect_ctx_t *ctx;
|
||
} election_timeout_ctx_t;
|
||
|
||
static int election_timeout_fn(void *arg) {
|
||
election_timeout_ctx_t *t = arg;
|
||
if (atomic_load(&t->cancelled)) return 0;
|
||
ha_elect_evaluate_and_signal(t->ha, t->ctx);
|
||
return 0;
|
||
}
|
||
|
||
/* Schedule: */
|
||
delay_task_ms(&c->delaymgr, ha->config.election_timeout_ms, 1, election_timeout_fn, ctx);
|
||
|
||
/* "Cancel" when quorum already reached: */
|
||
atomic_store(&ctx->cancelled, 1);
|
||
```
|
||
The existing heartbeat watchdog keeps its dedicated `sem_timedwait` pthread (precise rearm on every heartbeat — not suitable for delaymgr's polling model).
|
||
|
||
## File Structure Plan
|
||
|
||
```
|
||
src/lib/
|
||
├── netio.c # Add on_connected callback to endpoint_t; invoke in io_handler
|
||
├── delay.c # Add delay_task_ms() — ms-granularity using clock_gettime(CLOCK_MONOTONIC)
|
||
│ # Cancellation via caller-owned _Atomic int cancelled flag (no API needed)
|
||
└── delay.h # Expose delay_task_ms()
|
||
|
||
src/controller/
|
||
├── ha.h # Add to ha_state_t: client_ssl_ctx, phase2_done, repl_ack_count, n_peers
|
||
│ # Add to ha_config_t: committing_timeout_ms (default 10000ms)
|
||
│ # Add CTRL_PRIMARY_COMMITTING to ctrl_role_t enum
|
||
│ # (Note: repl_ack_count, committing_timeout_ms, and CTRL_PRIMARY_COMMITTING
|
||
│ # do not exist in current ha.h — superseded by ha-single-leader-role,
|
||
│ # see that spec's design.md. client_ssl_ctx, phase2_done, n_peers verified current.)
|
||
│ # Declare: ha_start_election_async(), ha_election_cleanup(),
|
||
│ # ha_repl_ack_received(), ha_arm/disarm_committing_timeout(),
|
||
│ # ha_send_ctrlhello_to_all_standbys()
|
||
│ # (Note: ha_repl_ack_received/ha_arm/disarm_committing_timeout do not
|
||
│ # exist — superseded by ha-single-leader-role. ha_send_ctrlhello_to_all_standbys
|
||
│ # verified current, called from raft_cb_become_leader.)
|
||
├── ha.c # Remove: vote_request_thread, vote_thread_ctx_t
|
||
│ # Add: vote_ep_ctx_t, on_vote_connected(), vote_reply_msg_handler()
|
||
│ # (Note: vote_reply_msg_handler was generalized into peer_ctrl_msg_handler —
|
||
│ # superseded by ha-prevote, see that spec's design.md)
|
||
│ # Add: ha_start_election_async(), election_timeout_watchdog
|
||
│ # Extend ha_elect_evaluate_and_signal: COMMITTING path + replication push
|
||
│ # Extend handle_request_vote: role-based deny gate
|
||
│ # (Note: handle_request_vote is now #ifdef XCE_TESTING-only — superseded
|
||
│ # by raft-adapter-conformance, see that spec's design.md)
|
||
│ # Add: ha_repl_ack_received(), ha_arm/disarm_committing_timeout()
|
||
│ # Extend ha_dispatch_gate: CTRL_PRIMARY_COMMITTING case
|
||
│ # (Note: none of these three exist — superseded by ha-single-leader-role;
|
||
│ # see the CTRL_PRIMARY_COMMITTING note in Key Decisions above)
|
||
├── cluster.c # cluster_init_phase2: vtask-safe (managers already in phase1)
|
||
│ # ha_send_ctrlhello_to_all_standbys() or reuse handle_ctrlhello path
|
||
│ # cluster_run: UNCHANGED
|
||
└── myapp.c # Restructured: phase1 + SSL + delaymgr/persist_start + ha_start_election_async + cluster_run
|
||
# Remove: ha_wait_for_heartbeat_or_timeout, am_i_primary_quorum,
|
||
# for(;;) retry loop, cluster_init_phase2, cluster_start calls,
|
||
# bootstrap_config_if_needed (replaced by freshness gate + primary push)
|
||
```
|
||
|
||
### Modified Files
|
||
- `src/lib/netio.c` — Add `on_connected` to `endpoint_t`; fire in `io_handler` after `ssl_handshake_done=1`
|
||
- `src/lib/delay.c` — Add `delay_task_ms()` using `clock_gettime(CLOCK_MONOTONIC)` for ms precision; precision still bounded by `mgr->precision_ms`
|
||
- `src/lib/delay.h` — Expose `delay_task_ms()`
|
||
- `src/controller/ha.h` — Add `client_ssl_ctx`, `phase2_done`, `repl_ack_count`, `CTRL_PRIMARY_COMMITTING`; declare new functions
|
||
- `src/controller/ha.c` — Main changes; `ha_dispatch_gate` extended for COMMITTING; uses `delay_task_ms`/`delay_task_cancel` for watchdogs
|
||
- `src/controller/cluster.c` — `ha_send_ctrlhello_to_all_standbys` helper; `OPCode_replCatchUpDone` calls `ha_repl_ack_received`; `cluster_run` unchanged
|
||
- `src/controller/myapp.c` — Restructured: non-blocking after phase1; `delaymgr_start`, `persist_start` moved here; `bootstrap_config_if_needed` removed
|
||
|
||
## System Flows
|
||
|
||
### 3-node startup — ctrl-a wins (with COMMITTING phase)
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Main as myapp_run
|
||
participant Async as ha_start_election_async
|
||
participant iomgr as iomgr Worker
|
||
participant vtask as vtaskmgr Worker
|
||
participant PeerB as ctrl-b endpoint
|
||
participant PeerC as ctrl-c endpoint
|
||
|
||
Main->>Main: cluster_init_phase1 + SSL + register_listener + delaymgr_start + persist_start
|
||
Main->>Async: ha_start_election_async (non-blocking)
|
||
Async->>PeerB: endpoint_connect(client_ssl_ctx)
|
||
Async->>PeerC: endpoint_connect(client_ssl_ctx)
|
||
Async->>iomgr: iomgr_monitor(peer_eps)
|
||
Async-->>Main: returns immediately
|
||
|
||
Main->>Main: cluster_run() — wbuf shutdown loop
|
||
|
||
iomgr->>PeerB: handshake → on_vote_connected → OP_requestVote
|
||
iomgr->>PeerC: handshake → on_vote_connected → OP_requestVote
|
||
iomgr->>vtask: vote replies → quorum reached
|
||
vtask->>vtask: ha_elect_evaluate_and_signal → cluster_init_phase2 (first boot)
|
||
vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
|
||
vtask->>PeerB: push_config_to_standbys + ha_send_ctrlhello (triggers WAL catch-up)
|
||
vtask->>PeerC: push_config_to_standbys + ha_send_ctrlhello
|
||
|
||
iomgr->>vtask: OPCode_replCatchUpDone from ctrl-b → ha_repl_ack_received (ack=1)
|
||
iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=2 ≥ quorum-1)
|
||
vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
|
||
Note over vtask: dispatch gate fully open — serves job traffic
|
||
```
|
||
|
||
### Primary crash — heartbeat timeout triggers promotion (RDHUP failover)
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant HBWatch as Heartbeat Watchdog on ctrl-b
|
||
participant vtask as vtaskmgr Worker
|
||
participant iomgr as iomgr on ctrl-b
|
||
participant PeerC as ctrl-c endpoint
|
||
|
||
Note over HBWatch: heartbeat_timeout_ms — primary silent
|
||
HBWatch->>vtask: ha_on_primary_ctrl_fd_rdhup → ha->role = CTRL_ELECTING
|
||
HBWatch->>iomgr: ha_start_election_async (retry — is_first_boot=false)
|
||
iomgr->>PeerC: vote exchange → ctrl-b wins
|
||
vtask->>vtask: ha_elect_evaluate_and_signal (is_first_boot=false → skip phase2)
|
||
vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
|
||
vtask->>PeerC: ha_send_ctrlhello (WAL already current — catch-up is fast)
|
||
iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=1 ≥ quorum-1)
|
||
vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
|
||
Note over vtask: ctrl-b is new primary — state already synchronized via prior replication
|
||
```
|
||
|
||
### Incoming request during election (NOT_READY)
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Client
|
||
participant iomgr as iomgr Worker
|
||
participant vtask as vtaskmgr Worker
|
||
|
||
Client->>iomgr: TCP connect + OP_jobsubmit
|
||
iomgr->>vtask: myreq_handler
|
||
vtask->>vtask: ha_dispatch_gate(CTRL_ELECTING, OPCode_jobsubmit) → XCE_ERR_NOT_READY
|
||
vtask->>Client: reply status = XCE_ERR_NOT_READY (client retries)
|
||
```
|
||
|
||
## Requirements Traceability
|
||
|
||
| Requirement | Summary | Components | Flows |
|
||
|-------------|---------|------------|-------|
|
||
| 1.1 | No pthreads for vote I/O | `ha_start_election_async` | 3-node startup |
|
||
| 1.2 | Vote RPC via iomgr path | `on_vote_connected`, `xce_request` | 3-node startup |
|
||
| 1.3 | Non-responding peer → deny | election watchdog → evaluate | — |
|
||
| 2.1 | Async TCP+TLS via iomgr | `endpoint_connect`, `iomgr_monitor` | 3-node startup |
|
||
| 2.2 | TLS client context | `client_ssl_ctx` in `ha_state_t` | 3-node startup |
|
||
| 2.3 | Send vote after handshake | `on_vote_connected` callback | 3-node startup |
|
||
| 2.4 | Connect/TLS failure → deny | `vote_ep_err_handler` | — |
|
||
| 3.1 | Reply via iomgr packet | `vote_reply_msg_handler`§ | 3-node startup |
|
||
| 3.2 | No capnp UAF | vtask handler reads before free | 3-node startup |
|
||
| 3.3 | Extract reply fields | `vote_reply_msg_handler`§ decodes | 3-node startup |
|
||
| 3.4 | Term fencing on reply | `handle_request_vote_reply` (existing) | — |
|
||
| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING† | `ha_elect_evaluate_and_signal` → committing state† | 3-node startup |
|
||
| 4.2 | Freshness gate guarantees ground-truth | `handle_request_vote` config_tag check‡ | — |
|
||
| 4.3 | Cold start: only highest config_tag wins | freshness gate + role gate combined | — |
|
||
| 4.4–4.7 | Retry, timeout, heartbeat abort | watchdog, re-arm, heartbeat watchdog | Primary crash |
|
||
| 5.1 | ELECTING gate: NOT_READY for non-election | `ha_dispatch_gate` (existing + COMMITTING)† | NOT_READY flow |
|
||
| 5.2 | COMMITTING: only repl acks allowed | `ha_dispatch_gate` new COMMITTING case† | 3-node startup |
|
||
| 5.3 | PRIMARY: full traffic | `ha_dispatch_gate` (existing) | — |
|
||
| 5.4–5.5 | STANDBY gate, term persistence | `ha_dispatch_gate`, `ha_persist_term` (existing) | — |
|
||
| 6.1 | PRIMARY denies all votes | `handle_request_vote` role gate‡ | — |
|
||
| 6.2 | STANDBY+fresh hb denies votes | `handle_request_vote` heartbeat check‡ | — |
|
||
| 6.3 | HB timeout → ELECTING | heartbeat watchdog fires RDHUP path | Primary crash |
|
||
| 6.4 | INIT/ELECTING evaluates Raft gates | `handle_request_vote` existing gates‡ | — |
|
||
| 6.5 | Primary crash → new primary | HB timeout → re-election → COMMITTING → PRIMARY† | Primary crash |
|
||
| 7.1 | Winner pushes config to standbys | `push_config_to_standbys()` in committing path | 3-node startup |
|
||
| 7.2 | Winner replicates WAL to standbys | `handle_ctrlhello()` catch-up (existing) | 3-node startup |
|
||
| 7.3 | config_tag=0 standby gets config via push | no independent fetch; primary pushes | 3-node startup |
|
||
| 7.4 | Ack tracked per standby | `ha_state_t.repl_ack_count`† | 3-node startup |
|
||
| 8.1 | COMMITTING blocks job traffic | `ha_dispatch_gate` COMMITTING case† | 3-node startup |
|
||
| 8.2 | Majority ack → CTRL_PRIMARY | `ha_repl_ack_received()` threshold check† | 3-node startup |
|
||
| 8.3 | CTRL_PRIMARY opens full traffic | `ha_dispatch_gate` PRIMARY case (existing) | 3-node startup |
|
||
| 8.4 | Ack timeout → degraded open | timeout watchdog in committing state† | — |
|
||
| 9.1–9.4 | Regressions | Single-node, ground-truth invariant, RDHUP hooks-only, priority stagger | — |
|
||
|
||
† Rows marked † describe the `CTRL_PRIMARY_COMMITTING` mechanism, removed by ha-single-leader-role —
|
||
see the Key Decisions note above (a winner now enters `CTRL_PRIMARY` directly; job-traffic admission
|
||
during startup is gated by `ha_has_write_quorum()` inside `ha_dispatch_gate`'s `CTRL_PRIMARY` case).
|
||
‡ Rows marked ‡ cite `handle_request_vote`, now `#ifdef XCE_TESTING`-only; production vote-request
|
||
handling is `cluster.c`'s `case OPCode_requestVote:` → `raft_on_vote_request()` (superseded by
|
||
raft-adapter-conformance, see that spec's design.md).
|
||
§ Rows marked § cite `vote_reply_msg_handler`, generalized into `peer_ctrl_msg_handler` (superseded
|
||
by ha-prevote, see that spec's design.md).
|
||
|
||
## Components and Interfaces
|
||
|
||
### Summary
|
||
|
||
| Component | File | Intent | Req Coverage |
|
||
|-----------|------|--------|--------------|
|
||
| `endpoint_t.on_connected` | netio.c | Async handshake notification | 2.1, 2.3 |
|
||
| `ha_state_t.client_ssl_ctx` | ha.h/myapp.c | TLS client context | 2.2 |
|
||
| `vote_ep_ctx_t` | ha.c | Per-peer context for callbacks | 1.1, 2.1–2.4 |
|
||
| `on_vote_connected()` | ha.c | Send vote RPC after handshake | 2.3, 3.1 |
|
||
| `vote_reply_msg_handler()`§ | ha.c | Decode reply; update tally | 3.1–3.4 |
|
||
| `ha_start_election_async()`¶ | ha.c | Non-blocking: connect peers + arm watchdog | 1.1–1.3, 4.1–4.3 |
|
||
| Election timeout watchdog | ha.c | Fire evaluate if deadline expires | 1.3, 4.2, 4.3 |
|
||
| `handle_request_vote` role gate‡ | ha.c | Deny votes from PRIMARY/healthy-STANDBY | 6.1–6.4 |
|
||
| `ha_elect_evaluate_and_signal` extended | ha.c | Enter COMMITTING; push config+WAL; set role† | 4.1, 7.1–7.4 |
|
||
| `CTRL_PRIMARY_COMMITTING` state† | ha.h + ha_dispatch_gate | Block job traffic; allow repl acks | 5.2, 8.1 |
|
||
| `ha_repl_ack_received()`† | ha.c | Count standby acks; transition to PRIMARY on quorum | 8.2–8.3 |
|
||
| `ha_state_t.repl_ack_count`† | ha.h | Atomic ack counter per election round | 7.4, 8.2 |
|
||
|
||
† = superseded by ha-single-leader-role (CTRL_PRIMARY_COMMITTING removed; see Key Decisions note above).
|
||
‡ = `handle_request_vote` is now test-only; production path is `raft_on_vote_request()` (raft-adapter-conformance).
|
||
§ = renamed/generalized to `peer_ctrl_msg_handler` (ha-prevote).
|
||
¶ = peer connect + vote tally now delegated to `raft_start_election()`/`raft_on_vote_reply()` (raft-log-replication, raft-commit-safety).
|
||
|
||
### Key Interfaces
|
||
|
||
#### `ha_start_election_async()`
|
||
|
||
**(Note: superseded by raft-log-replication and raft-commit-safety, see those specs' design.md. This
|
||
function no longer connects vote endpoints or tallies votes directly — it delegates to
|
||
`raft_start_election()`, which fires `raft_cb_send_vote_request` per peer and, on a terminal
|
||
result, `raft_cb_become_leader`/`raft_cb_become_follower`. The signature and non-blocking contract
|
||
below remain accurate.)**
|
||
|
||
```c
|
||
/* Non-blocking. Arms heartbeat watchdog + election watchdog.
|
||
* Connects outgoing vote endpoints. On terminal state:
|
||
* ha_elect_evaluate_and_signal() calls cluster_init_phase2() then sets ha->role. */
|
||
void ha_start_election_async(ha_state_t *ha,
|
||
cluster_t *c,
|
||
int n_peers,
|
||
xce_candidate_t **peers,
|
||
uint64_t my_config_tag,
|
||
uint64_t my_lsn,
|
||
const char *my_name);
|
||
```
|
||
|
||
#### `ha_elect_evaluate_and_signal` (extended)
|
||
|
||
**(Note: superseded by ha-single-leader-role, see that spec's design.md. The COMMITTING transition
|
||
below — `CTRL_PRIMARY_COMMITTING`, `repl_ack_count`, `ha_arm_committing_timeout` — no longer exists;
|
||
a winning candidate is set to `CTRL_PRIMARY` directly. The `is_first_boot`/`phase2_done` gating and
|
||
the STANDBY branch below remain accurate and current.)**
|
||
|
||
```c
|
||
/* After election determines role: */
|
||
int is_first_boot = !ha->phase2_done; /* phase2_done set after first cluster_init_phase2 call */
|
||
|
||
if (result == CTRL_PRIMARY) {
|
||
if (is_first_boot) {
|
||
cluster_init_phase2(c); /* load config+data; runs once per process lifetime */
|
||
ha->phase2_done = 1;
|
||
}
|
||
/* Enter COMMITTING: enable hooks but block job traffic until majority ack (Req 8.1) */
|
||
atomic_store(&ha->role, CTRL_PRIMARY_COMMITTING, memory_order_release);
|
||
ha->repl_ack_count = 0; /* reset ack counter for this round */
|
||
/* Push config and replicate WAL to standbys (Req 7.1, 7.2) */
|
||
push_config_to_standbys(c, ...);
|
||
ha_send_ctrlhello_to_all_standbys(c); /* triggers catch-up replication */
|
||
/* Arm committing timeout (degraded-mode fallback, Req 8.4) */
|
||
ha_arm_committing_timeout(ha, c);
|
||
} else {
|
||
if (is_first_boot) {
|
||
cluster_init_phase2(c); /* init structures; config arrives via primary push */
|
||
ha->phase2_done = 1;
|
||
}
|
||
atomic_store(&ha->role, CTRL_STANDBY, memory_order_release);
|
||
/* re-arm: heartbeat watchdog fires ha_on_primary_ctrl_fd_rdhup on next timeout */
|
||
}
|
||
```
|
||
|
||
#### `ha_repl_ack_received()` — called when a standby acks config+WAL catch-up
|
||
|
||
**(Note: this function does not exist in current code — superseded by ha-single-leader-role, see
|
||
that spec's design.md. Standby ack counting toward promotion was removed along with
|
||
`CTRL_PRIMARY_COMMITTING`; job-traffic admission is now `ha_has_write_quorum()`'s live count of
|
||
connected standbys, evaluated per-request inside `ha_dispatch_gate`, not a one-time per-round tally.)**
|
||
|
||
```c
|
||
void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) {
|
||
int acks = atomic_fetch_add(&ha->repl_ack_count, 1) + 1;
|
||
int quorum = (ha->n_peers + 2) / 2; /* majority of total cluster */
|
||
if (acks >= quorum - 1) { /* -1 because primary counts itself */
|
||
ha_disarm_committing_timeout(ha);
|
||
ha_promote_to_primary(c); /* sets role=CTRL_PRIMARY, enables hooks */
|
||
}
|
||
}
|
||
```
|
||
|
||
Called from the existing `handle_ctrlhello` reply path (CtrlHelloReply ack from standby after catch-up) and from `push_config_to_standbys` ack handler.
|
||
|
||
#### `handle_request_vote` role gate (extended)
|
||
|
||
**(Note: superseded by raft-adapter-conformance, see that spec's design.md. `handle_request_vote`
|
||
is now `#ifdef XCE_TESTING`-only, kept as a test harness entry point; production vote-request
|
||
dispatch is `cluster.c`'s `case OPCode_requestVote:`, which applies its own freshness gate and then
|
||
calls `raft_on_vote_request()` — the PRIMARY-role deny is now `on_vote_eligible`/`RAFT_LEADER`
|
||
inside the raft library, not this snippet. `CTRL_PRIMARY_COMMITTING` in the snippet below also no
|
||
longer exists — see the Key Decisions note above.)**
|
||
|
||
```c
|
||
ctrl_role_t my_role = atomic_load_explicit(&ha->role, memory_order_acquire);
|
||
if (my_role == CTRL_PRIMARY || my_role == CTRL_PRIMARY_COMMITTING)
|
||
return 0; /* deny — I am primary (or committing); don't grant votes to challengers */
|
||
if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha))
|
||
return 0; /* deny — my primary is alive */
|
||
/* CTRL_INIT, CTRL_ELECTING, or STANDBY without live primary: evaluate Raft gates below */
|
||
```
|
||
|
||
#### `ha_heartbeat_is_fresh()`
|
||
|
||
**(Fixed: signature and field name were stale — corrected to match current `ha.c`. The function is
|
||
non-static (exported) and reads `ha->last_activity`, a whole-second `time_t`, not a
|
||
`struct timespec last_heartbeat` — the design's original nsec-precision sketch was implemented as
|
||
whole-second precision, and it now null-guards `last_activity == 0` for "no primary message seen yet".)**
|
||
|
||
```c
|
||
int ha_heartbeat_is_fresh(ha_state_t *ha) {
|
||
time_t last = ha->last_activity; /* time_t load is atomic on LP64 */
|
||
if (last == 0)
|
||
return 0; /* no primary message seen yet */
|
||
struct timespec now;
|
||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||
long ms = (long)(now.tv_sec - last) * 1000L;
|
||
return ms < (long)ha->config.heartbeat_timeout_ms;
|
||
}
|
||
```
|
||
|
||
#### `myapp_run` (restructured — no blocking)
|
||
```c
|
||
int myapp_run(myconf_t *myconf) {
|
||
cluster_t c; memset(&c, 0, sizeof(c)); c.myconf = myconf;
|
||
cluster_init_phase1(&c); /* tunables, managers, endpoints zarray */
|
||
initialize_openssl();
|
||
c.ha.ssl_ctx = create_ssl_context(myconf->certfile, myconf->keyfile);
|
||
c.ha.client_ssl_ctx = SSL_CTX_new(TLS_client_method());
|
||
SSL_CTX_set_verify(c.ha.client_ssl_ctx, SSL_VERIFY_NONE, NULL);
|
||
iomgr_init(&c.iomgr); iomgr_set_ssl_ctx(&c.iomgr, c.ha.ssl_ctx);
|
||
vtaskmgr_init(&c.vtaskmgr, nvcpus);
|
||
iomgr_start(&c.iomgr); vtaskmgr_start(&c.vtaskmgr);
|
||
cluster_register_listener(&c); /* listener in epoll before election */
|
||
delaymgr_init(&c.delaymgr); delaymgr_start(&c.delaymgr); /* moved from cluster_start */
|
||
persist_init(&c.persist, ...); persist_start(&c.persist); /* moved from cluster_start */
|
||
load_primary_config(&c); /* pre-load config_tag for freshness gate */
|
||
/* bootstrap_config_if_needed() REMOVED — freshness gate + primary push replaces it */
|
||
/* Build peer array from myconf->peers_cfg ... */
|
||
ha_start_election_async(&c.ha, &c, n_peers, peer_ptrs, my_tag, my_lsn, myconf->name);
|
||
return cluster_run(&c); /* unchanged — just waits for stop signal */
|
||
}
|
||
```
|
||
|
||
**Note**: `vote_ep_ctx_t` carries `cluster_t *c` so that `ha_elect_evaluate_and_signal`
|
||
(called from vtask context via `vote_reply_msg_handler`§) has access to `cluster_t` for
|
||
`cluster_init_phase2`, `push_config_to_standbys`, and `ha_send_ctrlhello_to_all_standbys`.
|
||
§ (renamed/generalized to `peer_ctrl_msg_handler` — superseded by ha-prevote.)
|
||
|
||
#### `ha_dispatch_gate` — CTRL_PRIMARY_COMMITTING case
|
||
|
||
**(Note: superseded by ha-single-leader-role, see that spec's design.md. This `case` does not exist —
|
||
`ha_dispatch_gate`'s switch has only `CTRL_PRIMARY`, `CTRL_INIT`, `CTRL_ELECTING`, `CTRL_STANDBY`.
|
||
The write-admission gating this case describes now lives inside the `CTRL_PRIMARY` case itself, via
|
||
`ha_has_write_quorum()`: state-changing ops are refused with `XCE_ERR_NOT_READY` when the live
|
||
connected-standby count is below majority, while the same replication/read-only opcode allowlist
|
||
below — plus fact-bearing ops (`OPCode_notify`/`update`/`setpreason`/`clrpreason`) — stays admitted.)**
|
||
|
||
```c
|
||
case CTRL_PRIMARY_COMMITTING:
|
||
/* Allow replication acks (config push ack, CtrlHelloReply, WAL ack) so
|
||
* ha_repl_ack_received() can count them toward majority.
|
||
* Deny job dispatch and queue/resource management until majority ack. */
|
||
switch (opcode) {
|
||
case OPCode_heartbeat:
|
||
case OPCode_replConfig: /* config push ack from standby */
|
||
case OPCode_replConfigPush:
|
||
case OPCode_upgradeConfigAck:
|
||
case OPCode_hello: /* CtrlHelloReply ack from standby after catch-up */
|
||
case OPCode_walAck:
|
||
case OPCode_replEntry:
|
||
case OPCode_replAck:
|
||
case OPCode_replCatchUpDone: /* standby signals catch-up complete → count ack */
|
||
return 0;
|
||
default:
|
||
return -XCE_ERR_NOT_READY;
|
||
}
|
||
```
|
||
|
||
`OPCode_replCatchUpDone` handler in `cluster.c` calls `ha_repl_ack_received(ha, c)` to
|
||
increment the ack counter and conditionally transition to `CTRL_PRIMARY`.
|
||
(Note: `ha_repl_ack_received` does not exist — see notes above; superseded by ha-single-leader-role.)
|
||
|
||
## Error Handling
|
||
|
||
- **Connect failure**: endpoint skipped; peer absent from election. After watchdog timeout, election evaluates with available votes only.
|
||
- **SSL handshake failure**: `vote_ep_err_handler` calls `handle_request_vote_reply(grant=0)`.
|
||
- **`cluster_init_phase2` failure from vtask**: logs ERROR; posts stop to wbuf → clean shutdown.
|
||
- **Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING**: `ha_dispatch_gate` returns `XCE_ERR_NOT_READY`; client retries. (Note: `CTRL_PRIMARY_COMMITTING` no longer exists — superseded by ha-single-leader-role; the equivalent current behavior is `ha_dispatch_gate`'s `CTRL_PRIMARY` case refusing state-changing ops via `ha_has_write_quorum()` until majority standbys are connected.)
|
||
- **Committing timeout (Req 8.4)**: if majority ack not received within `committing_timeout_ms`, watchdog fires `ha_promote_to_primary` anyway (degraded mode) with a WARN log. Prevents deadlock when standbys are unavailable at startup.
|
||
- **Vote re-arm on STANDBY**: `ha_elect_evaluate_and_signal` re-arms `ha_start_election_async`. Heartbeat watchdog fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING → re-election.
|
||
|
||
## Testing Strategy
|
||
|
||
### Unit Tests
|
||
1. `ha_start_election_async` returns in < 5 ms without blocking
|
||
2. `on_vote_connected` encodes and sends correct `OP_requestVote`
|
||
3. `vote_reply_msg_handler` reads capnp fields before free — ASAN clean
|
||
4. `handle_request_vote` role gate: PRIMARY→deny; STANDBY+fresh→deny; STANDBY+stale→evaluate; ELECTING→evaluate
|
||
5. `ha_elect_evaluate_and_signal` enters COMMITTING then PRIMARY after majority ack
|
||
6. Election watchdog fires within `election_timeout_ms + 200ms` when no replies
|
||
7. `ha_repl_ack_received`: majority acks → `CTRL_PRIMARY`; below majority → stays COMMITTING
|
||
8. Committing timeout watchdog: promotes to PRIMARY (degraded) if acks don't arrive
|
||
|
||
### Integration Tests
|
||
9. 3-node startup: ctrl-a wins election → CTRL_PRIMARY_COMMITTING → pushes config to ctrl-b/c → receives replCatchUpDone × 2 → CTRL_PRIMARY → serves jobs
|
||
10. ctrl-b/c start with config_tag=0; receive config via primary push (no independent fetch); confirm via replCatchUpDone
|
||
11. Primary crash (SIGKILL): ctrl-b heartbeat timeout → re-election → COMMITTING → ack from ctrl-c → PRIMARY; serves traffic from last WAL state
|
||
12. Client during COMMITTING receives `XCE_ERR_NOT_READY` and retries successfully after gate opens
|
||
|
||
### Regression Tests
|
||
13. Single-node self-elects; no standbys → committing timeout fires → degraded PRIMARY immediately
|
||
14. ASAN clean: no UAF, no stack-use-after-return in election or committing path
|