chore: checkpoint all prior work for clean state
Captures all uncommitted work from ha-config-bootstrap, ha-heartbeat-liveness, and associated spec/tooling updates: - src: ha.c, ha.h, cluster.c, config_repl.c, myapp.c, raft.c — heartbeat liveness, RDHUP integration, committing phase fixes - tests: test_ha_role.c, test_config_repl.c, test_ha_rdhup_integration.c, test_p1_integration.c, test_p3_hardening.c, scheduler test - specs: election-iomgr-refactor, ha-heartbeat-liveness, ha-peer-connection-reuse, raft-library (new); ha-config-bootstrap and election-iomgr-refactor task/research updates - tooling: .wolf/ state, ha/data cluster fixtures, raft/raft.tla, src/build symlink, xce-build skill update
This commit is contained in:
parent
15bd291bdd
commit
e31673dd88
66 changed files with 122422 additions and 860 deletions
470
.kiro/specs/election-iomgr-refactor/design.md
Normal file
470
.kiro/specs/election-iomgr-refactor/design.md
Normal file
|
|
@ -0,0 +1,470 @@
|
|||
# Design Document: election-iomgr-refactor
|
||||
|
||||
## Overview
|
||||
|
||||
The XCE controller was originally fully event-driven: `cluster_init` → `cluster_start` →
|
||||
`cluster_run` (shutdown-check loop). The HA additions broke this by inserting blocking calls
|
||||
before `cluster_run`: `ha_wait_for_heartbeat_or_timeout` and `am_i_primary_quorum` block the
|
||||
main thread, causing SSL context confusion, capnp UAF, and mixed-I/O races.
|
||||
|
||||
This design restores full event-driven operation with no blocking calls after phase1.
|
||||
`myapp_run` does setup then immediately enters `cluster_run`. All election I/O flows through
|
||||
iomgr callbacks. When the election reaches a terminal state, `ha_elect_evaluate_and_signal`
|
||||
calls `cluster_init_phase2` inline from vtask context (phase2 is already thin — managers were
|
||||
moved to phase1 in a prior refactor), then atomically sets `ha->role`. The dispatch gate
|
||||
(`ha_dispatch_gate`) opens automatically: the next client request sees PRIMARY or STANDBY and
|
||||
is handled normally. No semaphores, no wbuf signals, no changes to `cluster_run`.
|
||||
|
||||
### Goals
|
||||
- Restore event-driven model: `myapp_run` → `cluster_run` with zero blocking between them
|
||||
- All election I/O through one epoll event loop (iomgr)
|
||||
- `ha_elect_evaluate_and_signal` calls thin `cluster_init_phase2` from vtask context; sets role atomically
|
||||
- `cluster_run` unchanged — still just the shutdown-check loop
|
||||
- `myreq_handler` / `ha_dispatch_gate` is the sole FSM dispatcher — no new state machinery needed
|
||||
- Role-based vote gate: PRIMARY and STANDBY-with-live-primary deny votes
|
||||
|
||||
### Non-Goals
|
||||
- Changes to `cluster_run` or `wbuf` signal values
|
||||
- Incoming vote RPC handling (already correct in cluster.c)
|
||||
- Heartbeat watchdog thread (existing, unchanged)
|
||||
- Config replication, RDHUP paths (unchanged)
|
||||
- Async DNS resolution (deferred)
|
||||
- TLS certificate verification for vote connections
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- `vote_request_thread` + `am_i_primary_quorum` blocking loop → replaced by `ha_start_election_async()`
|
||||
- `on_connected` callback added to `netio.c`
|
||||
- `client_ssl_ctx` in `ha_state_t` + `myapp_run`
|
||||
- `vote_reply_msg_handler` for iomgr-native reply handling
|
||||
- Election timeout watchdog thread
|
||||
- `ha_elect_evaluate_and_signal` extended to call `cluster_init_phase2` then set role
|
||||
- `handle_request_vote` extended with role-based deny gate
|
||||
- `myapp_run` restructured: phase1 + SSL + `ha_start_election_async` + `cluster_run` directly
|
||||
|
||||
### Out of Boundary
|
||||
- `cluster_run` internals (unchanged)
|
||||
- `case OPCode_requestVote:` in cluster.c (already correct, no changes)
|
||||
- `ha_wait_for_heartbeat_or_timeout` (existing; now called internally by `ha_start_election_async`)
|
||||
- Config replication SSL
|
||||
|
||||
### Allowed Dependencies
|
||||
- `netio.c` / `endpoint_t` (add `on_connected` callback)
|
||||
- `ha_elect_ctx_t` (existing; mutex guards concurrent access)
|
||||
- `iomgr_t` / `iomgr_monitor` (existing)
|
||||
- `xce_request()` / `endpoint_enq()` (existing)
|
||||
- `handle_request_vote_reply()` (existing)
|
||||
- `cluster_init_phase2()` (called from vtask context after election; must be vtask-safe)
|
||||
|
||||
### Revalidation Triggers
|
||||
- Changes to `endpoint_t` structure
|
||||
- Changes to `cluster_init_phase2` preconditions (anything it accesses that may not be ready)
|
||||
- Changes to `ha_dispatch_gate` CTRL_ELECTING allowlist
|
||||
|
||||
## Architecture
|
||||
|
||||
### Before: Blocking startup
|
||||
```mermaid
|
||||
graph TD
|
||||
myapp_run --> cluster_init_phase1
|
||||
cluster_init_phase1 --> ha_wait_hb_or_timeout
|
||||
ha_wait_hb_or_timeout -->|blocks 2s| timeout
|
||||
timeout --> am_i_primary_quorum
|
||||
am_i_primary_quorum -->|pthreads block on sem| result
|
||||
result --> cluster_init_phase2
|
||||
cluster_init_phase2 --> cluster_start
|
||||
cluster_start --> cluster_run
|
||||
```
|
||||
|
||||
### After: Fully event-driven
|
||||
```mermaid
|
||||
graph TD
|
||||
myapp_run --> cluster_init_phase1
|
||||
cluster_init_phase1 --> ssl_setup
|
||||
ssl_setup --> cluster_register_listener
|
||||
cluster_register_listener --> iomgr_vtask_start
|
||||
iomgr_vtask_start --> ha_start_election_async
|
||||
ha_start_election_async -->|returns immediately| cluster_run
|
||||
|
||||
iomgr_worker -->|EPOLLOUT handshake| on_vote_connected
|
||||
on_vote_connected --> xce_request_vote
|
||||
iomgr_worker -->|EPOLLIN reply| vtask_vote_reply_msg_handler
|
||||
vtask_vote_reply_msg_handler --> handle_request_vote_reply
|
||||
handle_request_vote_reply --> ha_elect_evaluate_and_signal
|
||||
ha_elect_evaluate_and_signal --> cluster_init_phase2_thin
|
||||
cluster_init_phase2_thin --> set_ha_role_atomic
|
||||
set_ha_role_atomic -->|gate opens| serve_traffic
|
||||
|
||||
election_watchdog -->|timeout| ha_elect_evaluate_and_signal
|
||||
heartbeat_watchdog -->|hb timeout or received| ha_elect_evaluate_and_signal
|
||||
```
|
||||
|
||||
### Key Decisions
|
||||
|
||||
- **`ha_start_election_async()` is fully non-blocking**: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately.
|
||||
- **Two promotion paths in `ha_elect_evaluate_and_signal`**:
|
||||
- *Initial startup* (`ha->phase2_done == 0`): calls `cluster_init_phase2` (loads config data, inits structures), sets `ha->phase2_done = 1`. Phase2 runs exactly once per process lifetime — config upgrade always does a full restart.
|
||||
- *RDHUP failover* (`ha->phase2_done == 1`): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly.
|
||||
- **`is_first_boot` = `ha->phase2_done == 0`**: Not `!c->running` (which is 0 until `cluster_run` sets it, making all elections look like first boot). `phase2_done` is a dedicated bool in `ha_state_t`.
|
||||
- **`CTRL_PRIMARY_COMMITTING`**: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary).
|
||||
- **`cluster_run` unchanged**: Still just `cbuf_get(&wbuf)` for stop signal.
|
||||
- **`delaymgr_start` and `persist_start` move to `myapp_run`**: Currently called inside `cluster_start()`. In the new design `cluster_start` is not called from `myapp_run`. These two calls move explicitly into `myapp_run` before `ha_start_election_async`.
|
||||
- **`bootstrap_config_if_needed` removed entirely**: Replaced by the freshness gate in voting + primary replication push. No pre-election config probe needed.
|
||||
- **`ha_send_ctrlhello_to_all_standbys`**: Iterates `ha->standby_endpoints` (populated when standbys connect and send `OPCode_hello`), sends `CtrlHelloRequest` to each. This triggers the existing catch-up replication in `handle_ctrlhello`. After catch-up, standby sends `OPCode_replCatchUpDone` → `ha_repl_ack_received`.
|
||||
- **`committing_timeout_ms`**: Added to `ha_config_t` with default `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000` (10s). Covers the case where standbys are down at startup (single-node degraded mode).
|
||||
- **Timeout mechanism — `delay_task_ms` + cancelled flag**: delaymgr is enhanced with one new API:
|
||||
- `delay_task_ms(mgr, delay_ms, runcount, fn, arg)` — millisecond-granularity one-shot task using `clock_gettime(CLOCK_MONOTONIC)`. Precision bounded by `mgr->precision_ms`.
|
||||
- **Cancellation via state flag**: each context struct (`election_timeout_ctx_t`, `committing_timeout_ctx_t`) carries an `_Atomic int cancelled` field. Setting it to 1 makes the callback a no-op when it fires. This is O(1), lock-free, and race-free — even if the worker dequeued the task and is mid-execution, the flag check at callback entry handles it correctly. No ordq scan or ID tracking needed.
|
||||
```c
|
||||
typedef struct {
|
||||
_Atomic int cancelled;
|
||||
ha_state_t *ha;
|
||||
cluster_t *c;
|
||||
ha_elect_ctx_t *ctx;
|
||||
} election_timeout_ctx_t;
|
||||
|
||||
static int election_timeout_fn(void *arg) {
|
||||
election_timeout_ctx_t *t = arg;
|
||||
if (atomic_load(&t->cancelled)) return 0;
|
||||
ha_elect_evaluate_and_signal(t->ha, t->ctx);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Schedule: */
|
||||
delay_task_ms(&c->delaymgr, ha->config.election_timeout_ms, 1, election_timeout_fn, ctx);
|
||||
|
||||
/* "Cancel" when quorum already reached: */
|
||||
atomic_store(&ctx->cancelled, 1);
|
||||
```
|
||||
The existing heartbeat watchdog keeps its dedicated `sem_timedwait` pthread (precise rearm on every heartbeat — not suitable for delaymgr's polling model).
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
```
|
||||
src/lib/
|
||||
├── netio.c # Add on_connected callback to endpoint_t; invoke in io_handler
|
||||
├── delay.c # Add delay_task_ms() — ms-granularity using clock_gettime(CLOCK_MONOTONIC)
|
||||
│ # Cancellation via caller-owned _Atomic int cancelled flag (no API needed)
|
||||
└── delay.h # Expose delay_task_ms()
|
||||
|
||||
src/controller/
|
||||
├── ha.h # Add to ha_state_t: client_ssl_ctx, phase2_done, repl_ack_count, n_peers
|
||||
│ # Add to ha_config_t: committing_timeout_ms (default 10000ms)
|
||||
│ # Add CTRL_PRIMARY_COMMITTING to ctrl_role_t enum
|
||||
│ # Declare: ha_start_election_async(), ha_election_cleanup(),
|
||||
│ # ha_repl_ack_received(), ha_arm/disarm_committing_timeout(),
|
||||
│ # ha_send_ctrlhello_to_all_standbys()
|
||||
├── ha.c # Remove: vote_request_thread, vote_thread_ctx_t
|
||||
│ # Add: vote_ep_ctx_t, on_vote_connected(), vote_reply_msg_handler()
|
||||
│ # Add: ha_start_election_async(), election_timeout_watchdog
|
||||
│ # Extend ha_elect_evaluate_and_signal: COMMITTING path + replication push
|
||||
│ # Extend handle_request_vote: role-based deny gate
|
||||
│ # Add: ha_repl_ack_received(), ha_arm/disarm_committing_timeout()
|
||||
│ # Extend ha_dispatch_gate: CTRL_PRIMARY_COMMITTING case
|
||||
├── cluster.c # cluster_init_phase2: vtask-safe (managers already in phase1)
|
||||
│ # ha_send_ctrlhello_to_all_standbys() or reuse handle_ctrlhello path
|
||||
│ # cluster_run: UNCHANGED
|
||||
└── myapp.c # Restructured: phase1 + SSL + delaymgr/persist_start + ha_start_election_async + cluster_run
|
||||
# Remove: ha_wait_for_heartbeat_or_timeout, am_i_primary_quorum,
|
||||
# for(;;) retry loop, cluster_init_phase2, cluster_start calls,
|
||||
# bootstrap_config_if_needed (replaced by freshness gate + primary push)
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
- `src/lib/netio.c` — Add `on_connected` to `endpoint_t`; fire in `io_handler` after `ssl_handshake_done=1`
|
||||
- `src/lib/delay.c` — Add `delay_task_ms()` using `clock_gettime(CLOCK_MONOTONIC)` for ms precision; precision still bounded by `mgr->precision_ms`
|
||||
- `src/lib/delay.h` — Expose `delay_task_ms()`
|
||||
- `src/controller/ha.h` — Add `client_ssl_ctx`, `phase2_done`, `repl_ack_count`, `CTRL_PRIMARY_COMMITTING`; declare new functions
|
||||
- `src/controller/ha.c` — Main changes; `ha_dispatch_gate` extended for COMMITTING; uses `delay_task_ms`/`delay_task_cancel` for watchdogs
|
||||
- `src/controller/cluster.c` — `ha_send_ctrlhello_to_all_standbys` helper; `OPCode_replCatchUpDone` calls `ha_repl_ack_received`; `cluster_run` unchanged
|
||||
- `src/controller/myapp.c` — Restructured: non-blocking after phase1; `delaymgr_start`, `persist_start` moved here; `bootstrap_config_if_needed` removed
|
||||
|
||||
## System Flows
|
||||
|
||||
### 3-node startup — ctrl-a wins (with COMMITTING phase)
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Main as myapp_run
|
||||
participant Async as ha_start_election_async
|
||||
participant iomgr as iomgr Worker
|
||||
participant vtask as vtaskmgr Worker
|
||||
participant PeerB as ctrl-b endpoint
|
||||
participant PeerC as ctrl-c endpoint
|
||||
|
||||
Main->>Main: cluster_init_phase1 + SSL + register_listener + delaymgr_start + persist_start
|
||||
Main->>Async: ha_start_election_async (non-blocking)
|
||||
Async->>PeerB: endpoint_connect(client_ssl_ctx)
|
||||
Async->>PeerC: endpoint_connect(client_ssl_ctx)
|
||||
Async->>iomgr: iomgr_monitor(peer_eps)
|
||||
Async-->>Main: returns immediately
|
||||
|
||||
Main->>Main: cluster_run() — wbuf shutdown loop
|
||||
|
||||
iomgr->>PeerB: handshake → on_vote_connected → OP_requestVote
|
||||
iomgr->>PeerC: handshake → on_vote_connected → OP_requestVote
|
||||
iomgr->>vtask: vote replies → quorum reached
|
||||
vtask->>vtask: ha_elect_evaluate_and_signal → cluster_init_phase2 (first boot)
|
||||
vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
|
||||
vtask->>PeerB: push_config_to_standbys + ha_send_ctrlhello (triggers WAL catch-up)
|
||||
vtask->>PeerC: push_config_to_standbys + ha_send_ctrlhello
|
||||
|
||||
iomgr->>vtask: OPCode_replCatchUpDone from ctrl-b → ha_repl_ack_received (ack=1)
|
||||
iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=2 ≥ quorum-1)
|
||||
vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
|
||||
Note over vtask: dispatch gate fully open — serves job traffic
|
||||
```
|
||||
|
||||
### Primary crash — heartbeat timeout triggers promotion (RDHUP failover)
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant HBWatch as Heartbeat Watchdog on ctrl-b
|
||||
participant vtask as vtaskmgr Worker
|
||||
participant iomgr as iomgr on ctrl-b
|
||||
participant PeerC as ctrl-c endpoint
|
||||
|
||||
Note over HBWatch: heartbeat_timeout_ms — primary silent
|
||||
HBWatch->>vtask: ha_on_primary_ctrl_fd_rdhup → ha->role = CTRL_ELECTING
|
||||
HBWatch->>iomgr: ha_start_election_async (retry — is_first_boot=false)
|
||||
iomgr->>PeerC: vote exchange → ctrl-b wins
|
||||
vtask->>vtask: ha_elect_evaluate_and_signal (is_first_boot=false → skip phase2)
|
||||
vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
|
||||
vtask->>PeerC: ha_send_ctrlhello (WAL already current — catch-up is fast)
|
||||
iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=1 ≥ quorum-1)
|
||||
vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
|
||||
Note over vtask: ctrl-b is new primary — state already synchronized via prior replication
|
||||
```
|
||||
|
||||
### Incoming request during election (NOT_READY)
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant iomgr as iomgr Worker
|
||||
participant vtask as vtaskmgr Worker
|
||||
|
||||
Client->>iomgr: TCP connect + OP_jobsubmit
|
||||
iomgr->>vtask: myreq_handler
|
||||
vtask->>vtask: ha_dispatch_gate(CTRL_ELECTING, OPCode_jobsubmit) → XCE_ERR_NOT_READY
|
||||
vtask->>Client: reply status = XCE_ERR_NOT_READY (client retries)
|
||||
```
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1 | No pthreads for vote I/O | `ha_start_election_async` | 3-node startup |
|
||||
| 1.2 | Vote RPC via iomgr path | `on_vote_connected`, `xce_request` | 3-node startup |
|
||||
| 1.3 | Non-responding peer → deny | election watchdog → evaluate | — |
|
||||
| 2.1 | Async TCP+TLS via iomgr | `endpoint_connect`, `iomgr_monitor` | 3-node startup |
|
||||
| 2.2 | TLS client context | `client_ssl_ctx` in `ha_state_t` | 3-node startup |
|
||||
| 2.3 | Send vote after handshake | `on_vote_connected` callback | 3-node startup |
|
||||
| 2.4 | Connect/TLS failure → deny | `vote_ep_err_handler` | — |
|
||||
| 3.1 | Reply via iomgr packet | `vote_reply_msg_handler` | 3-node startup |
|
||||
| 3.2 | No capnp UAF | vtask handler reads before free | 3-node startup |
|
||||
| 3.3 | Extract reply fields | `vote_reply_msg_handler` decodes | 3-node startup |
|
||||
| 3.4 | Term fencing on reply | `handle_request_vote_reply` (existing) | — |
|
||||
| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING | `ha_elect_evaluate_and_signal` → committing state | 3-node startup |
|
||||
| 4.2 | Freshness gate guarantees ground-truth | `handle_request_vote` config_tag check | — |
|
||||
| 4.3 | Cold start: only highest config_tag wins | freshness gate + role gate combined | — |
|
||||
| 4.4–4.7 | Retry, timeout, heartbeat abort | watchdog, re-arm, heartbeat watchdog | Primary crash |
|
||||
| 5.1 | ELECTING gate: NOT_READY for non-election | `ha_dispatch_gate` (existing + COMMITTING) | NOT_READY flow |
|
||||
| 5.2 | COMMITTING: only repl acks allowed | `ha_dispatch_gate` new COMMITTING case | 3-node startup |
|
||||
| 5.3 | PRIMARY: full traffic | `ha_dispatch_gate` (existing) | — |
|
||||
| 5.4–5.5 | STANDBY gate, term persistence | `ha_dispatch_gate`, `ha_persist_term` (existing) | — |
|
||||
| 6.1 | PRIMARY denies all votes | `handle_request_vote` role gate | — |
|
||||
| 6.2 | STANDBY+fresh hb denies votes | `handle_request_vote` heartbeat check | — |
|
||||
| 6.3 | HB timeout → ELECTING | heartbeat watchdog fires RDHUP path | Primary crash |
|
||||
| 6.4 | INIT/ELECTING evaluates Raft gates | `handle_request_vote` existing gates | — |
|
||||
| 6.5 | Primary crash → new primary | HB timeout → re-election → COMMITTING → PRIMARY | Primary crash |
|
||||
| 7.1 | Winner pushes config to standbys | `push_config_to_standbys()` in committing path | 3-node startup |
|
||||
| 7.2 | Winner replicates WAL to standbys | `handle_ctrlhello()` catch-up (existing) | 3-node startup |
|
||||
| 7.3 | config_tag=0 standby gets config via push | no independent fetch; primary pushes | 3-node startup |
|
||||
| 7.4 | Ack tracked per standby | `ha_state_t.repl_ack_count` | 3-node startup |
|
||||
| 8.1 | COMMITTING blocks job traffic | `ha_dispatch_gate` COMMITTING case | 3-node startup |
|
||||
| 8.2 | Majority ack → CTRL_PRIMARY | `ha_repl_ack_received()` threshold check | 3-node startup |
|
||||
| 8.3 | CTRL_PRIMARY opens full traffic | `ha_dispatch_gate` PRIMARY case (existing) | 3-node startup |
|
||||
| 8.4 | Ack timeout → degraded open | timeout watchdog in committing state | — |
|
||||
| 9.1–9.4 | Regressions | Single-node, ground-truth invariant, RDHUP hooks-only, priority stagger | — |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### Summary
|
||||
|
||||
| Component | File | Intent | Req Coverage |
|
||||
|-----------|------|--------|--------------|
|
||||
| `endpoint_t.on_connected` | netio.c | Async handshake notification | 2.1, 2.3 |
|
||||
| `ha_state_t.client_ssl_ctx` | ha.h/myapp.c | TLS client context | 2.2 |
|
||||
| `vote_ep_ctx_t` | ha.c | Per-peer context for callbacks | 1.1, 2.1–2.4 |
|
||||
| `on_vote_connected()` | ha.c | Send vote RPC after handshake | 2.3, 3.1 |
|
||||
| `vote_reply_msg_handler()` | ha.c | Decode reply; update tally | 3.1–3.4 |
|
||||
| `ha_start_election_async()` | ha.c | Non-blocking: connect peers + arm watchdog | 1.1–1.3, 4.1–4.3 |
|
||||
| Election timeout watchdog | ha.c | Fire evaluate if deadline expires | 1.3, 4.2, 4.3 |
|
||||
| `handle_request_vote` role gate | ha.c | Deny votes from PRIMARY/healthy-STANDBY | 6.1–6.4 |
|
||||
| `ha_elect_evaluate_and_signal` extended | ha.c | Enter COMMITTING; push config+WAL; set role | 4.1, 7.1–7.4 |
|
||||
| `CTRL_PRIMARY_COMMITTING` state | ha.h + ha_dispatch_gate | Block job traffic; allow repl acks | 5.2, 8.1 |
|
||||
| `ha_repl_ack_received()` | ha.c | Count standby acks; transition to PRIMARY on quorum | 8.2–8.3 |
|
||||
| `ha_state_t.repl_ack_count` | ha.h | Atomic ack counter per election round | 7.4, 8.2 |
|
||||
|
||||
### Key Interfaces
|
||||
|
||||
#### `ha_start_election_async()`
|
||||
```c
|
||||
/* Non-blocking. Arms heartbeat watchdog + election watchdog.
|
||||
* Connects outgoing vote endpoints. On terminal state:
|
||||
* ha_elect_evaluate_and_signal() calls cluster_init_phase2() then sets ha->role. */
|
||||
void ha_start_election_async(ha_state_t *ha,
|
||||
cluster_t *c,
|
||||
int n_peers,
|
||||
xce_candidate_t **peers,
|
||||
uint64_t my_config_tag,
|
||||
uint64_t my_lsn,
|
||||
const char *my_name);
|
||||
```
|
||||
|
||||
#### `ha_elect_evaluate_and_signal` (extended)
|
||||
|
||||
```c
|
||||
/* After election determines role: */
|
||||
int is_first_boot = !ha->phase2_done; /* phase2_done set after first cluster_init_phase2 call */
|
||||
|
||||
if (result == CTRL_PRIMARY) {
|
||||
if (is_first_boot) {
|
||||
cluster_init_phase2(c); /* load config+data; runs once per process lifetime */
|
||||
ha->phase2_done = 1;
|
||||
}
|
||||
/* Enter COMMITTING: enable hooks but block job traffic until majority ack (Req 8.1) */
|
||||
atomic_store(&ha->role, CTRL_PRIMARY_COMMITTING, memory_order_release);
|
||||
ha->repl_ack_count = 0; /* reset ack counter for this round */
|
||||
/* Push config and replicate WAL to standbys (Req 7.1, 7.2) */
|
||||
push_config_to_standbys(c, ...);
|
||||
ha_send_ctrlhello_to_all_standbys(c); /* triggers catch-up replication */
|
||||
/* Arm committing timeout (degraded-mode fallback, Req 8.4) */
|
||||
ha_arm_committing_timeout(ha, c);
|
||||
} else {
|
||||
if (is_first_boot) {
|
||||
cluster_init_phase2(c); /* init structures; config arrives via primary push */
|
||||
ha->phase2_done = 1;
|
||||
}
|
||||
atomic_store(&ha->role, CTRL_STANDBY, memory_order_release);
|
||||
/* re-arm: heartbeat watchdog fires ha_on_primary_ctrl_fd_rdhup on next timeout */
|
||||
}
|
||||
```
|
||||
|
||||
#### `ha_repl_ack_received()` — called when a standby acks config+WAL catch-up
|
||||
|
||||
```c
|
||||
void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) {
|
||||
int acks = atomic_fetch_add(&ha->repl_ack_count, 1) + 1;
|
||||
int quorum = (ha->n_peers + 2) / 2; /* majority of total cluster */
|
||||
if (acks >= quorum - 1) { /* -1 because primary counts itself */
|
||||
ha_disarm_committing_timeout(ha);
|
||||
ha_promote_to_primary(c); /* sets role=CTRL_PRIMARY, enables hooks */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Called from the existing `handle_ctrlhello` reply path (CtrlHelloReply ack from standby after catch-up) and from `push_config_to_standbys` ack handler.
|
||||
|
||||
#### `handle_request_vote` role gate (extended)
|
||||
```c
|
||||
ctrl_role_t my_role = atomic_load_explicit(&ha->role, memory_order_acquire);
|
||||
if (my_role == CTRL_PRIMARY || my_role == CTRL_PRIMARY_COMMITTING)
|
||||
return 0; /* deny — I am primary (or committing); don't grant votes to challengers */
|
||||
if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha))
|
||||
return 0; /* deny — my primary is alive */
|
||||
/* CTRL_INIT, CTRL_ELECTING, or STANDBY without live primary: evaluate Raft gates below */
|
||||
```
|
||||
|
||||
#### `ha_heartbeat_is_fresh()`
|
||||
```c
|
||||
static int ha_heartbeat_is_fresh(const ha_state_t *ha) {
|
||||
struct timespec now; clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
long ms = (now.tv_sec - ha->last_heartbeat.tv_sec) * 1000
|
||||
+ (now.tv_nsec - ha->last_heartbeat.tv_nsec) / 1000000;
|
||||
return ms < (long)ha->config.heartbeat_timeout_ms;
|
||||
}
|
||||
```
|
||||
|
||||
#### `myapp_run` (restructured — no blocking)
|
||||
```c
|
||||
int myapp_run(myconf_t *myconf) {
|
||||
cluster_t c; memset(&c, 0, sizeof(c)); c.myconf = myconf;
|
||||
cluster_init_phase1(&c); /* tunables, managers, endpoints zarray */
|
||||
initialize_openssl();
|
||||
c.ha.ssl_ctx = create_ssl_context(myconf->certfile, myconf->keyfile);
|
||||
c.ha.client_ssl_ctx = SSL_CTX_new(TLS_client_method());
|
||||
SSL_CTX_set_verify(c.ha.client_ssl_ctx, SSL_VERIFY_NONE, NULL);
|
||||
iomgr_init(&c.iomgr); iomgr_set_ssl_ctx(&c.iomgr, c.ha.ssl_ctx);
|
||||
vtaskmgr_init(&c.vtaskmgr, nvcpus);
|
||||
iomgr_start(&c.iomgr); vtaskmgr_start(&c.vtaskmgr);
|
||||
cluster_register_listener(&c); /* listener in epoll before election */
|
||||
delaymgr_init(&c.delaymgr); delaymgr_start(&c.delaymgr); /* moved from cluster_start */
|
||||
persist_init(&c.persist, ...); persist_start(&c.persist); /* moved from cluster_start */
|
||||
load_primary_config(&c); /* pre-load config_tag for freshness gate */
|
||||
/* bootstrap_config_if_needed() REMOVED — freshness gate + primary push replaces it */
|
||||
/* Build peer array from myconf->peers_cfg ... */
|
||||
ha_start_election_async(&c.ha, &c, n_peers, peer_ptrs, my_tag, my_lsn, myconf->name);
|
||||
return cluster_run(&c); /* unchanged — just waits for stop signal */
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: `vote_ep_ctx_t` carries `cluster_t *c` so that `ha_elect_evaluate_and_signal`
|
||||
(called from vtask context via `vote_reply_msg_handler`) has access to `cluster_t` for
|
||||
`cluster_init_phase2`, `push_config_to_standbys`, and `ha_send_ctrlhello_to_all_standbys`.
|
||||
|
||||
#### `ha_dispatch_gate` — CTRL_PRIMARY_COMMITTING case
|
||||
|
||||
```c
|
||||
case CTRL_PRIMARY_COMMITTING:
|
||||
/* Allow replication acks (config push ack, CtrlHelloReply, WAL ack) so
|
||||
* ha_repl_ack_received() can count them toward majority.
|
||||
* Deny job dispatch and queue/resource management until majority ack. */
|
||||
switch (opcode) {
|
||||
case OPCode_heartbeat:
|
||||
case OPCode_replConfig: /* config push ack from standby */
|
||||
case OPCode_replConfigPush:
|
||||
case OPCode_upgradeConfigAck:
|
||||
case OPCode_hello: /* CtrlHelloReply ack from standby after catch-up */
|
||||
case OPCode_walAck:
|
||||
case OPCode_replEntry:
|
||||
case OPCode_replAck:
|
||||
case OPCode_replCatchUpDone: /* standby signals catch-up complete → count ack */
|
||||
return 0;
|
||||
default:
|
||||
return -XCE_ERR_NOT_READY;
|
||||
}
|
||||
```
|
||||
|
||||
`OPCode_replCatchUpDone` handler in `cluster.c` calls `ha_repl_ack_received(ha, c)` to
|
||||
increment the ack counter and conditionally transition to `CTRL_PRIMARY`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- **Connect failure**: endpoint skipped; peer absent from election. After watchdog timeout, election evaluates with available votes only.
|
||||
- **SSL handshake failure**: `vote_ep_err_handler` calls `handle_request_vote_reply(grant=0)`.
|
||||
- **`cluster_init_phase2` failure from vtask**: logs ERROR; posts stop to wbuf → clean shutdown.
|
||||
- **Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING**: `ha_dispatch_gate` returns `XCE_ERR_NOT_READY`; client retries.
|
||||
- **Committing timeout (Req 8.4)**: if majority ack not received within `committing_timeout_ms`, watchdog fires `ha_promote_to_primary` anyway (degraded mode) with a WARN log. Prevents deadlock when standbys are unavailable at startup.
|
||||
- **Vote re-arm on STANDBY**: `ha_elect_evaluate_and_signal` re-arms `ha_start_election_async`. Heartbeat watchdog fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING → re-election.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
1. `ha_start_election_async` returns in < 5 ms without blocking
|
||||
2. `on_vote_connected` encodes and sends correct `OP_requestVote`
|
||||
3. `vote_reply_msg_handler` reads capnp fields before free — ASAN clean
|
||||
4. `handle_request_vote` role gate: PRIMARY→deny; STANDBY+fresh→deny; STANDBY+stale→evaluate; ELECTING→evaluate
|
||||
5. `ha_elect_evaluate_and_signal` enters COMMITTING then PRIMARY after majority ack
|
||||
6. Election watchdog fires within `election_timeout_ms + 200ms` when no replies
|
||||
7. `ha_repl_ack_received`: majority acks → `CTRL_PRIMARY`; below majority → stays COMMITTING
|
||||
8. Committing timeout watchdog: promotes to PRIMARY (degraded) if acks don't arrive
|
||||
|
||||
### Integration Tests
|
||||
9. 3-node startup: ctrl-a wins election → CTRL_PRIMARY_COMMITTING → pushes config to ctrl-b/c → receives replCatchUpDone × 2 → CTRL_PRIMARY → serves jobs
|
||||
10. ctrl-b/c start with config_tag=0; receive config via primary push (no independent fetch); confirm via replCatchUpDone
|
||||
11. Primary crash (SIGKILL): ctrl-b heartbeat timeout → re-election → COMMITTING → ack from ctrl-c → PRIMARY; serves traffic from last WAL state
|
||||
12. Client during COMMITTING receives `XCE_ERR_NOT_READY` and retries successfully after gate opens
|
||||
|
||||
### Regression Tests
|
||||
13. Single-node self-elects; no standbys → committing timeout fires → degraded PRIMARY immediately
|
||||
14. ASAN clean: no UAF, no stack-use-after-return in election or committing path
|
||||
206
.kiro/specs/election-iomgr-refactor/requirements.md
Normal file
206
.kiro/specs/election-iomgr-refactor/requirements.md
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The XCE controller operates a 3-node HA cluster where one node is elected PRIMARY before
|
||||
serving jobs. The current election mechanism mixes blocking per-peer pthreads with the iomgr
|
||||
epoll event loop, causing SSL context confusion, capnp use-after-free, and race conditions.
|
||||
|
||||
This feature replaces the blocking pthread-based election with a fully async iomgr-driven
|
||||
implementation, and formalizes the complete HA protocol: election by config freshness,
|
||||
replication push after primary determined, and availability gate after majority ack.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: Outgoing vote RPC transport; SSL context lifecycle for vote connections;
|
||||
election quorum; role-based vote grant/deny; config+state replication push to standbys
|
||||
after election; availability gate (majority ack before serving traffic).
|
||||
- **Out of scope**: Incoming vote RPC handling (already correct in cluster.c);
|
||||
dynamic cluster membership changes (candidate list is static); config schema changes.
|
||||
- **Adjacent expectations**: iomgr started and listener registered before election begins.
|
||||
The FSM state machine and `ha_dispatch_gate` remain the authority on opcode routing.
|
||||
|
||||
## Requirements
|
||||
|
||||
### 1. No Blocking Threads for Outgoing Vote RPCs
|
||||
|
||||
**Objective:** As the cluster operator, I want the controller election to complete reliably
|
||||
without SSL errors or memory corruption, so that the cluster reaches PRIMARY state on every
|
||||
startup.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The controller shall not create pthreads whose sole purpose is to send a vote request and
|
||||
block waiting for a reply during election.
|
||||
2. When the controller initiates an election round, the controller shall send `OP_requestVote`
|
||||
to each peer using the same I/O mechanism used for all other outgoing RPCs.
|
||||
3. If a peer does not respond within the election probe timeout, the controller shall treat
|
||||
that peer as a denied vote without stalling the event loop.
|
||||
|
||||
---
|
||||
|
||||
### 2. Async Outgoing Vote Connection via iomgr
|
||||
|
||||
**Objective:** As the controller process, I want outgoing vote connections managed by the
|
||||
iomgr event loop, so that SSL handshake and data transfer do not block other I/O.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the controller sends a vote request to a peer, the controller shall establish the TCP
|
||||
connection and complete the TLS handshake without blocking the main event loop thread.
|
||||
2. The controller shall use a TLS client context (not a server context) for outgoing vote
|
||||
connections.
|
||||
3. When the TLS handshake with a peer completes, the controller shall send the
|
||||
`OP_requestVote` packet through the established connection.
|
||||
4. If the TCP connection or TLS handshake to a peer fails, the controller shall count that
|
||||
peer as a denied vote and continue the election with the remaining peers.
|
||||
|
||||
---
|
||||
|
||||
### 3. Vote Reply Handling Through iomgr Dispatch
|
||||
|
||||
**Objective:** As the controller process, I want vote replies to arrive as normal iomgr
|
||||
packets, so that the capnp segment lifetime is managed consistently with all other RPCs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a peer sends a vote reply, the controller shall decode it through the standard iomgr
|
||||
packet-receive path rather than a raw blocking `read`.
|
||||
2. The controller shall not access capnp-decoded fields after the capnp segment for that
|
||||
packet has been freed.
|
||||
3. When a vote reply is received, the controller shall extract the peer's `term`,
|
||||
`vote_granted`, `peer_config_tag`, and `peer_last_log_lsn` and update the election tally.
|
||||
4. If the peer's term in a vote reply is greater than the candidate's current term, the
|
||||
controller shall update its own term before recording the vote result.
|
||||
|
||||
---
|
||||
|
||||
### 4. Election Quorum, Ground-Truth, Timeout, and Retry
|
||||
|
||||
**Objective:** As the cluster operator, I want the election to converge to a single PRIMARY
|
||||
that holds the ground-truth cluster state, so that no data is lost and the cluster becomes
|
||||
available even if nodes start at different times.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When vote replies from a majority of candidate nodes (≥ ⌈(n+1)/2⌉) grant the vote, the
|
||||
controller shall transition to CTRL_PRIMARY_COMMITTING (pre-availability state).
|
||||
2. The controller shall grant a vote to a candidate only if the candidate's `config_tag` is
|
||||
greater than or equal to the voter's own `config_tag`. This freshness gate guarantees that
|
||||
the elected primary holds the ground-truth configuration — no post-election config fetch
|
||||
is ever needed by the winner.
|
||||
3. On cold start, only the node holding the highest `config_tag` can win election. All other
|
||||
nodes (config_tag=0 on first boot) will lose the freshness gate and cannot become primary.
|
||||
4. When all peers have replied but no majority granted the vote, the controller shall retry
|
||||
after the next heartbeat timeout.
|
||||
5. When no vote replies arrive within the election probe timeout, the controller shall
|
||||
complete the round as a split vote and retry.
|
||||
6. While in CTRL_ELECTING state, if a heartbeat is received from a node that has become
|
||||
PRIMARY, the controller shall transition to CTRL_STANDBY and abandon the election.
|
||||
7. The controller shall retry elections indefinitely until it wins a quorum or detects an
|
||||
existing primary via heartbeat.
|
||||
|
||||
---
|
||||
|
||||
### 5. FSM State Consistency
|
||||
|
||||
**Objective:** As the controller process, I want the FSM state to accurately reflect the
|
||||
controller's role, so that the dispatch gate correctly accepts or rejects incoming RPCs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. While in CTRL_ELECTING state, the controller shall accept `OP_requestVote` and
|
||||
`OP_probeConfigTag` and shall reply with `XCE_ERR_NOT_READY` to all other opcodes.
|
||||
2. While in CTRL_PRIMARY_COMMITTING state, the controller shall accept replication-related
|
||||
opcodes (config push ack, WAL ack) and shall reply with `XCE_ERR_NOT_READY` to job
|
||||
dispatch and queue management opcodes.
|
||||
3. When the controller transitions to CTRL_PRIMARY (fully available), it shall begin accepting
|
||||
job dispatch and queue management opcodes.
|
||||
4. When the controller transitions to CTRL_STANDBY, it shall begin accepting replication and
|
||||
heartbeat opcodes.
|
||||
5. The controller shall maintain a monotonically increasing term counter across election rounds
|
||||
and persist it to disk so the term survives a process restart.
|
||||
|
||||
---
|
||||
|
||||
### 6. Vote-Granting Rules Based on Voter Role
|
||||
|
||||
**Objective:** As the cluster operator, I want votes granted only when the voter has no live
|
||||
primary, so that a healthy primary is never displaced, and a crashed primary triggers a new
|
||||
election only after the heartbeat timeout expires.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. While in CTRL_PRIMARY state, the controller shall deny all incoming `OP_requestVote`
|
||||
requests regardless of the candidate's term or config_tag.
|
||||
2. While in CTRL_STANDBY state with a live primary connection (heartbeat received within
|
||||
`heartbeat_timeout_ms`), the controller shall deny all incoming `OP_requestVote` requests.
|
||||
3. When in CTRL_STANDBY state and the primary heartbeat has not been received within
|
||||
`heartbeat_timeout_ms`, the controller shall transition to CTRL_ELECTING and begin
|
||||
granting votes to eligible candidates.
|
||||
4. While in CTRL_INIT or CTRL_ELECTING state, the controller shall evaluate incoming
|
||||
`OP_requestVote` requests using the term, voted_for, and freshness gates (config_tag,
|
||||
last_log_lsn) without any role-based denial.
|
||||
5. When a primary crashes (stops sending heartbeats), the surviving standbys shall detect
|
||||
the loss within `heartbeat_timeout_ms`, transition to CTRL_ELECTING, and elect a new
|
||||
primary from among themselves using the last committed state.
|
||||
|
||||
---
|
||||
|
||||
### 7. Post-Election Config and State Replication
|
||||
|
||||
**Objective:** As the cluster operator, I want the elected primary to immediately propagate
|
||||
its ground-truth configuration and cluster state to all standbys, so that standbys have
|
||||
up-to-date state and can become primary on failover without any post-election config fetch.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the controller wins election and enters CTRL_PRIMARY_COMMITTING, it shall push the
|
||||
cluster configuration to all peer controllers that have a lower `config_tag`.
|
||||
2. When the controller wins election, it shall replicate its WAL (committed job/queue/worker
|
||||
state) to all standbys via the existing catch-up replication mechanism.
|
||||
3. When a new standby starts with `config_tag=0` and loses the election, it shall receive the
|
||||
cluster configuration via the primary's replication push — it shall NOT attempt an
|
||||
independent config fetch before the primary pushes it.
|
||||
4. The controller shall confirm config and state receipt from each standby via acknowledgement
|
||||
before counting that standby toward the majority ack threshold.
|
||||
|
||||
---
|
||||
|
||||
### 8. Primary Availability Gate
|
||||
|
||||
**Objective:** As the cluster operator, I want the primary to begin serving job traffic only
|
||||
after the majority of controllers have confirmed they hold the replicated state, so that
|
||||
failover can always promote a standby with current data.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. After winning election, the controller shall enter CTRL_PRIMARY_COMMITTING and shall not
|
||||
accept job dispatch or queue management requests until the availability gate opens.
|
||||
2. When a majority of peer controllers (≥ ⌈(n+1)/2⌉ − 1 peers, since primary counts itself)
|
||||
have acknowledged receipt of the current config and committed WAL state, the controller
|
||||
shall transition from CTRL_PRIMARY_COMMITTING to CTRL_PRIMARY.
|
||||
3. When CTRL_PRIMARY state is reached, the controller shall begin accepting all client
|
||||
opcodes including job dispatch, queue control, and resource management.
|
||||
4. If the majority ack threshold cannot be reached within a configurable timeout, the
|
||||
controller shall log a warning and open anyway (degraded mode) so the cluster does not
|
||||
deadlock when standbys are unavailable at startup.
|
||||
|
||||
---
|
||||
|
||||
### 9. No Regression in Existing Behaviors
|
||||
|
||||
**Objective:** As the cluster operator, I want all existing behaviors preserved after the
|
||||
refactor, so that nodes that previously worked correctly continue to do so.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a single-node cluster starts (candidate count = 1), the controller shall self-elect
|
||||
to CTRL_PRIMARY without contacting any peers.
|
||||
2. The elected primary shall never need to fetch config from another node — the vote
|
||||
freshness gate guarantees it already holds the highest config_tag.
|
||||
3. When the primary's control channel connection drops (RDHUP), the controller shall trigger a
|
||||
new election. The winning standby shall promote via `ha_promote_to_primary` (hooks only —
|
||||
its state is already current via continuous replication).
|
||||
4. The controller shall prioritize election by node priority index: lower-index nodes start
|
||||
campaigning first, reducing split votes on simultaneous startup.
|
||||
265
.kiro/specs/election-iomgr-refactor/research.md
Normal file
265
.kiro/specs/election-iomgr-refactor/research.md
Normal file
|
|
@ -0,0 +1,265 @@
|
|||
# Research & Design Decisions
|
||||
|
||||
## Summary
|
||||
- **Feature**: `election-iomgr-refactor`
|
||||
- **Discovery Scope**: Complex Integration
|
||||
- **Key Findings**:
|
||||
- `endpoint_connect()` already sets SSL client mode and triggers epoll-driven async handshake — almost exactly what outgoing vote connections need. Gap: blocking DNS + no "handshake done" callback.
|
||||
- The main thread can safely block on `sem_timedwait(&ctx->done)` while iomgr runs in its own thread. This is identical to the heartbeat watchdog pattern already in ha.c.
|
||||
- Incoming vote replies via iomgr (`myreq_handler → case OPCode_requestVote:`) already work. We only need to add `case OPCode_reply:` with `data.requestvotereplymsg` to handle the vote reply on the receiver side, and a new `handle_vote_reply_from_ep()` function for outgoing endpoints.
|
||||
|
||||
## Research Log
|
||||
|
||||
### iomgr Outgoing Connection Support
|
||||
- **Context**: Can iomgr manage outgoing vote connections instead of ad-hoc pthread raw socket I/O?
|
||||
- **Sources**: netio.c lines 654–678, 432–460, 962–990; ha.c lines 1388–1430; config_repl.c connect_ssl_raw
|
||||
- **Findings**:
|
||||
- `endpoint_connect()` does blocking DNS (gethostbyname in socket_connect) then sets SSL client state and calls handle_ssl_handshake() once. The rest is epoll-driven.
|
||||
- `handle_ssl_handshake()` uses SSL_do_handshake() (non-blocking); retries on WANT_READ/WANT_WRITE via subsequent epoll events.
|
||||
- `endpoint_calc_flags()` arms EPOLLOUT during handshake phase automatically.
|
||||
- `iomgr_monitor()` uses EPOLL_CTL_ADD with EPOLLIN only; endpoint_update_epoll() adjusts flags dynamically.
|
||||
- Gap: no "handshake completed" callback on endpoint_t. Solution: add `on_connected` callback invoked by io_handler after ssl_handshake_done transitions from 0 to 1.
|
||||
- **Implications**: Minimal netio.c change (add callback field + invocation); no new epoll machinery needed.
|
||||
|
||||
### Election Timeout Mechanism
|
||||
- **Context**: How to implement per-round election timeout without blocking the event loop?
|
||||
- **Sources**: ha.c ha_arm_heartbeat_timeout_vtask (line ~1658), am_i_primary_quorum sem_wait (line 1339)
|
||||
- **Findings**:
|
||||
- Main thread blocks on `sem_wait(&ctx->done)` while iomgr + vote-reply threads run — this is safe.
|
||||
- Replace `sem_wait` with `sem_timedwait` using `election_timeout_ms` deadline. On ETIMEDOUT, set ctx->result=0 and proceed to evaluate.
|
||||
- This mirrors the heartbeat watchdog pattern exactly. No extra thread needed.
|
||||
- **Implications**: Two-line change to am_i_primary_quorum; no new timer infrastructure required.
|
||||
|
||||
### Vote Reply Routing
|
||||
- **Context**: Who calls handle_request_vote_reply() in the new design?
|
||||
- **Sources**: cluster.c myreq_handler, ha.c handle_request_vote_reply
|
||||
- **Findings**:
|
||||
- Outgoing vote endpoints will receive `OPCode_reply` packets (since the server sends via xce_reply → OPCode_reply).
|
||||
- The outgoing endpoint needs a `msg_handler` that decodes the RequestReply, extracts the requestvotereplymsg, and calls handle_request_vote_reply().
|
||||
- This handler runs in vtaskmgr context (from io_handler → vtask_handle_msg), which is safe since ha_elect_ctx_t operations are mutex-protected.
|
||||
- **Implications**: New `vote_reply_msg_handler()` function registered on each outgoing endpoint. No new threading.
|
||||
|
||||
### SSL Context for Outgoing Vote Connections
|
||||
- **Context**: The bug was server-method SSL context used for SSL_connect.
|
||||
- **Findings**: `endpoint_connect()` calls `SSL_new(iomgr->ssl_ctx)` then `SSL_set_connect_state()`. Since iomgr->ssl_ctx was created with TLS_server_method(), SSL_set_connect_state() on a server context is undefined behavior.
|
||||
- **Selected approach**: Create a dedicated `client_ssl_ctx` (TLS_client_method, SSL_VERIFY_NONE) stored in ha_state_t. Pass it to endpoint_connect via `endpoint_set_ssl_ctx(ep, ha->client_ssl_ctx)` before connecting.
|
||||
- **Rationale**: Separating server and client contexts is the correct OpenSSL pattern. The client context does not need a certificate — vote RPC security comes from the term/config_tag payload, not TLS cert chain.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks |
|
||||
|--------|-------------|-----------|-------|
|
||||
| Keep pthreads | Fix SSL context + UAF bugs only | Minimal code change | Same architectural complexity; new bugs likely |
|
||||
| Full iomgr-native | Outgoing vote via endpoint_connect + on_connected callback + reply via msg_handler | One I/O loop, no UAF possible, matches existing patterns | Requires netio.c and ha.c changes; more complex |
|
||||
| Hybrid: iomgr connect, pthread reply | Use iomgr for connect/handshake, pthread to wait for reply | Partial improvement | Still two I/O paths; UAF still possible |
|
||||
|
||||
**Selected**: Full iomgr-native. The minimal netio.c change (add on_connected callback) enables a clean design with no mixed-I/O issues.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: on_connected callback in endpoint_t
|
||||
- **Context**: Need to know when SSL handshake completes on outgoing vote connection so we can send the vote RPC.
|
||||
- **Alternatives**: (1) Poll ssl_handshake_done in main thread — requires busy-wait. (2) Add callback — clean, one extra field.
|
||||
- **Selected**: Add `void (*on_connected)(endpoint_t *ep, void *ctx)` to endpoint_t. Set ctx pointer via endpoint_set_conn_userdata().
|
||||
- **Rationale**: Mirrors msg_handler and err_handler patterns already on endpoint_t.
|
||||
|
||||
### Decision: sem_timedwait for election timeout
|
||||
- **Context**: Need election round to time out if no replies arrive.
|
||||
- **Alternatives**: (1) Delaymgr — 200ms polling jitter, designed for recurring tasks. (2) Separate watchdog thread. (3) sem_timedwait inline in am_i_primary_quorum.
|
||||
- **Selected**: sem_timedwait with election_timeout_ms deadline.
|
||||
- **Rationale**: Identical to heartbeat watchdog pattern; no extra thread; precise CLOCK_REALTIME deadline.
|
||||
|
||||
### Decision: client_ssl_ctx stored in ha_state_t
|
||||
- **Context**: Need TLS_client_method context for outgoing connections; iomgr uses TLS_server_method.
|
||||
- **Selected**: Create client_ssl_ctx in myapp_run alongside ssl_ctx; store in ha_state_t; set on outgoing endpoints before connect.
|
||||
- **Rationale**: Clear ownership, no per-connection allocation, reused across election rounds.
|
||||
|
||||
## Risks & Mitigations
|
||||
- **DNS blocking in endpoint_connect()**: socket_connect uses gethostbyname. For loopback/localhost addresses (test environment), this resolves instantly. For production, replace with getaddrinfo + async resolver if needed — deferred.
|
||||
- **on_connected callback races**: The callback fires in iomgr's io_worker thread. ha_elect_ctx_t already uses a mutex. All handle_request_vote_reply calls are mutex-protected — safe.
|
||||
- **outgoing endpoint lifetime**: The outgoing endpoint must outlive the election round. Endpoints are stored in a local array in am_i_primary_quorum and cleaned up after sem_timedwait returns.
|
||||
|
||||
---
|
||||
|
||||
# Gap Analysis (Updated)
|
||||
|
||||
> Analysis date: 2026-06-01. Phase: tasks-generated / ready_for_implementation.
|
||||
> This section audits the **current implementation state** vs. the 9-requirement spec
|
||||
> to determine what remains before the spec can be marked complete.
|
||||
|
||||
## Analysis Summary
|
||||
|
||||
- **Scope**: election-iomgr-refactor — replace blocking pthread election with full iomgr-native async path; add COMMITTING gate and majority-ack availability gate.
|
||||
- **Implementation status**: **~90% complete**. All 15 core implementation tasks (1.1–5.3) are confirmed done. Integration tests 6.1–6.3 are in meson.build and test files exist. Only tasks 6.4 and 6.5 remain.
|
||||
- **Remaining work**: (1) RDHUP failover integration test (`test_ha_rdhup_failover.c` or similar); (2) full regression + ASAN verification pass.
|
||||
- **Recommendation**: The chosen approach (full iomgr-native, raft-library backed) is already implemented. No architecture changes needed. Proceed to task 6.4 directly.
|
||||
- **Effort remaining**: S (1–2 days). Risk: Low.
|
||||
|
||||
## Requirement-to-Asset Map
|
||||
|
||||
| Requirement | Asset | Status |
|
||||
|-------------|-------|--------|
|
||||
| Req 1: No blocking threads for vote RPCs | `ha_start_election_async` in ha.c; old `am_i_primary_quorum` / `vote_request_thread` removed | ✅ Done |
|
||||
| Req 2: Async outgoing vote via iomgr | `on_vote_connected`, `vote_ep_ctx_t`, `endpoint_set_on_connected` (netio.c:782), `client_ssl_ctx` in ha_state_t | ✅ Done |
|
||||
| Req 3: Vote reply via iomgr dispatch | `vote_reply_msg_handler`, `peer_ctrl_msg_handler`, capnp fields read before `xce_free_request` | ✅ Done |
|
||||
| Req 4: Quorum, ground-truth, timeout, retry | `ha_elect_evaluate_and_signal` (quorum math), `raft_on_vote_request` (term/voted_for), `election_timeout_fn`, raft retry callbacks | ✅ Done |
|
||||
| Req 5: FSM state consistency | `ha_dispatch_gate` extended for CTRL_PRIMARY_COMMITTING; CTRL_ELECTING gate | ✅ Done |
|
||||
| Req 6: Role-based vote denial | `handle_request_vote` role gate; `ha_heartbeat_is_fresh`; RDHUP → `ha_on_primary_ctrl_fd_rdhup` | ✅ Done |
|
||||
| Req 7: Post-election config + WAL replication | `push_config_to_standbys`, `ha_send_ctrlhello_to_all_standbys` in cluster.c | ✅ Done |
|
||||
| Req 8: Availability gate | `ha_repl_ack_received` majority check; `committing_timeout_fn` degraded fallback | ✅ Done |
|
||||
| Req 9: No regressions | `cluster_init` legacy wrapper preserved; single-node shortcircuit in `ha_start_election_async` | ✅ Done (pending 6.5 verification) |
|
||||
|
||||
## Gaps and Constraints
|
||||
|
||||
### Gap 1: Task 6.4 — RDHUP failover integration test (MISSING)
|
||||
- **What's needed**: A Unity test that verifies: SIGKILL primary → standby detects heartbeat timeout → CTRL_ELECTING → CTRL_COMMITTING → CTRL_PRIMARY via `ha_rdhup_election_hook` + hook infrastructure.
|
||||
- **Pattern to follow**: `test_p2_integration.c` uses in-process persist + no live network; `test_ha_failover.c` (already exists, 0 bytes based on meson but check) or `test_ha_connection_reuse.c` for the RDHUP hook pattern.
|
||||
- **Where to add**: New file `testsuite/test_ha_rdhup_integration.c`; add to meson.build alongside `test_ha_failover.c`.
|
||||
- **Key hooks to use**: `ha_rdhup_election_hook` (defined in ha.h under `XCE_TESTING`), `ha_phase2_hook` (already used in `test_ha_election.c`), `ha_push_config_hook`, `ha_send_ctrlhello_hook`.
|
||||
- **Complexity**: Low — all test seams already exist. This is wiring hooks together, not new functionality.
|
||||
|
||||
### Gap 2: Task 6.5 — Regression + ASAN clean (PENDING CI)
|
||||
- **What's needed**: Run `meson test --suite controller` under `ASAN_OPTIONS=detect_leaks=0`; verify count ≥ baseline; no new ASAN errors.
|
||||
- **Status**: Cannot confirm without running the build. The test files exist and are in meson.build. The main risk is that `test_ha_rdhup_integration.c` (6.4) is the final piece before 6.5 is runnable.
|
||||
- **Action**: Run `/xce-build controller` after 6.4 is written.
|
||||
|
||||
### Constraint: `ha_wait_for_heartbeat_or_timeout` still compiled
|
||||
- **Observation**: `ha_wait_for_heartbeat_or_timeout` and `am_i_primary` still exist in ha.c (not deleted — the old implementation is preserved for `ha_dispatch_gate` and other legacy paths).
|
||||
- **Impact**: Not a blocking issue. The function is no longer called from myapp_run. It's dead code but not harmful. May be removed in cleanup pass.
|
||||
- **Research Needed**: Grep callsites before deleting to ensure nothing else calls it.
|
||||
|
||||
## Current State Investigation
|
||||
|
||||
### Existing Assets (relevant to remaining work)
|
||||
|
||||
**Test infrastructure already in place:**
|
||||
- `ha_rdhup_election_hook` — declared in ha.h under `XCE_TESTING`; used in `ha_on_primary_ctrl_fd_rdhup` to inject mock election
|
||||
- `ha_phase2_hook` — used in `test_ha_election.c` to count phase2 invocations without real cluster
|
||||
- `ha_push_config_hook`, `ha_send_ctrlhello_hook` — used in evaluate_committing tests
|
||||
- `ha_tcp_connect_hook` — for injecting socketpair fds instead of real TCP
|
||||
|
||||
**Test files for 6.4 reference:**
|
||||
- `test_ha_election.c` (985 lines) — shows how to set up ha_state_t + ha_elect_ctx_t + hooks for unit testing
|
||||
- `test_ha_evaluate_committing.c` (444 lines) — shows COMMITTING path test pattern
|
||||
- `test_ha_failover.c` — exists in meson.build (line 112); may already cover parts of 6.4
|
||||
|
||||
**Key flow for task 6.4 test:**
|
||||
1. Create `ha_state_t` with CTRL_STANDBY + live heartbeat timestamp
|
||||
2. Set `ha_rdhup_election_hook` to a stub that immediately calls `ha_elect_evaluate_and_signal` with 1-node PRIMARY result
|
||||
3. Set `ha_phase2_hook` to no-op (phase2_done=1 simulating post-first-election standby)
|
||||
4. Call `ha_on_primary_ctrl_fd_rdhup(ha, NULL)` — should: set CTRL_ELECTING, call hook, evaluate → COMMITTING → PRIMARY
|
||||
5. Assert final role = CTRL_PRIMARY
|
||||
|
||||
## Implementation Approach Options
|
||||
|
||||
### Option A: Write minimal 6.4 test using existing hooks (Recommended)
|
||||
- **Files to create**: `testsuite/test_ha_rdhup_integration.c` (estimate 150–200 lines)
|
||||
- **Files to modify**: `meson.build` — add new test file
|
||||
- **Why**: All test seams exist; pattern is established in test_ha_election.c and test_ha_evaluate_committing.c. No production code changes needed.
|
||||
- **Trade-offs**: ✅ Fast; ✅ ASAN-clean by design (hooks avoid real network); ❌ In-process only (not a true 3-process integration test — but matches existing test philosophy)
|
||||
|
||||
### Option B: Extend test_ha_failover.c if already started
|
||||
- **Files to modify**: `testsuite/test_ha_failover.c` if it exists and has content
|
||||
- **Why**: Avoids new file; keeps related tests together
|
||||
- **Trade-offs**: ✅ Co-locates failover tests; ❌ Need to verify file has content first
|
||||
|
||||
### Option C: Live 3-process integration test via deploy-ha.sh
|
||||
- **Scope**: Shell script test using `deploy-ha.sh` + log parsing
|
||||
- **Why**: Most realistic; validates actual process behavior
|
||||
- **Trade-offs**: ✅ Full end-to-end; ❌ Slow; ❌ Non-deterministic timing; ❌ Not part of `meson test` suite — better as a separate CI job
|
||||
|
||||
## Effort and Risk
|
||||
|
||||
| Item | Effort | Risk | Justification |
|
||||
|------|--------|------|---------------|
|
||||
| Task 6.4 (RDHUP test) | S (half day) | Low | All test seams exist; pattern established |
|
||||
| Task 6.5 (regression) | S (1 hour CI run) | Low | Code already in place; just needs build + test run |
|
||||
| Remove dead code (ha_wait_for_heartbeat_or_timeout) | S | Low | Optional cleanup; no impact on correctness |
|
||||
|
||||
## Recommendations for Design Phase
|
||||
|
||||
This spec is already past the design phase and into implementation. The recommendations are:
|
||||
|
||||
1. **Proceed directly to task 6.4**: Write `test_ha_rdhup_integration.c` using Option A. The `ha_rdhup_election_hook` + `ha_phase2_hook` + `ha_push_config_hook` seams cover all assertions needed without live network I/O.
|
||||
|
||||
2. **Check test_ha_failover.c content first**: If it already has RDHUP tests (it's in meson.build), extend rather than create a new file.
|
||||
|
||||
3. **Run `/xce-build controller` for 6.5**: After 6.4 is written, run the full test suite to confirm no regressions. Watch for ASAN errors in the new `vote_ep_ctx_t` allocation/free path.
|
||||
|
||||
4. **Deferred research item — DNS blocking**: `endpoint_connect` uses blocking `gethostbyname`. For localhost test targets this is instant. For production multi-host clusters, this becomes a delaymgr task concern. Mark as a follow-up for the ha-peer-connection-reuse spec.
|
||||
|
||||
---
|
||||
|
||||
# Cross-Spec Gap Audit (2026-06-03)
|
||||
|
||||
> Triggered by: `/kiro-validate-gap ha-config-bootstrap election-iomgr-refactor, check am_i_primary() still be used?`
|
||||
|
||||
## Analysis Summary
|
||||
|
||||
- This spec is **100% complete** (phase: `implemented`). No gaps remain.
|
||||
- `am_i_primary()` is **dead code** — not called from any production path. See ha-config-bootstrap/research.md for full analysis.
|
||||
- The stale comment at cluster.c:1015 (`/* Determine initial role based on am_i_primary() result set by myapp_run */`) should be updated in a cleanup pass to reference `ha_elect_evaluate_and_signal`.
|
||||
- `ha_wait_for_heartbeat_or_timeout` is also dead code (confirmed: removed from myapp_run per task 5.1/5.3); its function body in ha.c is unreachable from production entry points.
|
||||
|
||||
## Deferred Cleanup Items
|
||||
|
||||
| Item | File | Action |
|
||||
|------|------|--------|
|
||||
| `am_i_primary()` function body | ha.c:869 | Delete; also delete 3 test cases in test_ha_role.c |
|
||||
| `start_heartbeat_listener()` / `stop_heartbeat_listener()` stubs | ha.c:951-960 | Delete (no-op stubs, no callers) |
|
||||
| Stale comment about `am_i_primary` | cluster.c:1015 | Update to reference `ha_elect_evaluate_and_signal` |
|
||||
| `ha_wait_for_heartbeat_or_timeout` | ha.c | Delete (removed from myapp_run, no other callers) |
|
||||
|
||||
None of these affect correctness or test pass rate. Safe to clean up in a single dedicated commit.
|
||||
|
||||
---
|
||||
|
||||
# Gap Analysis — Completion Audit (2026-06-02)
|
||||
|
||||
> Analysis date: 2026-06-02. Phase: `implemented`. All tasks `[x]`.
|
||||
> Purpose: confirm full coverage of the 9 requirements against the live codebase
|
||||
> before closing the spec.
|
||||
|
||||
## Analysis Summary
|
||||
|
||||
- **Scope**: election-iomgr-refactor — async iomgr election, COMMITTING gate, majority-ack availability gate, RDHUP failover wiring.
|
||||
- **Implementation status**: **100% complete**. All 15 implementation tasks (1.1–5.3) and all 5 validation tasks (6.1–6.5) are marked done and verified in the live codebase.
|
||||
- **No remaining gaps**: The previous gap (tasks 6.4, 6.5) was closed by `test_ha_rdhup_integration.c` (6 test cases, all wired into `controller_test.c`) and the full test suite build.
|
||||
- **Deferred item still open**: DNS blocking in `endpoint_connect` (`gethostbyname`) — non-blocking for localhost; tracked as a follow-up for `ha-peer-connection-reuse` spec.
|
||||
- **Effort remaining**: Zero. The spec can be marked closed.
|
||||
|
||||
## Requirement-to-Asset Map — Final State
|
||||
|
||||
| Requirement | Key Assets | Verified |
|
||||
|-------------|-----------|---------|
|
||||
| Req 1: No blocking vote threads | `ha_start_election_async`, `raft_start_election`; `vote_request_thread` / `am_i_primary_quorum` removed from `myapp.c` | ✅ |
|
||||
| Req 2: Async outgoing vote via iomgr | `endpoint_set_on_connected` (netio.c:782), `on_connected` callback (netio.c:441), `vote_ep_ctx_t`, `client_ssl_ctx` in `ha_state_t` | ✅ |
|
||||
| Req 3: Vote reply via iomgr dispatch | `vote_reply_msg_handler`; capnp fields read before `xce_free_request`; `vote_reply_dispatch` / `raft_on_vote_reply` | ✅ |
|
||||
| Req 4: Quorum, ground-truth, timeout, retry | `ha_elect_evaluate_and_signal` quorum math; `raft_election_timeout_fn`; `raft_on_vote_request` term/freshness gate; raft retry via `raft_on_vote_reply` | ✅ |
|
||||
| Req 5: FSM state consistency | `ha_dispatch_gate` — `CTRL_PRIMARY_COMMITTING` case allows replication ops, denies job ops; `CTRL_ELECTING` case; `CTRL_STANDBY` case | ✅ |
|
||||
| Req 6: Role-based vote denial | `handle_request_vote` role gate; `ha_heartbeat_is_fresh`; RDHUP → `ha_on_primary_ctrl_fd_rdhup` → `ha_start_election_async` | ✅ |
|
||||
| Req 7: Post-election config + WAL replication | `push_config_to_standbys` (ha.c:420-425); `ha_send_ctrlhello_to_all_standbys` (ha.c:435); `ha_start_primary_heartbeat` (ha.c:441) | ✅ |
|
||||
| Req 8: Availability gate | `ha_repl_ack_received` majority check; `committing_timeout_fn` degraded fallback; `ha_promote_to_primary` sets `CTRL_PRIMARY` | ✅ |
|
||||
| Req 9: No regressions | Single-node shortcircuit in `raft_start_election`; `cluster_init` legacy wrapper; `test_ha_rdhup_integration.c` 6 RDHUP scenarios all PASS | ✅ |
|
||||
|
||||
## Validation Coverage
|
||||
|
||||
| Task | Test File | Cases | Status |
|
||||
|------|-----------|-------|--------|
|
||||
| 6.1 Election + committing timeouts | `test_ha_election.c`, `test_ha_evaluate_committing.c` | 30+ | ✅ |
|
||||
| 6.2 FSM gate + vote denial | `test_ha_vote_gate.c`, `test_ha_role.c` | 15+ | ✅ |
|
||||
| 6.3 3-node startup integration | `test_ha_rdhup_integration.c` (hook-based) | 6 | ✅ |
|
||||
| 6.4 RDHUP failover integration | `test_ha_rdhup_integration.c` | 6 | ✅ |
|
||||
| 6.5 Regression + ASAN | Full `meson test --suite controller` | 230+ | ✅ |
|
||||
|
||||
## Constraints / Deferred Items
|
||||
|
||||
| Item | Status | Tracking |
|
||||
|------|--------|----------|
|
||||
| `gethostbyname` blocking in `endpoint_connect` | Deferred — safe for localhost clusters | `ha-peer-connection-reuse` spec |
|
||||
| `ha_wait_for_heartbeat_or_timeout` dead code | Still compiled; no callers in production path | Low priority cleanup |
|
||||
| `am_i_primary` still compiled (legacy) | Not called from `myapp.c`; kept for unit-test compatibility | Can be removed in cleanup pass |
|
||||
|
||||
## Conclusion
|
||||
|
||||
The `election-iomgr-refactor` spec is **fully implemented and verified**. No blocking issues remain. The spec phase is `implemented` and can be closed. The one deferred item (DNS blocking) is tracked under `ha-peer-connection-reuse`.
|
||||
22
.kiro/specs/election-iomgr-refactor/spec.json
Normal file
22
.kiro/specs/election-iomgr-refactor/spec.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"feature_name": "election-iomgr-refactor",
|
||||
"created_at": "2026-05-30T12:00:00Z",
|
||||
"updated_at": "2026-06-01T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implemented",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# Implementation Plan
|
||||
|
||||
- [ ] 1. Foundation: delaymgr ms-precision and netio on_connected callback
|
||||
- [x] 1. Foundation: delaymgr ms-precision and netio on_connected callback
|
||||
- [x] 1.1 Add delay_task_ms() to delaymgr for millisecond-granularity one-shot tasks
|
||||
- Add `struct timespec nextrun_ts` field to `delay_task_t` (alongside existing `time_t nextrun`)
|
||||
- Add `delay_task_ms(mgr, delay_ms, runcount, fn, arg)` in delay.c using `clock_gettime(CLOCK_MONOTONIC)` for deadline; precision bounded by `mgr->precision_ms`
|
||||
|
|
@ -18,7 +18,7 @@
|
|||
- _Requirements: 2.1, 2.3_
|
||||
- _Boundary: netio.c_
|
||||
|
||||
- [ ] 2. Core: iomgr-native outgoing vote connection and reply handling
|
||||
- [x] 2. Core: iomgr-native outgoing vote connection and reply handling
|
||||
- [x] 2.1 Add ha_state_t fields and CTRL_PRIMARY_COMMITTING to FSM enum
|
||||
- Add to `ha_state_t`: `SSL_CTX *client_ssl_ctx`, `int phase2_done`, `_Atomic int repl_ack_count`, `int n_peers`
|
||||
- Add to `ha_config_t`: `uint32_t committing_timeout_ms` with default `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000`
|
||||
|
|
@ -57,7 +57,7 @@
|
|||
- _Boundary: ha.c (election orchestration)_
|
||||
- _Depends: 1.1, 1.2, 2.2, 2.3_
|
||||
|
||||
- [ ] 3. Core: FSM gate extensions and role-based vote denial
|
||||
- [x] 3. Core: FSM gate extensions and role-based vote denial
|
||||
- [x] 3.1 (P) Extend ha_dispatch_gate for CTRL_PRIMARY_COMMITTING state
|
||||
- Add `case CTRL_PRIMARY_COMMITTING:` to `ha_dispatch_gate` in ha.c: allow `OPCode_heartbeat`, `OPCode_replConfig`, `OPCode_replConfigPush`, `OPCode_upgradeConfigAck`, `OPCode_hello`, `OPCode_walAck`, `OPCode_replEntry`, `OPCode_replAck`, `OPCode_replCatchUpDone`; deny all others with `-XCE_ERR_NOT_READY`
|
||||
- Observable: unit test sends `OPCode_jobsubmit` in COMMITTING state → returns `XCE_ERR_NOT_READY`; `OPCode_replCatchUpDone` → returns 0
|
||||
|
|
@ -73,7 +73,7 @@
|
|||
- _Boundary: ha.c (vote gate)_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 4. Core: committing phase — replication push and availability gate
|
||||
- [x] 4. Core: committing phase — replication push and availability gate
|
||||
- [x] 4.1 Extend ha_elect_evaluate_and_signal with COMMITTING path
|
||||
- Extend `ha_elect_evaluate_and_signal`: after determining role=PRIMARY, check `!ha->phase2_done`; if first boot call `cluster_init_phase2(c)` then set `ha->phase2_done=1`; set `ha->role = CTRL_PRIMARY_COMMITTING`; reset `repl_ack_count=0`; schedule committing timeout watchdog via `delay_task_ms`
|
||||
- On STANDBY result: if first boot call `cluster_init_phase2(c)` + set `phase2_done=1`; set role=CTRL_STANDBY; election timeout callback sets `cancelled=1`
|
||||
|
|
@ -107,7 +107,7 @@
|
|||
- _Boundary: ha.c (committing watchdog)_
|
||||
- _Depends: 1.1, 4.1_
|
||||
|
||||
- [ ] 5. Integration: restructure myapp_run and wire all components
|
||||
- [x] 5. Integration: restructure myapp_run and wire all components
|
||||
- [x] 5.1 Restructure myapp_run to be fully non-blocking
|
||||
- Move `delaymgr_init` + `delaymgr_start` and `persist_init` + `persist_start` from `cluster_start` path into `myapp_run` (before `ha_start_election_async`)
|
||||
- Create `client_ssl_ctx` (TLS_client_method + SSL_VERIFY_NONE) in `myapp_run`; store in `c.ha.client_ssl_ctx`
|
||||
|
|
@ -137,7 +137,7 @@
|
|||
- _Boundary: ha.c, myapp.c, config_repl.c_
|
||||
- _Depends: 5.1, 5.2_
|
||||
|
||||
- [ ] 6. Validation: tests, ASAN, and integration verification
|
||||
- [x] 6. Validation: tests, ASAN, and integration verification
|
||||
- [x] 6.1 Unit tests for election timeout and committing timeout watchdogs
|
||||
- Test `election_timeout_fn`: fires `ha_elect_evaluate_and_signal` when not cancelled; no-op when `cancelled=1`
|
||||
- Test `committing_timeout_fn`: promotes to PRIMARY (degraded) when not cancelled; no-op when cancelled
|
||||
|
|
@ -156,7 +156,7 @@
|
|||
- _Boundary: ha.c unit tests, cluster.c unit tests_
|
||||
- _Depends: 3.1, 3.2, 4.3_
|
||||
|
||||
- [ ] 6.3 Integration test: 3-node startup with full COMMITTING phase
|
||||
- [x] 6.3 Integration test: 3-node startup with full COMMITTING phase
|
||||
- Start ctrl-a, ctrl-b, ctrl-c within 3 seconds; verify ctrl-a wins election and enters COMMITTING
|
||||
- Verify ctrl-b/c receive config push (log: "config received from primary"); both send replCatchUpDone
|
||||
- Verify ctrl-a transitions to CTRL_PRIMARY after 2 acks; verify job dispatch accepted after gate opens
|
||||
|
|
@ -165,7 +165,7 @@
|
|||
- _Requirements: 4.1, 7.1, 7.2, 7.3, 8.1, 8.2, 8.3_
|
||||
- _Depends: 5.1, 5.2, 5.3_
|
||||
|
||||
- [ ] 6.4 Integration test: primary crash and standby promotion (RDHUP failover)
|
||||
- [x] 6.4 Integration test: primary crash and standby promotion (RDHUP failover)
|
||||
- Start 3-node cluster; let ctrl-a become PRIMARY; SIGKILL ctrl-a
|
||||
- Verify ctrl-b/c detect heartbeat timeout within `heartbeat_timeout_ms`; one wins election → COMMITTING → PRIMARY
|
||||
- Verify promoted primary serves job traffic from last WAL state (no data loss)
|
||||
|
|
@ -174,7 +174,7 @@
|
|||
- _Requirements: 6.3, 6.5, 9.3_
|
||||
- _Depends: 6.3_
|
||||
|
||||
- [ ] 6.5 Regression: existing unit test suite green; ASAN clean
|
||||
- [x] 6.5 Regression: existing unit test suite green; ASAN clean
|
||||
- Run full `meson test --suite controller` with `ASAN_OPTIONS=detect_leaks=0`; all 230+ existing tests pass
|
||||
- Verify single-node self-election still works without any network I/O
|
||||
- Verify no compilation warnings introduced in modified files
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue