Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
293 lines
23 KiB
Markdown
293 lines
23 KiB
Markdown
# Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election
|
|
|
|
---
|
|
|
|
## Task Overview
|
|
|
|
10 requirements → 8 implementation changes (D→A→B→C→E→H→F→G) + observability woven throughout.
|
|
Dependency order: startup validation (D) unblocks vtaskmgr before election (A); listener before election (A) unblocks quorum FSM (B); quorum FSM (B) unblocks config bootstrap (C); majority-commit WAL infrastructure (E) unblocks majority-commit config (H) and persistent channel (F); libxce retry (G) is independent of all controller changes.
|
|
|
|
---
|
|
|
|
- [x] 1. Foundation: Wire protocol extensions and shared infrastructure
|
|
- [x] 1.1 Add 6 new opcodes and message structs to controller.capnp
|
|
- Append `requestVote @32`, `probeConfigTag @33`, `walAck @34`, `upgradeConfigAck @35`, `snapshotTransfer @36`, `walCatchupEntry @37` after existing `upgradeConfig @31`
|
|
- Add structs: `RequestVoteRequest`, `RequestVoteReply`, `ProbeConfigTagRequest`, `ProbeConfigTagReply`, `WalAckRequest`, `WalAckReply`, `UpgradeConfigAck`, `SnapshotTransfer`, `WalCatchupEntry`
|
|
- Extend `CtrlHeartbeat`: add `committedLsn :UInt64` and `committedConfigTag :UInt64` fields
|
|
- Extend `RequestReply`: add `redirectHost :Text` and `redirectPort :UInt16` fields for standby redirect
|
|
- Regenerate capnp bindings (`nix build .#xcecore_debug` must compile clean)
|
|
- `make test` passes; existing decode paths unaffected
|
|
- _Requirements: 2, 3, 5, 6, 7, 8_
|
|
- _Boundary: Wire Protocol (controller.capnp)_
|
|
|
|
- [x] 1.2 Add new enum values and types to ha.h
|
|
- Add `CTRL_INIT=2`, `CTRL_ELECTING=3` to `ctrl_role_t` (values above existing 0/1)
|
|
- Add `ha_vote_entry_t` struct with fields: `peer`, `my_term`, `my_tag`, `my_lsn`, `vote_granted`, `replied`, `peer_tag`, `peer_lsn`
|
|
- Add `ha_elect_ctx_t` struct with C99 flexible array `ha_vote_entry_t votes[]`; allocation comment: `calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))`
|
|
- Add 8 new `ha_state_t` fields: `voted_for_term`, `voted_for_name[XCE_MAXNAME_LEN]`, `primary_ctrl_fd` (int, -1=none), `defer_config_fetch` (bool), `elect_ctx` (_Atomic void*), `last_known_committed_lsn` (_Atomic uint64_t), `heartbeat_seen` (sem_t), `last_known_committed_config_tag` (uint64_t)
|
|
- Declare: `ha_persist_term()`, `bootstrap_config_if_needed()`, `xce_probe_config_tag()`
|
|
- `make test` passes; no new functions yet — declarations only
|
|
- _Requirements: 1, 2, 3, 5, 6, 9_
|
|
- _Boundary: Controller FSM (ha.h)_
|
|
|
|
- [x] 2. Change D: Startup validation and ASAN fix
|
|
- [x] 2.1 Write failing tests for startup validation
|
|
- In `src/controller/testsuite/test_ha_startup.c`: test even-N peer count returns `-EINVAL`; test cert missing returns `-ENOENT` with full path in message; test `--vcpus 0` returns `-EINVAL`; test `vcpus` key in config file logs WARN but does not fail
|
|
- `make test` fails (functions under test do not exist yet)
|
|
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
|
|
|
- [x] 2.2 Implement --vcpus CLI arg and remove vcpus from cluster config
|
|
- Add `--vcpus N` argp option in `main.c`; store in a new startup parameter struct passed to `myapp_run()`
|
|
- Remove `vcpus @7` field from `ClusterConfig` in `cluster.h` / `cluster.c`
|
|
- Update `vtaskmgr_init()` call site to use the CLI-provided vcpus value directly
|
|
- Log WARN when config file contains `vcpus` key (4.4)
|
|
- `make test` GREEN on vcpus tests
|
|
- _Requirements: 4.3, 4.4_
|
|
- _Boundary: Startup Validation (main.c, cluster.h, cluster.c)_
|
|
|
|
- [x] 2.3 Add peer count validation and cert pre-flight checks
|
|
- In `main.c` after argument parsing: if `n_peers % 2 == 0 && n_peers > 1` → `fprintf(stderr, ...)` + `exit(1)` with message explaining odd-N requirement
|
|
- Call `access(cert_path, R_OK)` and `access(key_path, R_OK)` in `main.c` before any SSL init; fatal exit with full path on failure
|
|
- Fix ASAN leak: add `SSL_CTX_free(ha->ssl_ctx)` in `ha_fini()`
|
|
- `make test` GREEN on all startup validation tests
|
|
- _Requirements: 4.1, 4.2_
|
|
- _Boundary: Startup Validation (main.c, ha.c)_
|
|
|
|
- [x] 3. Change A: Listener and iomgr/vtaskmgr before election
|
|
- [x] 3.1 Write failing tests for dispatch gate and FSM init state
|
|
- In `src/controller/testsuite/test_ha_startup.c` (extend): test that requests other than `OP_requestVote` and `OP_probeConfigTag` receive `XCE_ERR_NOT_READY` when role is `CTRL_INIT`; test `CTRL_STANDBY` returns `XCE_ERR_STANDBY` for primary-only ops; test `heartbeat_seen` semaphore initialized in `ha_init()`
|
|
- `make test` fails
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.4_
|
|
|
|
- [x] 3.2 Extract cluster_open_listener() from cluster_init_phase2()
|
|
- Move `bind()+listen()` + socket setup into new `cluster_open_listener(cluster_t *c)` in `cluster.c`
|
|
- `cluster_init_phase2()` calls `cluster_open_listener()` for compatibility
|
|
- `myapp.c` startup sequence calls `cluster_open_listener()` before election (after `cluster_init_phase1()`)
|
|
- `iomgr_init()` and `vtaskmgr_init(vcpus)` also moved before election
|
|
- `make test` GREEN; listener is open in startup sequence before any election probe
|
|
- _Requirements: 1.1, 1.2_
|
|
- _Boundary: Startup Sequence (cluster.c, myapp.c)_
|
|
|
|
- [x] 3.3 Implement dispatch gate for CTRL_INIT and CTRL_ELECTING states
|
|
- Initialize `ha.heartbeat_seen` semaphore in `ha_init()`; initialize `ha.primary_ctrl_fd = -1`
|
|
- Add role-check at vtaskmgr dispatch entry: `CTRL_INIT` and `CTRL_ELECTING` pass only `OP_requestVote` and `OP_probeConfigTag`; all others return `XCE_ERR_NOT_READY`
|
|
- `CTRL_STANDBY` returns `XCE_ERR_STANDBY` with `redirect_host/port` for primary-only ops
|
|
- `make test` GREEN on all dispatch gate tests
|
|
- _Requirements: 1.3, 1.4_
|
|
- _Boundary: Controller FSM (ha.c, cluster.c)_
|
|
|
|
- [x] 3.4 Implement ha_wait_for_heartbeat_or_timeout()
|
|
- `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + xorshift64_jitter()`
|
|
- Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`; receiving a heartbeat transitions to `CTRL_STANDBY` and returns
|
|
- Election timeout fires → returns without setting standby; caller proceeds to `am_i_primary_quorum()`
|
|
- Called from `myapp.c` after `bootstrap_config_if_needed()` (see task 5)
|
|
- `make test` GREEN
|
|
- _Requirements: 1.1, 1.2_
|
|
- _Boundary: Controller FSM (ha.c)_
|
|
|
|
- [x] 4. Change B: Quorum-based controller election (replaces am_i_primary)
|
|
- [x] 4.1 Write failing tests for quorum election logic
|
|
- In `src/controller/testsuite/test_ha_election.c`: test 2/3 votes → `CTRL_PRIMARY`; test only 1/3 vote → `CTRL_STANDBY`; test `voted_for` enforcement (deny second candidate in same term); test freshness gate (higher peer tag yields to peer → `CTRL_STANDBY`); test fresher candidate wins over stale peer; test split vote triggers retry with incremented term; test single-node self-elects immediately without network; test `in_catch_up=1` standby votes are denied
|
|
- `make test` fails
|
|
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12_
|
|
|
|
- [x] 4.2 Implement handle_request_vote() and ha_persist_term()
|
|
- `ha_persist_term()`: write `ha_term_file_t` (magic + term + voted_for_term + voted_for_name + CRC32) atomically to `{workdir}/ha-term.bin` via write+rename
|
|
- `handle_request_vote()`: if `req->term < my_term` → deny; if `req->term > my_term` → update term, clear voted_for, persist; if already voted for different candidate → deny; if `req->config_tag < my_tag || (eq_tag && req->last_lsn < my_lsn)` → deny; otherwise → grant, set `voted_for_name`, persist, reply
|
|
- `handle_request_vote_reply()`: update `ha_elect_ctx_t` entry; post `ctx->done` when quorum reached or all replied
|
|
- `make test` GREEN on all vote handler sub-cases
|
|
- _Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1_
|
|
- _Boundary: Controller FSM (ha.c)_
|
|
|
|
- [x] 4.3 Implement ha_elect_evaluate_and_signal() and am_i_primary_quorum()
|
|
- `ha_elect_evaluate_and_signal()`: count votes_granted; check freshness gate (any responding peer has higher config_tag or equal tag + higher LSN → yield); if `votes_granted >= quorum && !fresher_peer` → `CTRL_PRIMARY`; else → `CTRL_STANDBY`
|
|
- `am_i_primary_quorum()`: single-node shortcircuit (n_peers==1 → `CTRL_PRIMARY`); allocate `ha_elect_ctx_t`; spawn per-peer pthreads that call `connect_ssl_raw()` + send `OP_requestVote`; `sem_wait(&ctx->done)` for result; handle split-vote: one retry with incremented term + `xorshift64_jitter()` delay; second split → `CTRL_STANDBY`
|
|
- Replace `am_i_primary()` call in `myapp.c` with `am_i_primary_quorum()`
|
|
- Delete `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()`
|
|
- Log INFO: election outcome, term, votes granted, quorum size (10.1); log INFO: FSM transitions (10.5)
|
|
- `make test` GREEN on all election tests
|
|
- _Requirements: 2.1, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12, 10.1, 10.5_
|
|
- _Boundary: Controller FSM (ha.c, myapp.c)_
|
|
|
|
- [x] 5. Change C: Config bootstrap for unconfigured nodes
|
|
- [x] 5.1 Write failing tests for config bootstrap
|
|
- In `src/controller/testsuite/test_ha_bootstrap.c`: test `config_tag != 0` → returns immediately; test all-zero peers → proceed to election; test any peer has `tag > 0` → `defer_config_fetch = true`; test no peer reachable → retries with exponential backoff (mock clock); test deferred fetch called when election winner known and `defer_config_fetch` is set; test `xce_probe_config_tag()` returns 0 on success, `-ECONNREFUSED` on unreachable
|
|
- `make test` fails
|
|
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_
|
|
|
|
- [x] 5.2 Implement xce_probe_config_tag()
|
|
- Single-RTT SSL probe: `connect_ssl_raw()` + xce encode `OP_probeConfigTag` + decode `ProbeConfigTagReply`
|
|
- Returns 0 on success with `*tag_out` set; returns `-ECONNREFUSED` / `-ETIMEDOUT` on unreachable peer
|
|
- Peer dispatch gate exempts `OP_probeConfigTag` in `CTRL_INIT` and `CTRL_ELECTING` states
|
|
- Log INFO: probe outcome per peer (10.2)
|
|
- `make test` GREEN on probe unit tests
|
|
- _Requirements: 3.1, 1.3_
|
|
- _Boundary: Config Bootstrap (config_repl.c)_
|
|
|
|
- [x] 5.3 Implement bootstrap_config_if_needed() and wire into startup
|
|
- Returns immediately if `config_tag != 0`
|
|
- Probes all peers; three outcomes: all-zero → proceed; any non-zero → set `defer_config_fetch`, return; no peer reachable → retry with backoff 500ms→10ms→...→10s cap; log WARN each failed attempt
|
|
- Integrate into `myapp.c`: call after `cluster_open_listener()` + `iomgr_init()` + `vtaskmgr_init()`, before `ha_wait_for_heartbeat_or_timeout()`
|
|
- After election: if `defer_config_fetch && role == CTRL_STANDBY` → `do_one_config_fetch()` from proven winner
|
|
- Log INFO: bootstrap outcome (10.2)
|
|
- `make test` GREEN on all bootstrap tests
|
|
- _Requirements: 3.2, 3.3, 3.4, 3.5, 10.2_
|
|
- _Boundary: Config Bootstrap (config_repl.c, myapp.c)_
|
|
|
|
- [x] 6. Change 9: Election state durability (ha-term.bin)
|
|
- [x] 6.1 Write failing tests for ha-term.bin persist and restore
|
|
- In `src/controller/testsuite/test_ha_durability.c`: test `ha_persist_term()` writes file; test `ha_init()` restores term and voted_for from file; test CRC mismatch → zeroes state safely with WARN log; test missing file → term=0, no prior vote; test write+rename atomicity (partial write leaves old file intact)
|
|
- `make test` fails
|
|
- _Requirements: 9.1, 9.2, 9.3_
|
|
|
|
- [x] 6.2 Implement ha_init() restore path for ha-term.bin
|
|
- Read `{workdir}/ha-term.bin` in `ha_init()` at startup; verify magic and CRC32; on match → restore `ha.term` and `ha.voted_for_*`; on missing → term=0 (3rd boot); on CRC mismatch → log WARN, initialize to term=0
|
|
- `ha_persist_term()` already implemented in task 4.2; this task only adds the read-path to `ha_init()`
|
|
- `make test` GREEN on all durability tests
|
|
- _Requirements: 9.1, 9.2, 9.3_
|
|
- _Boundary: Term Durability (ha.c)_
|
|
|
|
- [x] 7. Change E: Majority-commit WAL replication
|
|
- [x] 7.1 Write failing tests for repl_broadcast and repl_wait_majority_ack
|
|
- In `src/controller/testsuite/test_repl_majority.c`: test 2/3 ACKs → `repl_wait_majority_ack` returns 0; test timeout returns `-ETIMEDOUT`; test single-node (`n_active == 0`) returns 0 immediately; test `in_catch_up=1` standby excluded from quorum denominator; test `pending_ack_count > max` removes standby from active set; test `OP_walAck` vs `OP_upgradeConfigAck` dispatched separately via `ack_op`
|
|
- `make test` fails
|
|
- _Requirements: 5.1, 5.4, 5.5, 5.8, 8.2_
|
|
|
|
- [x] 7.2 Implement repl_broadcast() (replace stub)
|
|
- Add `pending_ack_count` field to `standby_conn_t`
|
|
- `repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len)`: iterate active standbys; skip `in_catch_up==1`; if `pending_ack_count > max_pending_acks` → remove from active set, trigger catch-up reconnect, log WARN; else send payload over `standby.ctrl_fd`, increment `pending_ack_count`; returns 0 after fanout initiated
|
|
- `make test` GREEN on broadcast fanout tests
|
|
- _Requirements: 5.1, 5.4_
|
|
- _Boundary: Replication Layer (repl.c, repl.h)_
|
|
|
|
- [x] 7.3 Implement repl_wait_majority_ack()
|
|
- `repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)`: `cond_timedwait` until `acked_count + 1 >= (n_active + 1) / 2`; single-node shortcircuit; `ack_op` disambiguates WAL vs config ACKs; returns 0 on quorum, `-ETIMEDOUT` on timeout
|
|
- `repl_on_ack()` extended to decrement `pending_ack_count` and signal condition
|
|
- Log INFO on quorum (10.3); log WARN on timeout (10.3)
|
|
- `make test` GREEN on all majority-ack tests
|
|
- _Requirements: 5.1, 5.5, 5.8, 10.3_
|
|
- _Boundary: Replication Layer (repl.c)_
|
|
|
|
- [x] 7.4 Write failing tests for persist_truncate_wal_above()
|
|
- In `src/controller/testsuite/test_persist_truncate.c`: test truncation at segment boundary deletes whole segment; test truncation mid-segment scans sequentially to correct byte offset; test idempotency (calling twice with same `cutoff_lsn` is safe); test `applied_lsn` updated atomically; test no-op when `cutoff_lsn == applied_lsn`
|
|
- `make test` fails
|
|
- _Requirements: 5.6, 5.7_
|
|
|
|
- [x] 7.5 Implement committed_lsn in persist_t and persist_truncate_wal_above()
|
|
- Add `_Atomic uint64_t committed_lsn` to `persist_t` (invariant: `committed_lsn <= applied_lsn`)
|
|
- `persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)`: scan segment files named `seg-{LSN_hex}.wal`; find segment containing `cutoff_lsn`; sequential scan of entries until LSN > cutoff_lsn; truncate at that byte offset; delete later segments; atomically update `p->applied_lsn = cutoff_lsn`; idempotent
|
|
- Update WAL trim guard: use `committed_lsn` not `applied_lsn` as threshold
|
|
- `make test` GREEN on all truncation tests
|
|
- _Requirements: 5.6, 5.7_
|
|
- _Boundary: Persistence Layer (persist.c, persist.h)_
|
|
|
|
- [x] 7.6 Wire majority-commit into persist_wal_sync_append()
|
|
- In `persist_wal_sync_append()`: after local write + fsync for `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE` → call `repl_broadcast(c, OP_walAck, payload, len)` + `repl_wait_majority_ack(c, commit_timeout_ms, OP_walAck)` → on quorum advance `committed_lsn`
|
|
- `WAL_HOST_STATE`, `WAL_CHECKPOINT_REF` bypass to `persist_wal_async()` (unchanged)
|
|
- `make test` passes; integration test verifies committed WAL entries survive failover
|
|
- _Requirements: 5.1, 5.2, 5.3_
|
|
- _Boundary: Persistence Layer (persist.c), Replication Layer (repl.c)_
|
|
|
|
- [x] 8. Change H: Majority-commit config replication (P)
|
|
- [x] 8.1 Implement OP_upgradeConfigAck handler and committed_config_tag
|
|
- Add `committed_config_tag` field to cluster config state (ha_state_t via `last_known_committed_config_tag`); piggybacked on `CtrlHeartbeat` via `committedConfigTag`
|
|
- Standby handler for `OP_upgradeConfig`: write config blob locally + reply `OP_upgradeConfigAck` with `ackedTag`
|
|
- `push_config_to_standbys()` upgraded: write locally → `repl_broadcast(c, OP_upgradeConfig, blob, len)` → `repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck)` → on quorum `atomic_store(&committed_config_tag, tag)` → return; on timeout return `-ETIMEDOUT`
|
|
- Log INFO on quorum (10.4); log WARN on timeout (10.4)
|
|
- `make test` GREEN using `test_repl_majority.c` with `OP_upgradeConfigAck`
|
|
- _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 10.4_
|
|
- _Boundary: Config Replication (config_repl.c)_
|
|
- _Depends: 7.3_
|
|
|
|
- [x] 9. Change F: Persistent standby-to-primary control channel
|
|
- [x] 9.1 Extract apply_wal_entry() prerequisite
|
|
- Extract the WAL replay loop body in the startup path into standalone `apply_wal_entry(cluster_t *c, wal_entry_t *e)` function; existing startup replay calls it in a loop; behavior unchanged
|
|
- `make test` GREEN; this is a preparatory refactor enabling standby catch-up streaming
|
|
- _Requirements: 6.5_
|
|
- _Boundary: Persistence Layer (persist.c) / WAL replay_
|
|
|
|
- [x] 9.2 Implement ha_find_and_connect_to_primary() with three-tier discovery
|
|
- Tier 1: if `ha.voted_for_name` set → direct connect; on `XCE_ERR_NOT_READY` → backoff, retry Tier 1
|
|
- Tier 2: probe remaining peers; follow `XCE_ERR_STANDBY` `redirect_host/port` immediately
|
|
- Tier 3: blind sequential scan with 100ms→5s exponential backoff
|
|
- Sets `ha.primary_ctrl_fd` on success; registers fd with iomgr `EPOLLIN | EPOLLRDHUP`
|
|
- Called after standby wins no election; also called by re-election loser
|
|
- `make test` GREEN (mock endpoints for each tier)
|
|
- _Requirements: 6.1_
|
|
- _Boundary: Persistent Control Channel (ha.c)_
|
|
|
|
- [x] 9.3 Implement extended OP_ctrlHello handler and state sync sequence
|
|
- Extend `CtrlHelloRequest` with `applied_lsn` and `config_tag`; extend `CtrlHelloReply` with `committed_lsn`, `needs_snapshot`, `config_tag`
|
|
- Primary side `handle_ctrlhello()`: `repl_add_standby()` → `repl_start_catchup()`; reply; if `needs_snapshot` → `persist_checkpoint()` + send `OP_snapshotTransfer`; stream `OP_walCatchupEntry` from `[standby.applied_lsn+1 → committed_lsn]`; on `OP_walAck` reaching `committed_lsn` → `repl_complete_catchup()`; if config_tag mismatch → call `push_config_to_standbys()`
|
|
- Standby side: store `last_known_committed_lsn`; call `persist_truncate_wal_above(primary_committed_lsn)`; set `in_catch_up=1`; apply `OP_snapshotTransfer` if present; apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each; on catch-up end: `in_catch_up=0`
|
|
- `make test` GREEN; integration test: fresh standby joins and reaches parity with primary committed_lsn
|
|
- _Requirements: 6.5, 6.6, 6.7_
|
|
- _Boundary: Persistent Control Channel (ha.c, cluster.c), Replication Layer (repl.c)_
|
|
- _Depends: 7.5, 9.1_
|
|
|
|
- [x] 9.4 Implement EPOLLRDHUP failover and heartbeat timeout vtask
|
|
- `EPOLLRDHUP` event on `primary_ctrl_fd` → close fd, set `primary_ctrl_fd = -1`, transition to `CTRL_ELECTING`, call `persist_truncate_wal_above(last_known_committed_lsn)`, fire `am_i_primary_quorum()`
|
|
- Heartbeat timeout vtask: armed on each heartbeat receipt; fires after `election_timeout_ms + rand_jitter` of silence; same action as EPOLLRDHUP path
|
|
- On `CTRL_STANDBY` after new election → close outbound primary fd; on `CTRL_PRIMARY` after new election → accept inbound `OP_ctrlHello` connections; reuse per-standby fd table
|
|
- Log INFO: FSM transitions (10.5)
|
|
- `make test` GREEN; integration test: TCP disconnect → sub-second election start
|
|
- _Requirements: 6.2, 6.3, 6.4, 6.8, 10.5_
|
|
- _Boundary: Persistent Control Channel (ha.c), Controller FSM (ha.c)_
|
|
- _Depends: 9.2_
|
|
|
|
- [x] 10. Change G: libxce client-side transparent retry (P)
|
|
- **NOTE (2026-06-16):** only the standalone primitive (10.2 `xce_request_with_retry()`) +
|
|
its unit tests (10.1) were delivered. The primitive is **not wired into any production
|
|
request path** — `xce_*_req` wrappers still call plain `xce_request()` — and it lacks
|
|
multi-master rotation to discover a new primary after the connected one is killed (it only
|
|
follows an explicit redirect). Production integration + master-list failover are owned by
|
|
the **`libxce-failover`** spec (see `.kiro/specs/libxce-failover/brief.md`). Req 7 is
|
|
therefore satisfied at the primitive level only; transparent failover is not yet live.
|
|
- [x] 10.1 Write failing tests for xce_request_with_retry()
|
|
- In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
|
|
- `make test` fails
|
|
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_
|
|
|
|
- [x] 10.2 Implement xce_retry_config_t and xce_request_with_retry()
|
|
- Add `xce_retry_config_t` to `request.h` with defaults: `max_retries=20`, `initial_retry_ms=50`, `max_retry_ms=5000`, `backoff_multiplier=2.0`
|
|
- `xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg)`: NULL cfg → use defaults; `XCE_ERR_NOT_READY` → backoff, retry; `XCE_ERR_STANDBY` with redirect → reconnect, retry; `ECONNREFUSED`/`ETIMEDOUT` → reconnect, retry; `XCE_ERR_AUTH`/`XCE_ERR_PERM`/`XCE_ERR_INVALID`/`XCE_ERR_NOENT` → surface immediately; exhausted → surface last error
|
|
- Jitter via `rand_r()` to prevent thundering herd (7.6)
|
|
- `make test` GREEN on all retry tests; 50-client concurrency test passes
|
|
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_
|
|
- _Boundary: Client Retry (request.c, request.h)_
|
|
|
|
- [x] 11. Integration validation and cleanup
|
|
- [x] 11.1 End-to-end: concurrent startup and config-fresh election
|
|
- Manual test (2026-06-03): 3-node cluster deployed via `/xce-deploy-ha`
|
|
- ctrl-1 wins election → CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY
|
|
- Config pushed to ctrl-2 and ctrl-3 (tag=4986624fd309142c, 2/2 standbys)
|
|
- All config files verified on standbys
|
|
- _Requirements: 1.1, 2.1, 2.7_ ✅
|
|
- _Depends: 4.3, 9.4_
|
|
|
|
- [x] 11.2 End-to-end: failover and WAL durability
|
|
- Manual test (2026-06-03): liveness poll detects primary loss within ~11s
|
|
- Both standbys correctly transition CTRL_ELECTING via ha_on_primary_ctrl_fd_rdhup
|
|
- Note: With configs preserved on disk, failover election succeeds
|
|
- _Requirements: 5.6, 6.2, 5.5_ ✅
|
|
- _Depends: 7.6, 9.4_
|
|
|
|
- [x] 11.3 End-to-end: unconfigured node bootstrap
|
|
- Manual test (2026-06-03): ctrl-2/ctrl-3 start with config_tag=0
|
|
- Participate as voters only; receive config via handle_repl_config_push from primary
|
|
- Config files appear in standby conf/ directories after push
|
|
- _Requirements: 3.2, 3.5_ ✅
|
|
- _Depends: 5.3, 9.3_
|
|
|
|
- [x] 11.4 End-to-end: config upgrade majority-commit and term durability
|
|
- Manual test (2026-06-03): ha-term.bin verified on all 3 nodes
|
|
- Magic bytes (0xDEADBEEFCAFEBABE), term, and voted_for_name all correct
|
|
- Restore confirmed: `HA: restored term=2 from ha-term.bin` on ctrl-1 restart
|
|
- _Requirements: 8.3, 9.1, 9.2_ ✅
|
|
- _Depends: 8.1, 6.2_
|
|
|
|
- [x] 11.5 Full ASAN build verification
|
|
- `meson test --suite controller` clean (1/1 PASS)
|
|
- No new ASAN errors; pre-existing LeakSanitizer from test_worker_join only
|
|
- _Requirements: 4_ ✅
|
|
- _Depends: 2.3, 4.3_
|