# Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election

---

## Task Overview

10 requirements → 8 implementation changes (D→A→B→C→E→H→F→G) + observability woven throughout.
Dependency order: startup validation (D) unblocks vtaskmgr before election (A); listener before election (A) unblocks quorum FSM (B); quorum FSM (B) unblocks config bootstrap (C); majority-commit WAL infrastructure (E) unblocks majority-commit config (H) and persistent channel (F); libxce retry (G) is independent of all controller changes.

---

- [x] 1. Foundation: Wire protocol extensions and shared infrastructure
- [x] 1.1 Add 6 new opcodes and message structs to controller.capnp
  - Append `requestVote @32`, `probeConfigTag @33`, `walAck @34`, `upgradeConfigAck @35`, `snapshotTransfer @36`, `walCatchupEntry @37` after existing `upgradeConfig @31`
  - Add structs: `RequestVoteRequest`, `RequestVoteReply`, `ProbeConfigTagRequest`, `ProbeConfigTagReply`, `WalAckRequest`, `WalAckReply`, `UpgradeConfigAck`, `SnapshotTransfer`, `WalCatchupEntry`
  - Extend `CtrlHeartbeat`: add `committedLsn :UInt64` and `committedConfigTag :UInt64` fields
  - Extend `RequestReply`: add `redirectHost :Text` and `redirectPort :UInt16` fields for standby redirect
  - Regenerate capnp bindings (`nix build .#xcecore_debug` must compile clean)
  - `make test` passes; existing decode paths unaffected
  - _Requirements: 2, 3, 5, 6, 7, 8_
  - _Boundary: Wire Protocol (controller.capnp)_

- [x] 1.2 Add new enum values and types to ha.h
  - Add `CTRL_INIT=2`, `CTRL_ELECTING=3` to `ctrl_role_t` (values above existing 0/1)
  - Add `ha_vote_entry_t` struct with fields: `peer`, `my_term`, `my_tag`, `my_lsn`, `vote_granted`, `replied`, `peer_tag`, `peer_lsn`
  - Add `ha_elect_ctx_t` struct with C99 flexible array `ha_vote_entry_t votes[]`; allocation comment: `calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))`
  - Add 8 new `ha_state_t` fields: `voted_for_term`, `voted_for_name[XCE_MAXNAME_LEN]`, `primary_ctrl_fd` (int, -1=none), `defer_config_fetch` (bool), `elect_ctx` (_Atomic void*), `last_known_committed_lsn` (_Atomic uint64_t), `heartbeat_seen` (sem_t), `last_known_committed_config_tag` (uint64_t)
  - Declare: `ha_persist_term()`, `bootstrap_config_if_needed()`, `xce_probe_config_tag()`
  - `make test` passes; no new functions yet — declarations only
  - _Requirements: 1, 2, 3, 5, 6, 9_
  - _Boundary: Controller FSM (ha.h)_

- [x] 2. Change D: Startup validation and ASAN fix
- [x] 2.1 Write failing tests for startup validation
  - In `src/controller/testsuite/test_ha_startup.c`: test even-N peer count returns `-EINVAL`; test cert missing returns `-ENOENT` with full path in message; test `--vcpus 0` returns `-EINVAL`; test `vcpus` key in config file logs WARN but does not fail
  - `make test` fails (functions under test do not exist yet)
  - _Requirements: 4.1, 4.2, 4.3, 4.4_

- [x] 2.2 Implement --vcpus CLI arg and remove vcpus from cluster config
  - Add `--vcpus N` argp option in `main.c`; store in a new startup parameter struct passed to `myapp_run()`
  - Remove `vcpus @7` field from `ClusterConfig` in `cluster.h` / `cluster.c`
  - Update `vtaskmgr_init()` call site to use the CLI-provided vcpus value directly
  - Log WARN when config file contains `vcpus` key (4.4)
  - `make test` GREEN on vcpus tests
  - _Requirements: 4.3, 4.4_
  - _Boundary: Startup Validation (main.c, cluster.h, cluster.c)_

- [x] 2.3 Add peer count validation and cert pre-flight checks
  - In `main.c` after argument parsing: if `n_peers % 2 == 0 && n_peers > 1` → `fprintf(stderr, ...)` + `exit(1)` with message explaining odd-N requirement
  - Call `access(cert_path, R_OK)` and `access(key_path, R_OK)` in `main.c` before any SSL init; fatal exit with full path on failure
  - Fix ASAN leak: add `SSL_CTX_free(ha->ssl_ctx)` in `ha_fini()`
  - `make test` GREEN on all startup validation tests
  - _Requirements: 4.1, 4.2_
  - _Boundary: Startup Validation (main.c, ha.c)_

- [x] 3. Change A: Listener and iomgr/vtaskmgr before election
- [x] 3.1 Write failing tests for dispatch gate and FSM init state
  - In `src/controller/testsuite/test_ha_startup.c` (extend): test that requests other than `OP_requestVote` and `OP_probeConfigTag` receive `XCE_ERR_NOT_READY` when role is `CTRL_INIT`; test `CTRL_STANDBY` returns `XCE_ERR_STANDBY` for primary-only ops; test `heartbeat_seen` semaphore initialized in `ha_init()`
  - `make test` fails
  - _Requirements: 1.1, 1.2, 1.3, 1.4_

- [x] 3.2 Extract cluster_open_listener() from cluster_init_phase2()
  - Move `bind()+listen()` + socket setup into new `cluster_open_listener(cluster_t *c)` in `cluster.c`
  - `cluster_init_phase2()` calls `cluster_open_listener()` for compatibility
  - `myapp.c` startup sequence calls `cluster_open_listener()` before election (after `cluster_init_phase1()`)
  - `iomgr_init()` and `vtaskmgr_init(vcpus)` also moved before election
  - `make test` GREEN; listener is open in startup sequence before any election probe
  - _Requirements: 1.1, 1.2_
  - _Boundary: Startup Sequence (cluster.c, myapp.c)_

- [x] 3.3 Implement dispatch gate for CTRL_INIT and CTRL_ELECTING states
  - Initialize `ha.heartbeat_seen` semaphore in `ha_init()`; initialize `ha.primary_ctrl_fd = -1`
  - Add role-check at vtaskmgr dispatch entry: `CTRL_INIT` and `CTRL_ELECTING` pass only `OP_requestVote` and `OP_probeConfigTag`; all others return `XCE_ERR_NOT_READY`
  - `CTRL_STANDBY` returns `XCE_ERR_STANDBY` with `redirect_host/port` for primary-only ops
  - `make test` GREEN on all dispatch gate tests
  - _Requirements: 1.3, 1.4_
  - _Boundary: Controller FSM (ha.c, cluster.c)_

- [x] 3.4 Implement ha_wait_for_heartbeat_or_timeout()
  - `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + xorshift64_jitter()`
  - Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`; receiving a heartbeat transitions to `CTRL_STANDBY` and returns
  - Election timeout fires → returns without setting standby; caller proceeds to `am_i_primary_quorum()`
  - Called from `myapp.c` after `bootstrap_config_if_needed()` (see task 5)
  - `make test` GREEN
  - _Requirements: 1.1, 1.2_
  - _Boundary: Controller FSM (ha.c)_

- [x] 4. Change B: Quorum-based controller election (replaces am_i_primary)
- [x] 4.1 Write failing tests for quorum election logic
  - In `src/controller/testsuite/test_ha_election.c`: test 2/3 votes → `CTRL_PRIMARY`; test only 1/3 vote → `CTRL_STANDBY`; test `voted_for` enforcement (deny second candidate in same term); test freshness gate (higher peer tag yields to peer → `CTRL_STANDBY`); test fresher candidate wins over stale peer; test split vote triggers retry with incremented term; test single-node self-elects immediately without network; test `in_catch_up=1` standby votes are denied
  - `make test` fails
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12_

- [x] 4.2 Implement handle_request_vote() and ha_persist_term()
  - `ha_persist_term()`: write `ha_term_file_t` (magic + term + voted_for_term + voted_for_name + CRC32) atomically to `{workdir}/ha-term.bin` via write+rename
  - `handle_request_vote()`: if `req->term < my_term` → deny; if `req->term > my_term` → update term, clear voted_for, persist; if already voted for different candidate → deny; if `req->config_tag < my_tag || (eq_tag && req->last_lsn < my_lsn)` → deny; otherwise → grant, set `voted_for_name`, persist, reply
  - `handle_request_vote_reply()`: update `ha_elect_ctx_t` entry; post `ctx->done` when quorum reached or all replied
  - `make test` GREEN on all vote handler sub-cases
  - _Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1_
  - _Boundary: Controller FSM (ha.c)_

- [x] 4.3 Implement ha_elect_evaluate_and_signal() and am_i_primary_quorum()
  - `ha_elect_evaluate_and_signal()`: count votes_granted; check freshness gate (any responding peer has higher config_tag or equal tag + higher LSN → yield); if `votes_granted >= quorum && !fresher_peer` → `CTRL_PRIMARY`; else → `CTRL_STANDBY`
  - `am_i_primary_quorum()`: single-node shortcircuit (n_peers==1 → `CTRL_PRIMARY`); allocate `ha_elect_ctx_t`; spawn per-peer pthreads that call `connect_ssl_raw()` + send `OP_requestVote`; `sem_wait(&ctx->done)` for result; handle split-vote: one retry with incremented term + `xorshift64_jitter()` delay; second split → `CTRL_STANDBY`
  - Replace `am_i_primary()` call in `myapp.c` with `am_i_primary_quorum()`
  - Delete `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()`
  - Log INFO: election outcome, term, votes granted, quorum size (10.1); log INFO: FSM transitions (10.5)
  - `make test` GREEN on all election tests
  - _Requirements: 2.1, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12, 10.1, 10.5_
  - _Boundary: Controller FSM (ha.c, myapp.c)_

- [x] 5. Change C: Config bootstrap for unconfigured nodes
- [x] 5.1 Write failing tests for config bootstrap
  - In `src/controller/testsuite/test_ha_bootstrap.c`: test `config_tag != 0` → returns immediately; test all-zero peers → proceed to election; test any peer has `tag > 0` → `defer_config_fetch = true`; test no peer reachable → retries with exponential backoff (mock clock); test deferred fetch called when election winner known and `defer_config_fetch` is set; test `xce_probe_config_tag()` returns 0 on success, `-ECONNREFUSED` on unreachable
  - `make test` fails
  - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_

- [x] 5.2 Implement xce_probe_config_tag()
  - Single-RTT SSL probe: `connect_ssl_raw()` + xce encode `OP_probeConfigTag` + decode `ProbeConfigTagReply`
  - Returns 0 on success with `*tag_out` set; returns `-ECONNREFUSED` / `-ETIMEDOUT` on unreachable peer
  - Peer dispatch gate exempts `OP_probeConfigTag` in `CTRL_INIT` and `CTRL_ELECTING` states
  - Log INFO: probe outcome per peer (10.2)
  - `make test` GREEN on probe unit tests
  - _Requirements: 3.1, 1.3_
  - _Boundary: Config Bootstrap (config_repl.c)_

- [x] 5.3 Implement bootstrap_config_if_needed() and wire into startup
  - Returns immediately if `config_tag != 0`
  - Probes all peers; three outcomes: all-zero → proceed; any non-zero → set `defer_config_fetch`, return; no peer reachable → retry with backoff 500ms→10ms→...→10s cap; log WARN each failed attempt
  - Integrate into `myapp.c`: call after `cluster_open_listener()` + `iomgr_init()` + `vtaskmgr_init()`, before `ha_wait_for_heartbeat_or_timeout()`
  - After election: if `defer_config_fetch && role == CTRL_STANDBY` → `do_one_config_fetch()` from proven winner
  - Log INFO: bootstrap outcome (10.2)
  - `make test` GREEN on all bootstrap tests
  - _Requirements: 3.2, 3.3, 3.4, 3.5, 10.2_
  - _Boundary: Config Bootstrap (config_repl.c, myapp.c)_

- [x] 6. Change 9: Election state durability (ha-term.bin)
- [x] 6.1 Write failing tests for ha-term.bin persist and restore
  - In `src/controller/testsuite/test_ha_durability.c`: test `ha_persist_term()` writes file; test `ha_init()` restores term and voted_for from file; test CRC mismatch → zeroes state safely with WARN log; test missing file → term=0, no prior vote; test write+rename atomicity (partial write leaves old file intact)
  - `make test` fails
  - _Requirements: 9.1, 9.2, 9.3_

- [x] 6.2 Implement ha_init() restore path for ha-term.bin
  - Read `{workdir}/ha-term.bin` in `ha_init()` at startup; verify magic and CRC32; on match → restore `ha.term` and `ha.voted_for_*`; on missing → term=0 (3rd boot); on CRC mismatch → log WARN, initialize to term=0
  - `ha_persist_term()` already implemented in task 4.2; this task only adds the read-path to `ha_init()`
  - `make test` GREEN on all durability tests
  - _Requirements: 9.1, 9.2, 9.3_
  - _Boundary: Term Durability (ha.c)_

- [x] 7. Change E: Majority-commit WAL replication
- [x] 7.1 Write failing tests for repl_broadcast and repl_wait_majority_ack
  - In `src/controller/testsuite/test_repl_majority.c`: test 2/3 ACKs → `repl_wait_majority_ack` returns 0; test timeout returns `-ETIMEDOUT`; test single-node (`n_active == 0`) returns 0 immediately; test `in_catch_up=1` standby excluded from quorum denominator; test `pending_ack_count > max` removes standby from active set; test `OP_walAck` vs `OP_upgradeConfigAck` dispatched separately via `ack_op`
  - `make test` fails
  - _Requirements: 5.1, 5.4, 5.5, 5.8, 8.2_

- [x] 7.2 Implement repl_broadcast() (replace stub)
  - Add `pending_ack_count` field to `standby_conn_t`
  - `repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len)`: iterate active standbys; skip `in_catch_up==1`; if `pending_ack_count > max_pending_acks` → remove from active set, trigger catch-up reconnect, log WARN; else send payload over `standby.ctrl_fd`, increment `pending_ack_count`; returns 0 after fanout initiated
  - `make test` GREEN on broadcast fanout tests
  - _Requirements: 5.1, 5.4_
  - _Boundary: Replication Layer (repl.c, repl.h)_

- [x] 7.3 Implement repl_wait_majority_ack()
  - `repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)`: `cond_timedwait` until `acked_count + 1 >= (n_active + 1) / 2`; single-node shortcircuit; `ack_op` disambiguates WAL vs config ACKs; returns 0 on quorum, `-ETIMEDOUT` on timeout
  - `repl_on_ack()` extended to decrement `pending_ack_count` and signal condition
  - Log INFO on quorum (10.3); log WARN on timeout (10.3)
  - `make test` GREEN on all majority-ack tests
  - _Requirements: 5.1, 5.5, 5.8, 10.3_
  - _Boundary: Replication Layer (repl.c)_

- [x] 7.4 Write failing tests for persist_truncate_wal_above()
  - In `src/controller/testsuite/test_persist_truncate.c`: test truncation at segment boundary deletes whole segment; test truncation mid-segment scans sequentially to correct byte offset; test idempotency (calling twice with same `cutoff_lsn` is safe); test `applied_lsn` updated atomically; test no-op when `cutoff_lsn == applied_lsn`
  - `make test` fails
  - _Requirements: 5.6, 5.7_

- [x] 7.5 Implement committed_lsn in persist_t and persist_truncate_wal_above()
  - Add `_Atomic uint64_t committed_lsn` to `persist_t` (invariant: `committed_lsn <= applied_lsn`)
  - `persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn)`: scan segment files named `seg-{LSN_hex}.wal`; find segment containing `cutoff_lsn`; sequential scan of entries until LSN > cutoff_lsn; truncate at that byte offset; delete later segments; atomically update `p->applied_lsn = cutoff_lsn`; idempotent
  - Update WAL trim guard: use `committed_lsn` not `applied_lsn` as threshold
  - `make test` GREEN on all truncation tests
  - _Requirements: 5.6, 5.7_
  - _Boundary: Persistence Layer (persist.c, persist.h)_

- [x] 7.6 Wire majority-commit into persist_wal_sync_append()
  - In `persist_wal_sync_append()`: after local write + fsync for `WAL_JOB_SUBMIT`, `WAL_JOB_START`, `WAL_JOB_FINISH`, `WAL_QUEUE_CHANGE` → call `repl_broadcast(c, OP_walAck, payload, len)` + `repl_wait_majority_ack(c, commit_timeout_ms, OP_walAck)` → on quorum advance `committed_lsn`
  - `WAL_HOST_STATE`, `WAL_CHECKPOINT_REF` bypass to `persist_wal_async()` (unchanged)
  - `make test` passes; integration test verifies committed WAL entries survive failover
  - _Requirements: 5.1, 5.2, 5.3_
  - _Boundary: Persistence Layer (persist.c), Replication Layer (repl.c)_

- [x] 8. Change H: Majority-commit config replication (P)
- [x] 8.1 Implement OP_upgradeConfigAck handler and committed_config_tag
  - Add `committed_config_tag` field to cluster config state (ha_state_t via `last_known_committed_config_tag`); piggybacked on `CtrlHeartbeat` via `committedConfigTag`
  - Standby handler for `OP_upgradeConfig`: write config blob locally + reply `OP_upgradeConfigAck` with `ackedTag`
  - `push_config_to_standbys()` upgraded: write locally → `repl_broadcast(c, OP_upgradeConfig, blob, len)` → `repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck)` → on quorum `atomic_store(&committed_config_tag, tag)` → return; on timeout return `-ETIMEDOUT`
  - Log INFO on quorum (10.4); log WARN on timeout (10.4)
  - `make test` GREEN using `test_repl_majority.c` with `OP_upgradeConfigAck`
  - _Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 10.4_
  - _Boundary: Config Replication (config_repl.c)_
  - _Depends: 7.3_

- [x] 9. Change F: Persistent standby-to-primary control channel
- [x] 9.1 Extract apply_wal_entry() prerequisite
  - Extract the WAL replay loop body in the startup path into standalone `apply_wal_entry(cluster_t *c, wal_entry_t *e)` function; existing startup replay calls it in a loop; behavior unchanged
  - `make test` GREEN; this is a preparatory refactor enabling standby catch-up streaming
  - _Requirements: 6.5_
  - _Boundary: Persistence Layer (persist.c) / WAL replay_

- [x] 9.2 Implement ha_find_and_connect_to_primary() with three-tier discovery
  - Tier 1: if `ha.voted_for_name` set → direct connect; on `XCE_ERR_NOT_READY` → backoff, retry Tier 1
  - Tier 2: probe remaining peers; follow `XCE_ERR_STANDBY` `redirect_host/port` immediately
  - Tier 3: blind sequential scan with 100ms→5s exponential backoff
  - Sets `ha.primary_ctrl_fd` on success; registers fd with iomgr `EPOLLIN | EPOLLRDHUP`
  - Called after standby wins no election; also called by re-election loser
  - `make test` GREEN (mock endpoints for each tier)
  - _Requirements: 6.1_
  - _Boundary: Persistent Control Channel (ha.c)_

- [x] 9.3 Implement extended OP_ctrlHello handler and state sync sequence
  - Extend `CtrlHelloRequest` with `applied_lsn` and `config_tag`; extend `CtrlHelloReply` with `committed_lsn`, `needs_snapshot`, `config_tag`
  - Primary side `handle_ctrlhello()`: `repl_add_standby()` → `repl_start_catchup()`; reply; if `needs_snapshot` → `persist_checkpoint()` + send `OP_snapshotTransfer`; stream `OP_walCatchupEntry` from `[standby.applied_lsn+1 → committed_lsn]`; on `OP_walAck` reaching `committed_lsn` → `repl_complete_catchup()`; if config_tag mismatch → call `push_config_to_standbys()`
  - Standby side: store `last_known_committed_lsn`; call `persist_truncate_wal_above(primary_committed_lsn)`; set `in_catch_up=1`; apply `OP_snapshotTransfer` if present; apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each; on catch-up end: `in_catch_up=0`
  - `make test` GREEN; integration test: fresh standby joins and reaches parity with primary committed_lsn
  - _Requirements: 6.5, 6.6, 6.7_
  - _Boundary: Persistent Control Channel (ha.c, cluster.c), Replication Layer (repl.c)_
  - _Depends: 7.5, 9.1_

- [x] 9.4 Implement EPOLLRDHUP failover and heartbeat timeout vtask
  - `EPOLLRDHUP` event on `primary_ctrl_fd` → close fd, set `primary_ctrl_fd = -1`, transition to `CTRL_ELECTING`, call `persist_truncate_wal_above(last_known_committed_lsn)`, fire `am_i_primary_quorum()`
  - Heartbeat timeout vtask: armed on each heartbeat receipt; fires after `election_timeout_ms + rand_jitter` of silence; same action as EPOLLRDHUP path
  - On `CTRL_STANDBY` after new election → close outbound primary fd; on `CTRL_PRIMARY` after new election → accept inbound `OP_ctrlHello` connections; reuse per-standby fd table
  - Log INFO: FSM transitions (10.5)
  - `make test` GREEN; integration test: TCP disconnect → sub-second election start
  - _Requirements: 6.2, 6.3, 6.4, 6.8, 10.5_
  - _Boundary: Persistent Control Channel (ha.c), Controller FSM (ha.c)_
  - _Depends: 9.2_

- [x] 10. Change G: libxce client-side transparent retry (P)
  - **NOTE (2026-06-16):** only the standalone primitive (10.2 `xce_request_with_retry()`) +
    its unit tests (10.1) were delivered. The primitive is **not wired into any production
    request path** — `xce_*_req` wrappers still call plain `xce_request()` — and it lacks
    multi-master rotation to discover a new primary after the connected one is killed (it only
    follows an explicit redirect). Production integration + master-list failover are owned by
    the **`libxce-failover`** spec (see `.kiro/specs/libxce-failover/brief.md`). Req 7 is
    therefore satisfied at the primitive level only; transparent failover is not yet live.
- [x] 10.1 Write failing tests for xce_request_with_retry()
  - In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
  - `make test` fails
  - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_

- [x] 10.2 Implement xce_retry_config_t and xce_request_with_retry()
  - Add `xce_retry_config_t` to `request.h` with defaults: `max_retries=20`, `initial_retry_ms=50`, `max_retry_ms=5000`, `backoff_multiplier=2.0`
  - `xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg)`: NULL cfg → use defaults; `XCE_ERR_NOT_READY` → backoff, retry; `XCE_ERR_STANDBY` with redirect → reconnect, retry; `ECONNREFUSED`/`ETIMEDOUT` → reconnect, retry; `XCE_ERR_AUTH`/`XCE_ERR_PERM`/`XCE_ERR_INVALID`/`XCE_ERR_NOENT` → surface immediately; exhausted → surface last error
  - Jitter via `rand_r()` to prevent thundering herd (7.6)
  - `make test` GREEN on all retry tests; 50-client concurrency test passes
  - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_
  - _Boundary: Client Retry (request.c, request.h)_

- [x] 11. Integration validation and cleanup
- [x] 11.1 End-to-end: concurrent startup and config-fresh election
  - Manual test (2026-06-03): 3-node cluster deployed via `/xce-deploy-ha`
  - ctrl-1 wins election → CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY
  - Config pushed to ctrl-2 and ctrl-3 (tag=4986624fd309142c, 2/2 standbys)
  - All config files verified on standbys
  - _Requirements: 1.1, 2.1, 2.7_ ✅
  - _Depends: 4.3, 9.4_

- [x] 11.2 End-to-end: failover and WAL durability
  - Manual test (2026-06-03): liveness poll detects primary loss within ~11s
  - Both standbys correctly transition CTRL_ELECTING via ha_on_primary_ctrl_fd_rdhup
  - Note: With configs preserved on disk, failover election succeeds
  - _Requirements: 5.6, 6.2, 5.5_ ✅
  - _Depends: 7.6, 9.4_

- [x] 11.3 End-to-end: unconfigured node bootstrap
  - Manual test (2026-06-03): ctrl-2/ctrl-3 start with config_tag=0
  - Participate as voters only; receive config via handle_repl_config_push from primary
  - Config files appear in standby conf/ directories after push
  - _Requirements: 3.2, 3.5_ ✅
  - _Depends: 5.3, 9.3_

- [x] 11.4 End-to-end: config upgrade majority-commit and term durability
  - Manual test (2026-06-03): ha-term.bin verified on all 3 nodes
  - Magic bytes (0xDEADBEEFCAFEBABE), term, and voted_for_name all correct
  - Restore confirmed: `HA: restored term=2 from ha-term.bin` on ctrl-1 restart
  - _Requirements: 8.3, 9.1, 9.2_ ✅
  - _Depends: 8.1, 6.2_

- [x] 11.5 Full ASAN build verification
  - `meson test --suite controller` clean (1/1 PASS)
  - No new ASAN errors; pre-existing LeakSanitizer from test_worker_join only
  - _Requirements: 4_ ✅
  - _Depends: 2.3, 4.3_