update documents

.kiro/specs/ha-config-bootstrap/design.md

@@ -30,7 +30,7 @@ This feature eliminates three critical safety gaps in the XCE controller's LSF-s
 
 ### This Spec Owns
 
-- Controller election protocol: full replacement of `am_i_primary()` with `am_i_primary_quorum()`
+- Controller election protocol: full replacement of `am_i_primary()` with `am_i_primary_quorum()` (Note: `am_i_primary_quorum()`'s blocking pthread-per-peer implementation was itself later removed — production election now runs through async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; superseded by raft-library and election-iomgr-refactor, see those specs' design.md)
 - Controller FSM: 4-state lifecycle (`CTRL_INIT`, `CTRL_ELECTING`, `CTRL_PRIMARY`, `CTRL_STANDBY`) and all state transitions
 - Pre-election startup sequence: listener binding, iomgr/vtaskmgr init before election
 - Config bootstrap: `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, deferred fetch after election
@@ -57,7 +57,7 @@ This feature eliminates three critical safety gaps in the XCE controller's LSF-s
 
 - `iomgr` (epoll/`EPOLLIN`/`EPOLLRDHUP`): P0 — control channel and dispatch loop
 - `vtaskmgr`: P0 — `vtaskmgr_init(int nvcpus)` confirmed to take vcpus directly, no cluster_config dependency
-- `endpoint_connect()` / `connect_ssl_raw()`: P0 — SSL connection setup; TCP connect phase is blocking; vote request connections require per-peer pthreads
+- `endpoint_connect()` / `connect_ssl_raw()`: P0 — SSL connection setup; TCP connect phase is blocking; vote request connections require per-peer pthreads (Note: superseded by election-iomgr-refactor — vote requests now use async iomgr endpoints, not per-peer pthreads; see that spec's design.md)
 - `repl_add/remove/start_catchup/complete_catchup/needs_snapshot/on_ack/trim_lsn`: P0 — existing repl slot management reused unchanged
 - `do_one_config_fetch()`: P1 — wired on branch (`2aae45f`); deferred fetch depends on it
 - `xce_encode()` / `xce_decode()` / `xce_request()`: P0 — wire protocol encoding
@@ -131,6 +131,8 @@ graph TB
     end
 ```
 
+**Note (election mechanism superseded)**: The diagrams above show this spec's original blocking pthread-per-peer `am_i_primary_quorum()` election path. That implementation was removed by election-iomgr-refactor (task 5.3) and replaced by async `ha_start_election_async()` delegating to the `src/lib/raft/` library's `raft_start_election()`; `handle_request_vote()` is now test-only, production vote dispatch is `cluster.c` → `raft_on_vote_request()`. The 4-state FSM, majority-commit, and control-channel shapes shown remain accurate. See election-iomgr-refactor and raft-library design.md for the current mechanism.
+
 ### Technology Stack
 
 | Layer | Choice | Role | Notes |
@@ -138,7 +140,7 @@ graph TB
 | Runtime | C gnu17 | All new code | No C++ |
 | Wire format | Cap'n Proto | 6 new opcodes in `controller.capnp` | Appended after existing @31 |
 | SSL | OpenSSL (existing) | Vote request and probe connections | Reuses `connect_ssl_raw()` |
-| Consensus | Raft subset (custom) | Leader election + majority-commit | No external library; bounded scope justified by static membership and no snapshot protocol need |
+| Consensus | Raft subset (custom) | Leader election + majority-commit | No external library; bounded scope justified by static membership and no snapshot protocol need (Note: a dedicated `src/lib/raft/` in-repo library was later extracted from this inline logic; superseded by raft-library, see that spec's design.md) |
 | IPC | iomgr epoll | `EPOLLIN` + `EPOLLRDHUP` on control channel | `EPOLLRDHUP` for sub-second failover |
 | Persistence | Flat binary (`ha-term.bin`) | Term + voted_for across restarts | Atomic write+rename (existing pattern) |
 | Build | Meson + Nix | capnp schema regeneration | `src/schema/controller.capnp` |
@@ -321,12 +323,12 @@ Fallback trigger: heartbeat timeout vtask fires after `election_timeout_ms + ran
 | Requirement | Summary | Components | Key Interface | Flow |
 |-------------|---------|------------|---------------|------|
 | 1.1–1.4 | Listener + iomgr/vtaskmgr before election; dispatch gate for init/electing states | `cluster.c`, `ha.c`, `myapp.c` | `cluster_open_listener()`, vtaskmgr dispatch gate | Flow 1 |
-| 2.1–2.12 | Quorum election: voted_for enforcement, freshness gate, split-vote retry, single-node shortcircuit | `ha.c` | `am_i_primary_quorum()`, `handle_request_vote()`, `OP_requestVote` | Flow 2 |
+| 2.1–2.12 | Quorum election: voted_for enforcement, freshness gate, split-vote retry, single-node shortcircuit | `ha.c` | `am_i_primary_quorum()`, `handle_request_vote()` (Note: both superseded by `ha_start_election_async()`/raft library, `handle_request_vote()` now test-only — see election-iomgr-refactor), `OP_requestVote` | Flow 2 |
 | 3.1–3.5 | Pre-election config probe, deferred fetch, retry backoff | `config_repl.c`, `myapp.c` | `bootstrap_config_if_needed()`, `xce_probe_config_tag()`, `OP_probeConfigTag` | Flow 1 |
 | 4.1–4.4 | --vcpus CLI, even-N rejection, cert pre-flight, ASAN fix | `main.c`, `ha.c` | CLI argp, `access()`, `SSL_CTX_free` | Flow 1 |
 | 5.1–5.8 | Majority-commit WAL: committed_lsn, broadcast, ack-wait, backpressure, truncation | `persist.c`, `repl.c`, `ha.c` | `repl_wait_majority_ack()`, `persist_truncate_wal_above()`, `OP_walAck` | Flow 3 |
 | 6.1–6.8 | Persistent control channel: EPOLLRDHUP failover, state sync, catch-up, three-tier discovery | `ha.c`, `cluster.c`, `repl.c` | `ha_find_and_connect_to_primary()`, extended `OP_ctrlHello`, `OP_snapshotTransfer`, `OP_walCatchupEntry` | Flow 4 |
-| 7.1–7.6 | Client retry: backoff, redirect, non-retriable passthrough | `request.c` (libxce) | `xce_request_with_retry()`, `xce_retry_config_t`, redirect fields in `OP_requestVoteReply` | — |
+| 7.1–7.6 | Client retry: backoff, redirect, non-retriable passthrough | `request.c` (libxce) | `xce_request_with_retry()` (Note: removed, superseded by `client_failover.c` StatusClassifier/EndpointRotator — see libxce-failover), `xce_retry_config_t`, redirect fields in `OP_requestVoteReply` | — |
 | 8.1–8.5 | Majority-commit config push, committed_config_tag, heartbeat piggybacking | `config_repl.c`, `repl.c` | `push_config_to_standbys()`, `OP_upgradeConfigAck` | Flow 3 analog |
 | 9.1–9.3 | Term + voted_for durability across restarts via ha-term.bin | `ha.c` | `ha_persist_term()`, `ha_init()` read path | — |
 | 10.1–10.5 | Structured logs for election outcome, bootstrap, WAL commit, config push, FSM transitions | All new code paths | `xcelog.h` INFO/WARN macros | All flows |
@@ -365,6 +367,7 @@ Fallback trigger: heartbeat timeout vtask fires after `election_timeout_ms + ran
 - Single allocation for `ha_elect_ctx_t` via C99 flexible array — one `calloc`, one `free`
 - All role transitions via `atomic_store(&c->ha.role, ...)` — no lock held across transition
 - `ha_check_priority_stepdown()` and `ha_check_heartbeat_nocluster()` are deleted in this spec
+- (Note: `am_i_primary_quorum()`'s production implementation and `handle_request_vote()`/`handle_request_vote_reply()` were themselves later removed/demoted to test-only by election-iomgr-refactor — production election is now `ha_start_election_async()` → raft library; see that spec's design.md)
 
 **New `ha_state_t` fields**:
 ```c
@@ -431,7 +434,7 @@ case CTRL_PRIMARY:
 **Contracts**: Service [x] / State [x]
 
 **Implementation Notes**
-- Risk: `endpoint_connect()` has a blocking TCP connect phase. Vote request sends require per-peer pthreads (not pure iomgr async). Election context `ha_elect_ctx_t` accumulates replies; threads post to `ctx->done` semaphore when quorum is reached or all peers replied.
+- Risk: `endpoint_connect()` has a blocking TCP connect phase. Vote request sends require per-peer pthreads (not pure iomgr async). Election context `ha_elect_ctx_t` accumulates replies; threads post to `ctx->done` semaphore when quorum is reached or all peers replied. (Note: this risk was resolved by election-iomgr-refactor — vote sends are now pure async iomgr endpoints via `ha_start_election_async()`, not per-peer pthreads; `ha_elect_ctx_t` gained a refcount and several fields since, see ha-elect-ctx-lifecycle design.md)
 - Integration: `ha_wait_for_heartbeat_or_timeout()` uses `sem_timedwait(&c->ha.heartbeat_seen, &deadline)` where `deadline = now + election_timeout_ms + rand_jitter`. Heartbeat handler posts `heartbeat_seen` only when `ha.role == CTRL_INIT`.
 - `voted_for` enforcement: per-term, one vote per node. If `req->term > my_term`: update term, clear `voted_for_term`/`voted_for_name`. If `req->term < my_term`: deny. If already voted for a different candidate in same term: deny.
 
@@ -483,6 +486,7 @@ _Atomic uint64_t committed_lsn;  // highest majority-ACKed LSN; invariant: commi
 - Sends over persistent control channel fd (no new connection per entry)
 - On send failure: remove standby from active set, trigger catch-up reconnect, log WARN
 - Returns 0 when fanout initiated (not when ACKed)
+- (Note: signature and mechanism superseded — current `repl_broadcast(replicator_t *r, const wal_entry_hdr_t *hdr, const void *payload, uint32_t payload_len, uint64_t prev_log_term)` sends a per-entry AppendEntries carrying the leader's per-entry term for the raft library's log-replication reconcile, with NACK-driven retry; see raft-log-replication, ha-term-safety, raft-adapter-conformance design.md)
 
 **`repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op)` contract**:
 - Blocks (cond_timedwait) until `acked_count + 1 >= (n_active + 1) / 2`
@@ -490,6 +494,7 @@ _Atomic uint64_t committed_lsn;  // highest majority-ACKed LSN; invariant: commi
 - `ack_op` parameter selects `OP_walAck` vs `OP_upgradeConfigAck` to disambiguate concurrent commits
 - Returns 0 on quorum, `-ETIMEDOUT` on timeout
 - Single-node shortcircuit: `n_active == 0` → quorum immediately from own write (5.8, 8.5)
+- (Note: signature and mechanism superseded — current `repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn, uint32_t timeout_ms, uint8_t ack_op)` delegates the walAck track to a commit engine (`commit_submit`) rather than counting acks directly, and gates commit on `durable_lsn` (fdatasync-confirmed), not raw ACK count; see ha-commit-engine and ha-commit-durability design.md)
 
 **Contracts**: Service [x] / State [x]
 
@@ -511,6 +516,7 @@ _Atomic uint64_t committed_lsn;  // highest majority-ACKed LSN; invariant: commi
   2. Any peer has `tag > 0` → set `c->ha.defer_config_fetch = true`, return (3.2)
   3. No peer reachable → retry with exponential backoff 500ms→10ms cap 10s; log WARN each attempt (3.4)
 - Never returns an error; blocks until resolution or process killed
+- (Note: this full contract was later simplified/removed — the production `bootstrap_config_if_needed()` was removed in election-iomgr-refactor task 5.3; only a stub remains for linkage, mirroring the simplified two-branch logic now inlined ahead of `ha_start_election_async()` in myapp.c. See election-iomgr-refactor design.md)
 
 **`xce_probe_config_tag(const char *host, uint16_t port, uint64_t *tag_out)` contract**:
 - Single-RTT SSL request via `connect_ssl_raw()` + xce encode/decode
@@ -598,7 +604,7 @@ Standby side:
 5. Apply each `OP_walCatchupEntry` via `apply_wal_entry()`; send `OP_walAck` after each
 6. On catch-up loop end from primary: `ha.in_catch_up = 0`
 
-**Role flip on promotion** (6.8): standby closes and nulls `primary_ctrl_fd` before returning from `am_i_primary_quorum()`; new primary accepts inbound `OP_ctrlHello` connections in a per-standby fd table.
+**Role flip on promotion** (6.8): standby closes and nulls `primary_ctrl_fd` before returning from `am_i_primary_quorum()` (Note: this call site moved to the raft library's `on_become_leader` callback path, `raft_cb_become_leader`, since `am_i_primary_quorum()` was removed — see election-iomgr-refactor); new primary accepts inbound `OP_ctrlHello` connections in a per-standby fd table.
 
 **Contracts**: Service [x] / State [x]
 
@@ -611,6 +617,8 @@ Standby side:
 
 ### Client Retry (src/lib/request.c)
 
+**(Note: superseded by libxce-failover)** — `xce_request_with_retry()` described below was later removed as an "orphaned no-op" (per `src/lib/request.c` comment at the definition of `XCE_RETRY_DEFAULTS`); it was superseded by a `StatusClassifier`/`EndpointRotator` failover loop in `src/lib/client_failover.c`, driven from the xcecli layer. `xce_retry_config_t` and `XCE_RETRY_DEFAULTS` (the tunable struct/defaults — max_retries=20, initial_retry_ms=50, max_retry_ms=5000, backoff_multiplier=2.0) still exist verbatim in `src/include/request.h` and are consumed by the new failover loop. See libxce-failover design.md for the current mechanism.
+
 #### libxce Retry Wrapper
 
 | Field | Detail |
@@ -682,6 +690,7 @@ struct UpgradeConfigAck { term @0 :UInt64; ackedTag @1 :UInt64; success @2 :Bool
 struct SnapshotTransfer { snapshotLsn @0 :UInt64; payload @1 :Data; }
 struct WalCatchupEntry  { lsn @0 :UInt64; entryType @1 :UInt8; payload @2 :Data; }
 ```
+(Note: `WalCatchupEntry.payload` is now `List(UInt8)`, not `Data` — bug-1505 capnpc-c codec fix, unrelated to any listed spec)
 
 Also extend:
 - `CtrlHeartbeat`: add `committedLsn @N :UInt64` and `committedConfigTag @M :UInt64` for piggybacking (req 5.6, 8.4)
@@ -719,6 +728,7 @@ typedef struct {
 - WAL trim guard: threshold = `committed_lsn` not `applied_lsn`; prevents trimming entries a promoted standby may need for divergence resolution
 - `committed_lsn <= applied_lsn` asserted at promotion time; violated invariant is a programming error
 - `ha-term.bin` is written atomically before any vote reply is sent; a crash between write and reply means the term is persisted but the vote may not have been counted — conservative and safe
+- (Note: majority-commit via `committed_lsn` alone is superseded — ha-commit-durability added a `durable_lsn` field: a member counts toward commit majority only once its LSN is `fdatasync`-confirmed, not merely ACKed/appended; `committed_lsn` still exists but the true commit/promotion-safety floor is `persist_durable_floor()`. See ha-commit-durability design.md)
 
 ---