docs(specs): gap-analysis validation pass across all 9 HA-family specs

Appended "Gap Analysis — Validation Pass (2026-06-09)" to each spec's
research.md with requirement-to-asset maps, status tags, and effort/risk.

Key outcomes:
- ha-repl-hardening: REOPENED with task series 7.1-7.7 — replication
  payload data plane is missing (snapshot chunks and WAL catch-up ship
  header-only; standbys converge on LSN without holding entity state)
- worker-join-refactor, resval-id-decoupling: marked implemented
  (all criteria re-verified, full test suite green)
- buglog: 335/336 (scheduler dequeue, open), 337/338/339 (fixed in
  previous commit); cerebrum Decision Log corrected (term-only vote
  deny was recorded as intentional but was a regression)

.kiro/specs/ha-config-bootstrap/research.md

@@ -499,3 +499,214 @@ No new code needed. Proceed directly to manual integration test execution:
 3. Record results in the Results Template at the bottom of `manual-tests.md`.
 4. Mark tasks 11.1–11.5 `[x]` on pass.
 5. Run `nix build .#xcecore_debug` for 11.5 ASAN verification.
+
+---
+
+# Gap Analysis — Validation Pass (2026-06-09)
+
+## Scope and Method
+
+Validation pass against the working tree on branch `persistent_ha` (including uncommitted changes to `ha.c`, `ha.h`, `myapp.c`, `raft.c`). All 50 acceptance criteria across the 10 requirements were checked against on-disk code. Note that the architecture has evolved past this spec's original design via four successor specs (`raft-library`, `election-iomgr-refactor`, `ha-heartbeat-liveness`, `ha-repl-hardening`, `ha-peer-connection-reuse`): the blocking election (`am_i_primary_quorum` + semaphore wait) was replaced by a fully async raft-library-driven election inside iomgr, and the standby-pull config bootstrap was replaced by a primary-push model. Several "gaps" below are therefore *sanctioned deviations*, not defects — they need requirements-doc reconciliation, not code.
+
+All checkboxes in `tasks.md` (1.1–11.5) are marked `[x]`, including 9.3+, and manual integration tests 11.1–11.4 are recorded as executed 2026-06-03. The memory note "9.3+ remaining" is stale.
+
+## 1. Requirement-to-Asset Map
+
+Status legend: **Implemented** / **Partial** / **Missing** / **Unknown** / **Constraint** (sanctioned architectural deviation needing spec reconciliation).
+
+### Requirement 1 — Listener and Connection Framework Before Election
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 1.1 listener bound before election | Implemented | `myapp.c:113` `cluster_open_listener()` before `ha_start_election_async()`; `cluster_register_listener` at `myapp.c:138` |
+| 1.2 iomgr/vtaskmgr before election | Implemented | `myapp.c:122-131` iomgr+vtaskmgr init/start precede election (Step 3/4) |
+| 1.3 not-ready rejection in INIT/ELECTING | Implemented | `ha_dispatch_gate()` `ha.c:1412` returns `-XCE_ERR_NOT_READY`; wired at vtask dispatch entry `cluster.c:226` |
+| 1.4 standby error includes redirect host/port | **Partial** | `XCE_ERR_STANDBY` / `E_XCE_REDIRECT` returned (`ha.c:1481`, `join.c:241`), `ha_should_redirect()` exists, capnp fields `redirectHost/redirectPort` exist (`controller.capnp:565-566`) — **but no reply path ever populates `redirect_host/redirect_port`**; replies are sent with `data_which = nodata` |
+
+### Requirement 2 — Quorum-Based Controller Election
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 2.1 strict-majority quorum | Implemented | `raft.c:223-234` quorum = `n_peers/2 + 1`; `ha_elect_evaluate_and_signal` `ha.c:1995` |
+| 2.2 lower-term deny | Implemented | `raft_on_vote_request` Step 1 (`raft.c:110`) |
+| 2.3 higher-term update + clear vote | Implemented | `raft.c:121-130` (also steps a live leader down to follower) |
+| 2.4 freshness gate deny | Implemented | `raft_cb_vote_eligible` `ha.c:202`; `handle_request_vote` `ha.c:1791` |
+| 2.5 catch-up standby vote denied | Implemented | `in_catch_up` check `ha.c:218`, `ha.c:1845` |
+| 2.6 one vote per term | Implemented | `raft.c:157-164` (with idempotent re-grant to same candidate — safe extension) |
+| 2.7 yield to fresher peer | Implemented | yield gate `ha.c:2017-2031` (`max_resp_tag/max_resp_lsn`) |
+| 2.8 tag-zero candidate vs configured peer → fatal exit | **Constraint** | Implemented as *voter-only standby*, not fatal exit: `ha.c:3133-3145` (tag==0 && n_peers>1 → CTRL_STANDBY, wait for push). Strictly safer for ops (node self-heals via config push) but deviates from the EARS text "exit with a fatal error" |
+| 2.9 single-node immediate self-elect | Implemented | `raft.c:83-88` `RAFT_ELECTED_IMMEDIATELY`; zero-peer guard `ha.c:1985` |
+| 2.10/2.11 split vote → one retry w/ jitter+term++, then standby | **Constraint** | raft library becomes FOLLOWER on no-quorum/timeout (`raft.c:237-244`, `raft_on_election_timeout`); retries are driven indefinitely by the standby liveness watchdog → `raft_on_leader_lost` → delayed `raft_check_and_start_election` (`raft.c:268-290`). Net behavior: unbounded jittered retry loop instead of exactly-once retry. Liveness outcome equivalent or better; spec text outdated |
+| 2.12 identical-freshness tiebreak by jitter, not priority | **Partial** | Winner choice never uses priority values; but election *timing* uses deterministic priority-based stagger (`myapp.c:229-239` `priority * election_priority_step_ms`; `raft.c:272-273` `priority * election_timeout_ms`). Priority is repurposed as stagger index, not vote weight — spec wording should be reconciled |
+
+### Requirement 3 — Config Bootstrap for Unconfigured Nodes
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 3.1 probe all peers' config tag before candidacy | **Missing (as specified)** | `xce_probe_config_tag()` fully implemented (`config_repl.c:1140`, single-RTT SSL probe) and unit-tested, but **never called in production** — `bootstrap_config_if_needed()` was removed from `myapp_run` (see Constraints in prior pass). Replacement: unconditional voter-only gate when `my_config_tag==0 && n_peers>1` (`ha.c:3133`) |
+| 3.2 voter-only participation, deferred fetch | Implemented (pivoted) | voter-only path `ha.c:3133-3145`; config arrives via primary push (`push_config_to_standbys` from `raft_cb_become_leader`/COMMITTING driver) |
+| 3.3 all peers tag-zero → normal candidacy | **Partial / Unknown** | No probe means a tag-0 node cannot distinguish "peers have config" from "all-zero cold start"; it always goes voter-only first. Recovery relies on the heartbeat watchdog firing → `ha_on_primary_ctrl_fd_rdhup` → `raft_on_leader_lost` re-election, which **bypasses** the tag-0 gate (`ha.c:3183-3291`). All-zero cold start thus converges, but only after `heartbeat_timeout_ms` (~10s) delay per round; not covered by an automated test |
+| 3.4 exponential-backoff probe retry 500ms→10s | **Missing (as specified)** | No probe retry loop exists; the passive watchdog loop replaces it. Spec text obsolete under the push model |
+| 3.5 deferred fetch from proven winner | Implemented (pivoted) | Primary pushes config + ctrlhello to all standbys on COMMITTING (`ha.c:2087-2103`); `fetch_config_from_primary`/`do_one_config_fetch` standby-pull removed by design (CLAUDE.md: standbys never dial out) |
+
+### Requirement 4 — Startup Validation and Safety Checks
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 4.1 even-N peer rejection | Implemented | `validate_peer_count()` `startup_validation.c:90`; double-enforced in `raft_init` (`raft.c:21`) |
+| 4.2 cert/key pre-flight with full path | Implemented | `validate_cert_files()` `startup_validation.c:106` (-ENOENT, path in message), wired in `main.c` |
+| 4.3 vcpus >= 1 | Implemented | `validate_vcpus()` `startup_validation.c:25`; `--vcpus` argp option in `main.c` |
+| 4.4 config-file vcpus deprecation WARN | Implemented | `validate_startup_config()` `startup_validation.c:46`; warned at `myapp.c:80-86` |
+
+### Requirement 5 — Majority-Commit WAL Replication
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 5.1 job submit / state transitions / queue changes majority-committed | **Partial** | Only `WAL_JOB_SUBMIT` routes through `persist_wal_sync_append()` (`persist.c:1118`). Job state transitions (`WAL_JOB_START/FINISH/FAIL/...`, `persist.c:1163`) and queue changes (`WAL_QUEUE_CTRL`, `persist.c:1194`) use `persist_wal_async()` — no majority wait |
+| 5.2 host state / checkpoint refs async | Implemented | worker/host events → `persist_wal_async`; checkpoint markers never replicated (`persist.c:395`) |
+| 5.3 standby durable write + ACK with LSN | Implemented | `OPCode_walAck` flow, `repl_on_ack` (`repl.c:178`), `acked_lsn` tracking |
+| 5.4 lag eviction + catch-up trigger | Implemented | `pending_ack_count > max` eviction with inline catch-up start, `repl.c:214-222` |
+| 5.5 commit timeout → error to caller | **Constraint** | Deliberate *degraded mode*: `persist_wal_sync_append` reports `-ETIMEDOUT/-ENODEV` via `sync_rc_out` but returns 0 and the operation proceeds with local-only durability (`persist.c:1003-1043`; hook comment `persist.c:1114`). This is an AP-leaning deviation from the spec's CP statement — documented in code, not in requirements |
+| 5.6 truncate above committed LSN on promotion | Implemented | `ha_compute_safe_committed_lsn()` from voter replies (`ha.c:373-437`) → `persist_truncate_wal_above(safe_lsn)` in `raft_cb_become_leader` (`ha.c:471`); per controller CLAUDE.md this is the verified-against-TLA+ design |
+| 5.7 truncate on catch-up reconnect | **Partial** | RDHUP/watchdog failover path truncates to `last_known_committed_lsn` before re-election (`ha.c:3204-3207`). A standby that *restarts* (rather than detecting RDHUP) replays its full WAL including uncommitted tail; no truncation occurs before it rejoins and receives catch-up entries — divergent-tail handling on the restart path is unverified |
+| 5.8 single-node local write satisfies majority | **Partial** | `repl_wait_majority_ack` returns 0 when `n_active == 0` (`repl.c:384`), but `persist_wal_sync_append` short-circuits *before* calling it when `nstandbys == 0` and sets `sync_err = -ENODEV` **without advancing `committed_lsn`** (`persist.c:1023-1031`). Single-node `committed_lsn` appears to advance only via the election-time safe-LSN path — runtime advancement per entry is not evident. Needs verification (interacts with WAL trim guard, which uses `committed_lsn` as threshold) |
+
+### Requirement 6 — Persistent Standby-to-Primary Control Channel
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 6.1 standby establishes persistent TLS connection to primary | **Constraint** | Direction inverted by deliberate design (controller CLAUDE.md "Inbound vs. Outbound Connection Ownership"): the **primary dials out** to standbys; standby-side `ha_find_and_connect_to_primary()` is dead code (declared `ha.h:741`, never called in production). The functional goal (persistent channel, no per-RPC connections) is achieved |
+| 6.2 disconnect → immediate election | Implemented | EPOLLRDHUP on `primary_ctrl_fd` → `ha_on_primary_ctrl_fd_rdhup()` (`ha.c:3183`) |
+| 6.3 heartbeat resets timeout | Implemented | `ha_on_heartbeat_received()` on every NOP/ctrlhello (`cluster.c:249, 283`) |
+| 6.4 heartbeat timeout + jitter → election | Implemented | `ha_arm_heartbeat_timeout_vtask` liveness poll; manual test 11.2 measured ~11s detection (configured timeout, not sub-second — sub-second applies to the RDHUP path) |
+| 6.5 sync missing state before admitting to replication set | Implemented | heartbeat-reply-driven driver: standby `stateReport` → `ha_repl_decide()` (`ha.c:1157-1176`) → stream WAL / snapshot / config; `in_catch_up` gates quorum membership (`repl.c:367`) |
+| 6.6 snapshot when standby older than oldest WAL segment | Implemented | `standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT` (`ha.c:1176`), `ha_send_snapshot_to_standby()` chunked transfer (`config_repl.c:1578`), standby handler `cluster.c:182` |
+| 6.7 config push after catch-up on tag mismatch | Implemented | same driver pushes config on tag divergence; `push_config_to_standbys` |
+| 6.8 winner closes outbound, accepts inbound | Implemented | role flip handled in `raft_cb_become_leader` / RDHUP path; surviving vote eps promoted to `standby_endpoints`; late joiners get fresh outbound connects from `primary_heartbeat_fn` |
+
+### Requirement 7 — Client-Side Transparent Retry (libxce)
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 7.1 NOT_READY → backoff retry | Implemented | `xce_request_with_retry()` `request.c:1664`; tests `test_libxce_retry.c` |
+| 7.2 STANDBY + redirect → follow redirect | **Partial** | Client logic implemented and unit-tested via hook, but **no controller reply ever carries `redirect_host/port`** (see 1.4) — end-to-end the redirect-follow path can never trigger; falls back to backoff |
+| 7.3 TCP refused/timeout → reconnect retry | Implemented | `request.c` retry classification |
+| 7.4 non-retriable surfaced immediately | Implemented | AUTH/PERM/INVALID/NOENT surfaced without retry |
+| 7.5 configurable params + defaults | Implemented | `xce_retry_config_t`, `XCE_RETRY_DEFAULTS` (`request.h:140-150`) |
+| 7.6 thundering-herd jitter | Implemented | jittered exponential backoff; 50-client concurrency test |
+| (adoption) | **Missing** | Zero production call sites: worker, scheduler, and apiserver backend never call `xce_request_with_retry` — the wrapper is shipped but unused |
+
+### Requirement 8 — Majority-Commit Config Replication
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 8.1 majority commit for config push | Implemented | `push_config_to_standbys` → `repl_wait_majority_ack(..., OPCode_upgradeConfigAck)` (`config_repl.c:905`) → `committed_config_tag` advanced (`config_repl.c:908`) |
+| 8.2 standby durable write + ACK | Implemented | `handle_repl_config_push` + `OP_upgradeConfigAck`; `acked_config_lsn` tracked separately from WAL acks (`repl.c:375-380`) |
+| 8.3 timeout → error | Implemented | `-ETIMEDOUT` from `repl_wait_majority_ack`, WARN logged |
+| 8.4 promoted standby serves only committed config tag | **Partial** | `last_known_committed_config_tag` captured from CtrlHelloReply (`cluster.c:261`) and `committedConfigTag` piggybacked on heartbeat (capnp); `committed_config_tag` consulted at `ha.c:2514-2521` and safe-tag computed on promotion (`ha.c:485`). Enforcement at *serve time* (refusing to serve a fresher-but-uncommitted config) not explicitly verified |
+| 8.5 single-node local write = majority | Implemented | safe-tag path on become-leader; `committed_config_tag` advanced on push without standbys |
+
+### Requirement 9 — Election State Durability
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 9.1 persist term+vote before reply | Implemented | raft `on_persist` fires before any grant/RPC (`raft.c:77, 176`); `ha_persist_term()` atomic write+rename with magic+CRC32 (`ha.c:1647-1681`) |
+| 9.2 restore on startup | Implemented | `ha_restore_term()` (`ha.c:1683+`); manual test 11.4 confirmed `restored term=2` on restart |
+| 9.3 missing file → term 0, no vote | Implemented | missing/short-read/bad-magic/CRC-mismatch all degrade safely to term 0 with WARN (`ha.c:1715-1737`) |
+
+### Requirement 10 — Observability
+
+| AC | Status | Assets / Evidence |
+|----|--------|-------------------|
+| 10.1 election outcome: term + votes granted + quorum size | **Partial** | FSM transition INFO logs carry term (`ha.c:507, 659, 870, 881, 2080, 2158`); majority-ack log carries acks+quorum (`ha.c:2929`); but no single election-complete record logs *votes granted* and *quorum size* together |
+| 10.2 bootstrap probe finding | **Partial** | voter-only decision logged (`ha.c:3134`); per-peer probe logs exist in `xce_probe_config_tag` but that path is unused in production |
+| 10.3 WAL quorum INFO / timeout WARN | Implemented | `repl.c:408-410` |
+| 10.4 config quorum INFO / timeout WARN | Implemented | `config_repl.c` push path logs both |
+| 10.5 FSM transition INFO | Implemented | all transitions logged, including RDHUP (`ha.c:3199`) |
+
+### Summary Tally (by requirement)
+
+| Req | Verdict |
+|-----|---------|
+| R1 | Implemented except redirect payload (1.4 Partial) |
+| R2 | Implemented; 2.8/2.10/2.11/2.12 are sanctioned deviations needing spec reconciliation |
+| R3 | Pivoted to primary-push; 3.1/3.4 Missing-as-specified, 3.3 Unknown (cold-start convergence untested) |
+| R4 | Implemented (4/4) |
+| R5 | Core implemented; 5.1 coverage gap, 5.5 CP→degraded-mode deviation, 5.7 restart-path gap, 5.8 needs verification |
+| R6 | Implemented; 6.1 direction inverted by design (Constraint) |
+| R7 | Library implemented; redirect payload + production adoption Missing |
+| R8 | Implemented; 8.4 serve-time enforcement Partial |
+| R9 | Implemented (3/3) |
+| R10 | Implemented except consolidated election-outcome record (10.1) |
+
+**Net: ~40 of 50 acceptance criteria fully implemented; 7 partial; 3 missing-as-specified (all three superseded or deliberately pivoted).** No requirement is wholly unimplemented.
+
+## 2. Remaining Gaps — Concrete List
+
+1. **G1 — Redirect payload never populated** (R1.4, R7.2): capnp fields exist; standby replies send `nodata`. Client redirect-follow logic is dead end-to-end.
+2. **G2 — WAL majority-commit covers only job submission** (R5.1): state transitions and queue changes are async-only; a failover can lose acknowledged state changes.
+3. **G3 — Commit timeout is degraded-mode, not an error** (R5.5): spec promises CP (timeout error to caller); code chooses availability. Needs an explicit, documented decision.
+4. **G4 — Single-node `committed_lsn` runtime advancement unverified** (R5.8): `-ENODEV` short-circuit skips the advance; interacts with the WAL trim guard.
+5. **G5 — Standby restart path skips divergent-tail truncation** (R5.7): truncation happens only in the RDHUP/watchdog path.
+6. **G6 — Config probe path unwired** (R3.1/3.3/3.4): `xce_probe_config_tag` is production-dead; all-zero cold-start converges only via watchdog timeout and lacks a test.
+7. **G7 — libxce retry has zero production adopters** (R7 objective): worker/scheduler/apiserver still have whatever ad-hoc behavior they had.
+8. **G8 — Observability**: no consolidated election-outcome log with vote tally + quorum size (R10.1).
+9. **G9 — Spec/code reconciliation debt**: 2.8 (standby instead of fatal exit), 2.10/2.11 (watchdog retry loop instead of retry-once), 2.12 (priority-based stagger), 3.x (push model), 5.5 (degraded mode), 6.1 (inverted channel direction) — all stable code behavior contradicting EARS text.
+10. **G10 — Cleanup**: `ha_find_and_connect_to_primary` dead code; bounded per-RDHUP `elect_ctx` leak (`ha.c:3252-3258`, documented UAF-avoidance).
+
+## 3. Implementation Approach Options
+
+### Option A — Extend existing components in place (code-first)
+Close G1–G8 directly in `ha.c` / `persist.c` / `repl.c` / `join.c` / worker+scheduler clients; patch requirements.md text afterward.
+- Files: `join.c`+`reply.c` (G1), `persist.c` hook routing (G2/G4), `cluster.c` startup replay (G5), `myapp.c`+`config_repl.c` (G6), worker/scheduler request sites (G7), `ha.c` logging (G8).
+- ✅ No new modules; follows established patterns (every gap has an existing seam and a testsuite file).
+- ✅ Fastest path to closing the spec.
+- ❌ G2/G3 are *policy* changes with latency and availability consequences — landing them silently inside this aging spec hides a significant decision.
+- ❌ `ha.c` is already 3,818 lines; further growth degrades maintainability.
+
+### Option B — Spec-first reconciliation, then targeted follow-up spec
+Amend requirements.md/design.md to match the as-built architecture (push bootstrap, inverted channel, watchdog retry, degraded mode), close only the unambiguous small defects (G1, G4, G8, G10) here, and move G2/G3/G5/G7 into the active `ha-repl-hardening` spec (or a new `ha-cp-commit` spec) with their own requirements review.
+- ✅ Honest record: the EARS text stops contradicting shipped behavior; the CP-vs-degraded decision gets explicit operator-facing review.
+- ✅ Keeps this spec closeable; G2/G3 get proper TDD treatment in a spec that already owns repl hardening.
+- ❌ More process; durability gap (G2) stays open until the follow-up lands.
+- ❌ Risk that "deferred" becomes "dropped" without tracking.
+
+### Option C — Hybrid (recommended)
+Phase 1 (this spec, S-size code): G1 redirect payload, G4 single-node commit advance verification/fix, G8 election-outcome log, G10 cleanup, plus requirements.md deviation annotations (G9) and a cold-start integration test for G6's watchdog fallback.
+Phase 2 (`ha-repl-hardening` spec): G2 (extend sync-append coverage to state transitions/queue changes — likely with group-commit batching to bound latency), G3 (explicit CP-vs-degraded decision + config knob), G5 (restart-path truncation), G7 (retry adoption across daemons).
+- ✅ Ships every low-risk fix now; routes every policy-bearing change through review.
+- ✅ Matches the existing spec topology (repl-hardening already owns quorum/commit semantics).
+- ❌ Requires coordinating two specs; G6's probe-vs-watchdog question must be answered during reconciliation (recommend: delete the probe path, document the watchdog fallback, keep the test).
+
+## 4. Effort & Risk (remaining work)
+
+| Gap | Effort | Risk | Justification |
+|-----|--------|------|---------------|
+| G1 redirect payload | S | Low | Fields exist in schema/typemap; populate in 2 reply sites + 1 e2e test |
+| G2 sync-append coverage | M | Medium | Mechanical routing change but puts majority-RTT on hot job-state path; needs batching design + latency measurement |
+| G3 CP-vs-degraded decision | S (code) / M (decision) | High | One-line behavior flip, but it trades availability for consistency cluster-wide; needs operator sign-off and failure-mode tests |
+| G4 single-node committed_lsn | S | Medium | Small fix, but wrong answer corrupts WAL trim guard invariant (`committed_lsn <= applied_lsn`) |
+| G5 restart-path truncation | M | Medium | Touches startup replay ordering; must honor TLA+ safe-commit-point rule (never truncate via stale ctrlhello value) |
+| G6 cold-start test + probe removal | S | Low | Behavior already converges via watchdog; work is test + dead-code deletion + doc |
+| G7 retry adoption | M | Medium | Multi-daemon change; behavioral shift in error surfacing for worker/scheduler/apiserver |
+| G8 election-outcome log | S | Low | Data already in `ha_elect_ctx_t`/raft state at decision points |
+| G9 spec reconciliation | S | Low | Documentation only; six identified deltas |
+| G10 cleanup | S | Low | Dead-code deletion + bounded-leak fix behind existing tests |
+
+Aggregate remaining: **~M total for Phase 1 (all S items), ~L for Phase 2** (G2+G3+G5+G7 under ha-repl-hardening).
+
+## 5. Recommendations
+
+1. **Adopt Option C.** Close this spec with Phase 1; transfer G2/G3/G5/G7 into `ha-repl-hardening` as explicit requirements so they get their own approval gate.
+2. **Reconcile requirements.md now** (G9): rewrite AC 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 to describe the as-built push/watchdog/outbound-primary architecture. The controller CLAUDE.md and TLA+ mapping are already correct — the spec is the lagging artifact.
+3. **Decide G3 deliberately.** The spec's headline promise is "zero data loss on failover" (CP); the code ships degraded-mode availability. Either add a `commit_strict` config knob defaulting per the decision, or amend the spec's CP claim. Do not leave the contradiction implicit.
+4. **Fix G1 immediately** — it is the cheapest fix with end-to-end user impact (failover redirect for clients), and it unblocks the already-tested libxce retry path.
+5. **Delete dead code** (`ha_find_and_connect_to_primary`, production `xce_probe_config_tag` path if reconciliation confirms the push model) rather than leaving two competing bootstrap narratives in the tree.
+
+### Research Needed
+
+- **RN1**: Does single-node `committed_lsn` ever advance at runtime (G4)? Trace `persist_wal_sync_append` + WAL trim guard in a single-node soak; check whether `ha-repl-hardening` task 6.5 already addressed this.
+- **RN2**: Latency budget for extending majority-commit to job state transitions (G2) — measure per-entry RTT on 3-node LAN; evaluate group commit / pipelined acks before committing to a design.
+- **RN3**: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether `apply_wal_entry` overwrite semantics mask or corrupt the divergent tail.
+- **RN4**: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary.
+- **RN5**: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above `committed_config_tag` in any window between promotion and first quorum push?