myxce/.kiro/specs/ha-term-safety/research.md

# Research & Design Decisions

## Summary

- **Feature**: `ha-term-safety`
- **Discovery Scope**: Complex Integration (controller-side adapter + config marker + out-of-band config-push byte delivery + live regression umbrella for the Raft TLA+ Conformance library series; the configuration-upgrade materialized-state snapshot is descoped to `config-upgrade-snapshot`).
- **Key Findings**:
  - The WAL on-disk header **already carries** `wal_entry_hdr_t.term` at offset 0 of a 40-byte layout (`src/include/wal_types.h:29-39`). R1.7 ("preserve on-disk layout, value fix only") is satisfied by the format as-is. The defect is purely that `persist_t.term` (`src/include/persist.h:160`) is **never assigned** anywhere — every `hdr.term = p->term` site (`persist.c:588, 726, 1040`) therefore stamps `0`. The fix is a `persist_set_term` seam fed by the library's term, plus reading `hdr.term` back on replay (`replay_wal` reads but discards it today, `persist.c:~1500-1620`).
  - The controller uses the **new library** `src/lib/raft/raft.c` exclusively (no legacy `src/controller/raft.c` exists); ten election callbacks are already wired via `s_real_cbs` (`ha.c:1309-1320`). The log/commit callbacks defined by the three upstream specs (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) are **not yet wired** — that wiring is this spec's core deliverable.
  - **G5 (term not adopted on the replication/heartbeat path) is confirmed in code**: only `raft_on_vote_request`/`reply` route a higher term into raft state (`raft.c:125-134` → `raft_cb_persist` → `ha_persist_term`, `ha.c:1251-1294, 2441-2474`). The heartbeat consume path (`ha_consume_heartbeat_committed`, `ha.c:5013-5036`) only *drops* stale heartbeats; it never adopts a higher term. With the library now owning `UpdateTerm`-on-any-RPC, the controller must route incoming heartbeat/replication RPCs through `raft_on_append_entries_request`.
  - **G3 (reconcile is committed-boundary, not `(term,lsn)`) is confirmed**: `ha_rejoin_reconcile` (`ha_convergence.c:295-366`) clamps the truncation cutoff to `committed_lsn` and truncates the suffix blindly via `persist_truncate_wal_above` (LSN-only, `persist.c:2455-2565`). It cannot detect same-LSN/different-term divergence. The library's `raft_on_append_entries_request` + `on_truncate_from` replace this decision.
  - **R8 SNAPSHOT FEASIBILITY → RESOLVED BY DESCOPE (2026-06-21)**: the snapshot-feasibility investigation found that snapshots are a **16-byte LSN sentinel stub** today (`"XCESNAP\0"` + checkpoint_lsn; `write_snapshot` `persist.c:1068-1101`, `load_snapshot` `persist.c:1280-1307`) carrying **zero entity state**; the COW machinery a full upgrade snapshot needs (`xzarray_freeze`/`xzarray_thaw`, `zarray.c:695-757`) is built and unit-tested but **dormant** (`kvmgr.c:297-298` `vfreeze = NULL /* wired in P0-2 */`, zero production callers); the roadmap's **"entity-bearing snapshots (Task 11)"** (`roadmap.md:43`) is unstarted. Because the full configuration-upgrade materialized-state snapshot is a **HARD dependency on entity-bearing-snapshots that does not exist**, the former Requirement 8 was **descoped into the independent `config-upgrade-snapshot` spec** (which HARD-depends on `entity-bearing-snapshots`). `ha-term-safety` ships R1–R7 with **no snapshot dependency**: R7's config **marker** (consensus) is in-log, and the config **bytes** are delivered via the existing **out-of-band config-push transport** (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. The config marker never depended on entity-bearing snapshots; only the descoped upgrade snapshot did.

## Research Log

### Upstream library callback contract (the seam this spec wires to)

- **Context**: This spec must delegate the Raft protocol to the library and wire the controller as a thin adapter, reusing the upstream callback names verbatim.
- **Sources Consulted**: `.kiro/specs/raft-log-replication/design.md`, `.kiro/specs/raft-commit-safety/design.md`, `.kiro/specs/raft-sim-conformance/design.md`, `TLA-gap-analysis.md`.
- **Findings**:
  - `raft-log-replication` owns: `raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`, `raft_append_result_t`; callbacks `on_persist_entry(ctx,index,term,entry,len)`, `on_get_entry_term(ctx,index)->term|0`, `on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)`, `on_truncate_from(ctx,index)`; per-peer `next_index[]`/`match_index[]`; **index 1-based == WAL LSN**; **term 0 == absent**; `UpdateTerm` applied on every RPC (G5); conflict branch = truncate-then-append in one handler call (G3).
  - `raft-commit-safety` owns: `raft_advance_commit(s)` with the Figure-8 current-term rule (G4); `on_commit_advanced(ctx,commit_index)` fired only on strict advance — **triggers** the existing barrier, never replaces it; vote/PreVote freshness becomes `(lastLogTerm,lastLogIndex)` lexicographic (G6), with the candidate's `last_log_term` added to `raft_on_vote_request`/`raft_on_pre_vote_request` signatures.
  - `raft-sim-conformance` owns the in-process oracle (Election Safety / Log Matching / Leader Completeness / State Machine Safety) — the verification cross-reference; the live regression here (R6) **complements** it (the sim proves the library; the live test proves the controller adapter wiring).
- **Implications**: Every R1–R6 protocol invariant is realized by the library; ha-term-safety implements the **adapter callback bodies** + the term plumbing on the wire + the reconcile delegation + the live regression. The controller must NOT re-implement `logOk`, conflict truncation, the current-term commit rule, or freshness — those live in `raft.c`.

### WAL term stamping & persistence seam (R1, R2)

- **Sources Consulted**: `wal_types.h:29-59`, `persist.h:160`, `persist.c:588/726/1040/1500-1620/2455-2565`, `typemap.h:338-387`, `repl.c:239-311`.
- **Findings**:
  - `wal_entry_hdr_t` is 40 bytes, `_Static_assert`-locked, `term` already present at offset 0 → **R1.7 is free**.
  - `persist_t.term` exists but `grep "p->term ="` / `"->term ="` returns **no assignment** in persist.c → every entry stamps term 0 (G1). Fix: a `persist_set_term(p, term)` seam called from the adapter when the library establishes/adopts a term.
  - Replication transport: the active wire format is `wal_catchup_entry_t` (`typemap.h:338-351`) which has **no term field**; the term currently rides implicitly inside the serialized header bytes. `capnp_repl_envelope_t.term` exists (`typemap.h:380-387`) but is a test-only mapped type. The adapter's `on_send_append_entries` must carry `prevLogTerm` + the entry's term explicitly so a standby persists the leader's term, not its local `p->term` (closing the `persist_wal_write_replicated` "best-effort p->term" hazard at `persist.c:719-726`).
  - `replay_wal` reads `hdr.term` but discards it; recovery must surface per-entry term to feed `on_get_entry_term` and restore `initial_log_len`/last-log-term for the library at init.
  - Next free WAL type id: `WAL_NOOP=40`, `WAL_CHECKPOINT=100` → `WAL_CONFIG_MARKER = 50` slots cleanly (R7).

### Term adoption, reconcile, commit, promotion seams (R2, R3, R4, R5)

- **Sources Consulted**: `ha.c:113-1320, 480-509, 880-1033, 5013-5036, 5338-5402`, `ha_convergence.c:240-384`, `commit.c:79-395`, `cluster.c:1200-1299`, `myapp.c` standby branch, `persist.c:691-706`.
- **Findings**:
  - **R2/G5**: heartbeat/replication never adopt a higher term (`ha_consume_heartbeat_committed` only drops stale). Routing incoming heartbeat/repl through `raft_on_append_entries_request` makes the library do `UpdateTerm`; the adapter's `on_persist` (already `ha_persist_term`) durably records it. Monotonic-term durability before appending as leader (R2.1/2.2) is realized by the library's `BecomeLeader` + the own-term `WAL_NOOP` gate already present (`raft_cb_become_leader` arms `commit_set_writable_lsn`, `ha.c:880-1033` / `commit.c:383-395`).
  - **R3/G3**: `ha_rejoin_reconcile` committed-boundary clamp replaced by routing the leader's AppendEntries through `raft_on_append_entries_request`; the library decides match/truncate/append and calls `on_truncate_from` (adapter → `persist_truncate_wal_above`) + `on_persist_entry`.
  - **R4/G4**: `ha_compute_safe_committed_lsn` (`ha.c:480-509`) is term-blind (collects voter LSNs/tags, no term). It is superseded by the library's `commit_index` (current-term-gated) delivered via `on_commit_advanced`, which **triggers** the existing synchronous barrier. The barrier and durability (`majority_candidate_locked` self-caps at `persist_durable_lsn`, `commit.c:79-138`; persist-before-ACK; `wait_done_locked` own-term gate) are PRESERVED AS POLICY (ha-commit-durability) — only the *input* (safe commit position) changes source.
  - **R5**: clean-failover behaviors to preserve — recovery-on-promotion + truncate-to-safe (`raft_cb_become_leader`, `ha.c:917-919`), pre-promotion eligibility fence (`ha.c:933-951`, bugs 848/849), advertise-real durable floor for freshness (`raft_cb_vote_eligible`, `ha.c:299-360`), and the contiguity guard refusing a forward gap (`persist_wal_write_replicated`, `persist.c:691-706`, bug-850). These remain; the library's `(term,index)` matching is strictly additive on top.

### Config replication seam (R7)

- **Sources Consulted**: `config_repl.c:941-979, 1001-1163, 1857-1944`, `ha.c:5338-5402, 4980-5036`, `ha_convergence.c:372-384`.
- **Findings**: config commits today through a **separate** `committed_config_tag` index (`push_config_to_standbys` waits majority ACK then `atomic_store committed_config_tag`, `config_repl.c:965-970`), carried on heartbeats (`ha.c:5343-5345`), with **no term** and tag-based anti-entropy — entirely outside the WAL `(term,lsn)` log. This is the second parallel violation. R7 replaces it: a `WAL_CONFIG_MARKER` entry (version_tag + sha256, **not bytes**) appended via `raft_log_append`, ordered/committed/reconciled in the same `(term,lsn)` log; committed config derived from the latest committed marker within the committed prefix (replaces the separate index). Config **bytes** continue to travel via the **existing out-of-band config-push transport** (`OPCode_replConfigPush`), now keyed to the marker's `{version_tag, sha256}` and hash-verified against the committed marker before activation — never in the WAL record. (The reverted byte-delivery decision: the prior draft routed bytes through a full upgrade snapshot; that snapshot is descoped to `config-upgrade-snapshot`, so the byte transport reverts to the existing config-push path.)

### Live regression infrastructure (R6)

- **Sources Consulted**: `src/controller/testsuite/waldump.{c,h}`, `test.c:129-1760`, `.kiro/specs/ha-commit-durability/repro-851/`, `INVESTIGATION-failover-job-loss.md`, `test_commit_barrier_rollout.c`, `src/tools/deploy-ha.sh`, `test-commit-durability.sh`, `roadmap.md:55`.
- **Findings**: Infrastructure is ~90% present.
  - `waldump_path()` (`waldump.h:42`) already decodes per-entry `(LSN, TERM, TYPE, PLEN, PCRC)` and validates the CRC chain, with a payload-CRC fingerprint for cross-member comparison and nonzero exit on gaps/crc-break.
  - `xce_test` script primitives exist with exit-code verdicts: `jobsub` (captures `$jobid`), `wait-primary`, `expect-jobs N`, `expect-job $jobid`, `ctrlstatus`, `waldump`, `!shell` escape, `$?`/`$jobid` vars, `g_session_failed` → process exit code (`test.c:129-1760`).
  - `/xce-deploy-ha` + `deploy-ha.sh`/`start.sh` launch a 3-node cluster; `kill_node`/`start_node` + deterministic `--wal-sync-ms 30000` drive chaotic kills (`test-commit-durability.sh:113-198`).
  - **Gaps to build**: (a) `waldump` currently prints `TERM=0` for all entries because of G1 — once R1 lands it will show real terms; the R6 test must assert per-LSN term identity across members; (b) a `walcompare <dirA> <dirB>` cross-member `(term,lsn,pcrc)` divergence diff (already on the roadmap as planned tooling, `roadmap.md:55`) is the missing assertion primitive. R6 builds the chaotic-multi-kill script + `walcompare`, reusing all existing primitives.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| **Thin callback adapter (chosen)** | Controller implements the upstream callback ports over WAL/repl/commit; library owns all protocol decisions | Honors the TLA-gap-analysis DECISION; one consistency domain; no protocol logic duplicated in controller; sim already proves the library | Requires deleting/neutralizing the term-blind controller logic (`ha_compute_safe_committed_lsn`, committed-boundary reconcile) rather than extending it | Matches Ports-and-Adapters already established by the election callbacks |
| Patch the controller's split log (rejected) | Add `(term,lsn)` matching directly into `ha_convergence.c` + `ha_compute_safe_committed_lsn` | No library dependency | Re-implements raft.tla in the controller; two consistency domains; cannot be sim-verified; explicitly rejected by the 2026-06-21 DECISION | Rejected |
| Config bytes in WAL (rejected) | Carry config bytes in `WAL_CONFIG_MARKER` payload | Self-contained log | Bloats the log; R7.2/R7.4 forbid it | Rejected by requirements |

## Design Decisions

### Decision: Unify R1–R6 protocol onto the library; controller is a thin adapter

- **Context**: TLA-gap-analysis DECISION (2026-06-21) mandates the library own the protocol; the controller adapts.
- **Alternatives Considered**: (1) thin adapter (chosen); (2) patch controller split-log (rejected — duplicates protocol, unverifiable).
- **Selected Approach**: Wire five new library callbacks to controller I/O sites; route incoming heartbeat/replication through `raft_on_append_entries_request`; drive append via `raft_log_append`; replace the term-blind safe-commit and committed-boundary reconcile with the library's `commit_index`/`on_truncate_from`. No protocol logic is added to the controller.
- **Rationale**: Single consistency domain; the deterministic sim already verifies the protocol, so the live regression need only prove the *wiring*.
- **Trade-offs**: Must neutralize existing term-blind controller paths (a deletion, not an extension); upstream signature changes are revalidation triggers.
- **Follow-up**: Confirm the upstream callback/entry-point signatures are stable before wiring (Revalidation Triggers below).

### Decision: Generalize job-state and config-change onto one `(term,lsn)` replicated log (R7)

- **Context**: Config is a second parallel consistency domain (`committed_config_tag`, no term) exposed to the same divergence.
- **Selected Approach**: Record each config change as a lightweight `WAL_CONFIG_MARKER` (version_tag + sha256, no bytes) appended via `raft_log_append` so it inherits term-safety, reconcile, and commit. Derive committed config from the latest committed marker; retire the separate `committed_config_tag` commit index.
- **Rationale**: One log, one reconcile, one commit rule — the Generalization lens: config change is a special case of "an ordered committed log entry".
- **Trade-offs**: Config bytes are delivered out-of-band via the existing config-push transport (`OPCode_replConfigPush`), hash-verified against the committed marker before activation — no snapshot dependency. During migration, `committed_config_tag` becomes a derived value, not an authoritative index.
- **Follow-up**: R7's marker is implementable now (no entity-bearing snapshots needed); the config bytes ride the existing config-push install path. The full materialized-state upgrade snapshot is descoped to `config-upgrade-snapshot` — see the resolved decision below.

### Decision (RESOLVED 2026-06-21): former R8 descoped into `config-upgrade-snapshot`; R7 reverts to out-of-band config-push byte delivery

- **Context**: The former Requirement 8 required a **full entity-bearing state snapshot** embedding the new config, installed on every member, so no member replays pre-upgrade WAL under the new config (the bug-857 hazard).
- **Finding (the prior blocker)**: Snapshots are a 16-byte LSN sentinel (`persist.c:1068-1101/1280-1307`); the entity-serialization machinery (`xzarray_freeze/thaw`) is dormant (`kvmgr.c:297`); roadmap "entity-bearing snapshots (Task 11)" is **unstarted** (`roadmap.md:43`). The full upgrade snapshot is therefore a HARD dependency on entity-bearing-snapshots, which does not exist.
- **Alternatives Considered**:
  1. Make entity-bearing-snapshots a hard prerequisite and sequence the upgrade snapshot after it, inside this spec.
  2. Carry config bytes in the WAL marker — **violates R7.2/R7.4**.
  3. Per-node replay under new config — the exact bug-857 hazard.
  4. **Descope the materialized-state-on-upgrade work into an independent spec (selected)** — split the entity-snapshot blocker out of `ha-term-safety` so R1–R7 can ship now.
- **Selected Approach**: The configuration-upgrade materialized-state consistency (former R8) is **descoped into the independent `config-upgrade-snapshot` spec**, which **HARD-depends on `entity-bearing-snapshots`**. `ha-term-safety` retains R7's consensus half (the in-log `(term,lsn)` config marker) and **reverts** the configuration-byte delivery to the **existing out-of-band config-push transport** (`OPCode_replConfigPush`), keyed to the marker's `{version_tag, sha256}` and hash-verified against the committed marker before activation. No snapshot is built or depended on here. `config-upgrade-snapshot` later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land.
- **Rationale**: Do not paper over an infeasible requirement, and do not block the implementable R1–R7 (protocol term-safety + config marker) on an unstarted entity-serialization backlog. The marker + commit + reconcile (R7) never required entities; only the materialized-state install did. The byte delivery falls back to the already-working config-push path.
- **Trade-offs**: `ha-term-safety` no longer delivers the full upgrade-snapshot guarantee (now owned by `config-upgrade-snapshot`); the interim config-push byte delivery is superseded for the upgrade case once that spec lands.
- **Follow-up**: `config-upgrade-snapshot` brief + requirements written (`../config-upgrade-snapshot/{brief.md,requirements.md}`); it anchors on this spec's `(term,lsn)` config marker and consumes `entity-bearing-snapshots`.

### Decision: Preserve the commit barrier + durability + clean-failover behaviors as policy

- **Context**: R4 and R5 must not regress ha-commit-durability (bugs 661/828) or bugs 848/849/850.
- **Selected Approach**: `on_commit_advanced` only **triggers** the existing synchronous barrier; `majority_candidate_locked` self-cap, persist-before-ACK, the pre-promotion fence, advertise-real-durable-floor freshness, and the contiguity guard are unchanged. The library supplies the safe commit position; the durable act stays in the controller.
- **Rationale**: The barrier/durability is verified by tests + the Protocol Verification Checklist (not by raft.tla, which assumes a durable log); changing source-of-truth for the commit position must not change the durability mechanism.
- **Trade-offs**: None functional; R4.x latency for the steady state is unchanged (R5.4) because the barrier path is untouched.

## Risks & Mitigations

- **Former R8 infeasible without entity-bearing snapshots** → RESOLVED by descoping the configuration-upgrade materialized-state work into the independent `config-upgrade-snapshot` spec (HARD-depends on `entity-bearing-snapshots`); `ha-term-safety` ships R1–R7 with out-of-band config-push byte delivery and no snapshot dependency.
- **Upstream signature drift** (the three library specs are generated but not yet implemented) → Listed as Revalidation Triggers; the adapter wires to the exact published signatures; if a signature changes, re-check the adapter table.
- **Neutralizing term-blind controller paths could regress clean failover** → R5 acceptance criteria + the existing durability tests are the guard; the live R6 regression asserts both chaotic AND clean convergence.
- **`waldump` shows TERM=0 pre-fix** → expected (G1); R6 asserts non-zero, per-LSN-consistent terms post-fix, and the negative direction (pre-fix repro must fail) per R6.5.
- **Migration of `committed_config_tag` to marker-derived** → derive-not-store; on first boot after upgrade, a cluster with no marker treats config as term-0/lowest-known (R1.8 analogue) until the first marker commits.

## References

- `.kiro/specs/ha-term-safety/requirements.md` — R1–R7 (former R8 descoped 2026-06-21).
- `.kiro/specs/config-upgrade-snapshot/{brief.md,requirements.md}` — the descoped configuration-upgrade materialized-state snapshot (HARD-depends on `entity-bearing-snapshots`).
- `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` — gaps G1–G6 + the library-owns-protocol DECISION.
- `.kiro/specs/raft-log-replication/design.md`, `.kiro/specs/raft-commit-safety/design.md`, `.kiro/specs/raft-sim-conformance/design.md` — upstream callback contracts (consumed verbatim).
- `.kiro/steering/persistence.md`, `ha-async-io.md`, `roadmap.md:13-21,43,55` — durability policy, process-lifecycle invariant, Raft conformance series, Task 11 backlog, planned `walcompare` tooling.
- `.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md` + `repro-851/` — bug-851/857 evidence and the regression baseline.