myxce/.kiro/specs/ha-term-safety/design.md

# Design Document

## Overview

**Purpose**: `ha-term-safety` is the **controller-side integration + config + live-regression umbrella** for the Raft TLA+ Conformance library series. It wires the XCE controller as a **thin callback adapter** to the protocol that `raft-log-replication`, `raft-commit-safety`, and `raft-sim-conformance` implement in `src/lib/raft/raft.c`, closing bug-851 (silent committed-prefix divergence under chaotic failover) **without re-implementing any Raft protocol logic in the controller**. (The related configuration-upgrade materialized-state divergence, bug-857, is descoped to `config-upgrade-snapshot`.)

**Users**: The XCE HA controller cluster (as Raft nodes), operators relying on "acknowledged ⇒ never lost across any failover pattern", and maintainers needing a live regression that reproduces and fences bug-851.

**Impact**: Today the replicated log, term propagation, reconcile, and commit are hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c`/`config_repl.c` and match by LSN only (term-blind). After this feature the controller **delegates** those decisions to the library through five new callbacks and three new entry points, **stamps the real election term** into the existing 40-byte WAL header, **derives committed config from an in-log `(term,lsn)` marker** instead of a separate index, and ships a live chaotic-multi-kill regression. Configuration **bytes** continue to move through the existing out-of-band config-push transport (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. The Raft protocol itself (logOk, `(term,index)` truncation, current-term commit, freshness) is **not** in this spec — it is realized by the three library specs.

### Goals
- Wire the controller adapter to the upstream callbacks **verbatim** (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), stamping `wal_entry_hdr_t.term` from the library's term (R1) and routing incoming heartbeat/replication through the library so term adoption happens on every RPC (R2/R3).
- Replace the term-blind safe-commit (`ha_compute_safe_committed_lsn`) and committed-boundary reconcile (`ha_rejoin_reconcile`) with the library's `commit_index` (via `on_commit_advanced`) and `(term,lsn)` truncation (via `on_truncate_from`), **preserving** the synchronous commit barrier + persist-before-ACK durability and the clean-failover behaviors of bugs 848/849/850 (R4/R5).
- Order configuration changes through the same `(term,lsn)` log as a lightweight `WAL_CONFIG_MARKER` (version_tag + sha256, no bytes); derive committed config from the latest committed marker; deliver the configuration **bytes** via the existing out-of-band config-push transport, hash-verified against the committed marker before activation (R7).
- Ship a live `xce_test` chaotic-multi-kill regression asserting cross-member `(term,lsn)`-fingerprinted committed-prefix identity (R6).

### Non-Goals
- **The Raft protocol itself** — logOk/`prevLogTerm` match, `(term,index)` conflict truncation, the Figure-8 current-term commit rule, `(lastLogTerm,lastLogIndex)` freshness, `UpdateTerm`-on-any-RPC — owned by `raft-log-replication` + `raft-commit-safety`; consumed verbatim.
- **The commit-barrier / durability mechanism** — the synchronous barrier, group-commit flusher, persist-before-ACK, reply gate, durable-floor freshness — owned by `ha-commit-durability`; this spec only changes the barrier's *input*, never its mechanism.
- **The in-process conformance oracle** — Election Safety / Log Matching / Leader Completeness / State Machine Safety in `raft_sim.c` — owned by `raft-sim-conformance`; the R6 live regression complements it, it does not duplicate it.
- **Configuration-upgrade materialized-state consistency** — establishing a full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state (former R8) — is **descoped to the independent `config-upgrade-snapshot` spec**, which HARD-depends on `entity-bearing-snapshots`. It is **out of scope here**; this spec delivers the config marker (consensus) plus out-of-band config-push byte delivery, not a full upgrade snapshot.
- Voting-set membership changes / joint consensus (`ha-config-bootstrap`); bugs 854/847/845/828 (tracked separately).

## Boundary Commitments

### This Spec Owns
- **The controller callback-adapter bodies** implementing the five upstream callbacks over the WAL / repl transport / commit barrier (and an in-process route for incoming AppendEntries/heartbeat).
- **`wal_entry_hdr_t.term` value stamping**: a `persist_set_term` seam so every append records the library's live term; reading `hdr.term` back on `replay_wal` and feeding it to `on_get_entry_term` + the library's restore (`initial_log_len`, last-log-term).
- **`ha_convergence` delegation**: replacing the committed-boundary truncate with routing the leader's AppendEntries through `raft_on_append_entries_request` + `on_truncate_from`.
- **Commit-source switch**: feeding the library's `commit_index` (via `on_commit_advanced`) into the existing synchronous barrier, retiring the term-blind `ha_compute_safe_committed_lsn` as the commit source.
- **`WAL_CONFIG_MARKER` (R7)**: the new WAL entry type + append-via-`raft_log_append` + committed-config-derived-from-latest-committed-marker, retiring `committed_config_tag` as an authoritative index.
- **Out-of-band config-push byte delivery + hash-verify-before-activate (R7)**: keying the existing config-push transport (`OPCode_replConfigPush`) to the committed marker's version_tag + sha256, verifying held bytes against the marker hash before activation, and fencing a member from serving under mismatched configuration.
- **The live R6 regression**: the chaotic-multi-kill `xce_test` script + the `walcompare` cross-member `(term,lsn,pcrc)` divergence tool.

### Out of Boundary
- All Raft protocol decisions (library: `raft-log-replication`, `raft-commit-safety`).
- The commit barrier, durability/fsync, persist-before-ACK, durable-floor freshness mechanism (`ha-commit-durability`).
- The conformance oracle / four-invariant checkers (`raft-sim-conformance`).
- **Configuration-upgrade materialized-state consistency** — the full entity-bearing state snapshot at the change boundary (so no member re-derives post-upgrade state) and its `xzarray_freeze`/`thaw` + entity encode/decode payload — is **descoped to the independent `config-upgrade-snapshot` spec** (which HARD-depends on `entity-bearing-snapshots`). This spec ships the config marker (consensus) + out-of-band config-push byte delivery; it does **not** build the upgrade snapshot or entity serialization.
- Membership/quorum transitions (`ha-config-bootstrap`); the on-disk WAL byte layout (unchanged — value fix only).

### Allowed Dependencies
- `src/lib/raft/raft.{c,h}` after the three upstream specs land: the five callbacks + entry points + `raft_append_result_t` + `commit_index`/`next_index[]`/`match_index[]` + `(lastLogTerm,lastLogIndex)` freshness, consumed **verbatim**.
- The existing controller infrastructure: `persist.c` WAL/recovery, `repl.c` transport, `commit.c` synchronous barrier + durability (`ha-commit-durability`), the out-of-band **config-push** transport (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`), the ten already-wired election callbacks (`s_real_cbs`, `ha.c:1309-1320`).
- The test infrastructure: `waldump.{c,h}`, `xce_test` script primitives, `/xce-deploy-ha`.

### Revalidation Triggers
Re-check this design if any of the following change:
- The signature/semantics of any upstream callback (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) or entry point (`raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), including `raft_append_result_t`.
- The 1-based-index ↔ WAL-LSN identity or the "term 0 == absent" convention.
- The addition of `last_log_term` to `raft_on_vote_request`/`raft_on_pre_vote_request` (the controller must plumb it onto the wire and read it from `hdr.term` on recovery).
- The `commit_index` restore convention (`initial_commit_index`) or `initial_log_len`.
- The config-push transport contract (`OPCode_replConfigPush`) or the marker's `{version_tag, sha256}` key — the byte delivery is hash-verified against the committed marker before activation.
- The synchronous commit barrier / persist-before-ACK contract (`ha-commit-durability`) — `on_commit_advanced` must continue to *trigger* it unchanged.
- The split of the configuration-upgrade materialized-state work into `config-upgrade-snapshot` — if that spec changes how it consumes the `(term,lsn)` config marker, re-check the marker contract here.

## Architecture

### Existing Architecture Analysis

The controller already drives the **election** half of the library through ten callbacks wired in `s_real_cbs` (`ha.c:1309-1320`): vote/PreVote send, freshness gate (`raft_cb_vote_eligible`, `ha.c:299-360`), promotion/demotion (`raft_cb_become_leader`/`follower`, `ha.c:880-1242`), and term persistence (`raft_cb_persist` → `ha_persist_term`, `ha.c:1251-1294, 2441-2474`). The controller uses **only** `src/lib/raft/raft.c` (no legacy `src/controller/raft.c`). The **log** half is hand-rolled and term-blind:
- Every WAL append stamps `hdr.term = p->term` (`persist.c:588/726/1040`) but `p->term` is **never assigned** → term 0 everywhere (G1). The 40-byte header already has the `term` field (`wal_types.h:29-39`) — a value fix, not a layout change.
- Incoming heartbeats/replication never adopt a higher term (`ha_consume_heartbeat_committed` drops stale only, `ha.c:5013-5036`) (G5).
- Reconcile truncates by committed-boundary, not `(term,lsn)` (`ha_rejoin_reconcile`, `ha_convergence.c:295-366`; `persist_truncate_wal_above` LSN-only, `persist.c:2455-2565`) (G3).
- Safe commit is computed term-blind from voter LSNs (`ha_compute_safe_committed_lsn`, `ha.c:480-509`) (G4); config commits through a separate `committed_config_tag` with no term (`config_repl.c:941-979`), with the bytes pushed out-of-band via `OPCode_replConfigPush` — the marker (consensus) will replace the term-blind index, while the byte transport is reused as-is.

The durability layer (`ha-commit-durability`) is correct and must be preserved: `majority_candidate_locked` self-caps at `persist_durable_lsn` (`commit.c:79-138`), persist-before-ACK on standbys (`persist_wal_write_replicated`), the pre-promotion eligibility fence (`ha.c:933-951`), advertise-real-durable-floor freshness, and the contiguity guard (`persist.c:691-706`, bug-850).

This design follows the **same Ports-and-Adapters discipline already established by the election callbacks**: the library decides; the controller performs I/O.

### Architecture Pattern & Boundary Map

```mermaid
flowchart TB
    subgraph Library["src/lib/raft/raft.c — PROTOCOL OWNER (NOT this spec)"]
        APP["raft_log_append (ClientRequest)"]
        AEREQ["raft_on_append_entries_request<br/>UpdateTerm + logOk + truncate"]
        AEREP["raft_on_append_entries_reply"]
        ADV["raft_advance_commit<br/>current-term Figure-8"]
        VOTE["vote freshness lastLogTerm,lastLogIndex"]
    end
    subgraph Adapter["src/controller — THIN ADAPTER (this spec)"]
        PE["on_persist_entry<br/>persist_set_term + WAL write"]
        GET["on_get_entry_term<br/>WAL index / replay read-back"]
        SEND["on_send_append_entries<br/>repl envelope prevLogTerm+entryTerm"]
        TRUNC["on_truncate_from<br/>persist_truncate_wal_above"]
        COMMIT["on_commit_advanced<br/>TRIGGER existing sync barrier"]
        ROUTE["heartbeat/repl intake -> raft_on_append_entries_request<br/>(replaces ha_consume_heartbeat + ha_rejoin_reconcile)"]
    end
    subgraph Config["Config marker + out-of-band byte delivery (this spec)"]
        MARK["WAL_CONFIG_MARKER<br/>version_tag + sha256 via raft_log_append"]
        CPUSH["config-push byte delivery<br/>OPCode_replConfigPush, hash-verify vs marker before activate"]
    end
    subgraph Preserved["PRESERVED policy (other specs)"]
        BARRIER["sync commit barrier + persist-before-ACK<br/>ha-commit-durability"]
        CFGXPORT["existing config-push transport<br/>OPCode_replConfigPush"]
    end
    subgraph Test["Live regression (this spec)"]
        WC["walcompare term,lsn,pcrc"]
        SCRIPT["xce_test chaotic multi-kill"]
    end
    APP --> PE
    AEREQ --> GET & PE & TRUNC
    ROUTE --> AEREQ
    AEREP --> SEND
    ADV --> COMMIT
    COMMIT --> BARRIER
    MARK --> APP
    CPUSH --> CFGXPORT
    SCRIPT --> WC
```

**Architecture Integration**:
- **Selected pattern**: thin callback adapter (Ports-and-Adapters), identical to the election-callback pattern already in `ha.c`. The library owns protocol; the controller owns I/O.
- **Domain boundaries**: protocol decisions (append/truncate/commit/vote) in `raft.c`; durable I/O (WAL write, repl send, barrier, config-push transport) in the controller; the marker + out-of-band byte delivery in this spec; the configuration-upgrade materialized-state snapshot is descoped to `config-upgrade-snapshot`.
- **Existing patterns preserved**: `s_real_cbs` callback wiring, persist-before-ACK, the synchronous barrier, the pre-promotion fence, the contiguity guard, "controller never self-terminates".
- **Steering compliance**: `ha-async-io.md` (no blocking in iomgr/vtask; barrier on its own path), `persistence.md` (durable floor / acknowledged⇒durable preserved; never persist commit index), `error-handling.md` (status returns, fail-loud non-promotable fence, never `exit`/`abort`), `testing.md` (TDD; tests in `testsuite/` before C changes).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Protocol core | `src/lib/raft/raft.c` (gnu17, after 3 upstream specs) | Owns Raft log/commit/freshness | Consumed verbatim; not modified here |
| Adapter / WAL | `src/controller/persist.c` + `wal_types.h` | `hdr.term` stamping, recovery read-back, truncate | 40-byte header unchanged; `persist_set_term` seam added |
| Adapter / transport | `src/controller/repl.c`, `typemap.h` | carry `prevLogTerm` + entry term on AppendEntries | extend repl envelope fields |
| Adapter / commit | `src/controller/commit.c` (`ha-commit-durability`) | barrier triggered by `on_commit_advanced` | mechanism unchanged |
| Config | `src/controller/config_repl.c` + `wal_types.h` | `WAL_CONFIG_MARKER`, marker-derived committed config | retires `committed_config_tag` index |
| Config bytes | existing config-push transport (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`) | deliver bytes out-of-band; hash-verify vs marker before activate | transport reused as-is; no snapshot |
| Test | `xce_test` + `waldump.{c,h}` + `/xce-deploy-ha` | live chaotic regression + `walcompare` | ~90% infra exists |

## File Structure Plan

### Modified Files
- `src/include/wal_types.h` — add `WAL_CONFIG_MARKER = 50` to `wal_entry_type`; add the marker payload struct `wal_config_marker_t { uint64_t version_tag; uint8_t sha256[32]; }` (no config bytes). **No change to `wal_entry_hdr_t` (40 bytes preserved, R1.7).**
- `src/include/persist.h` — declare `persist_set_term(persist_t *p, uint64_t term)`; declare recovery accessors that surface per-entry term + restored `log_len`/last-log-term for the library init.
- `src/controller/persist.c` — implement `persist_set_term` (sets `p->term`); make `replay_wal` retain `hdr.term` per LSN (in-memory term index) so `on_get_entry_term` and library restore can read it; `persist_truncate_wal_above` reused unchanged by `on_truncate_from`.
- `src/controller/repl.c` + `src/include/typemap.h` — extend the AppendEntries-carrying envelope (`wal_catchup_entry_t`/`capnp_repl_envelope_t`) to carry `prev_log_index`, `prev_log_term`, and the entry's `term` explicitly, so a standby persists the leader's term (not local `p->term`).
- `src/controller/ha.c` — wire the five new callbacks into `s_real_cbs` (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`); add the heartbeat/replication intake that routes through `raft_on_append_entries_request` (replacing the term-blind `ha_consume_heartbeat_committed` term handling); drive `raft_log_append` on the primary append path; pass `last_log_term` on the vote wire and decode it on intake; retire `ha_compute_safe_committed_lsn` as the commit source (the library's `commit_index` drives the barrier via `on_commit_advanced`).
- `src/controller/ha_convergence.c` — replace `ha_rejoin_reconcile`'s committed-boundary truncate with delegation: route the leader's AppendEntries through the library; `on_truncate_from` performs the truncate. Keep the non-promotable retry-fence behavior (R3.5/R5).
- `src/controller/config_repl.c` — `do_upgrade_config_with_bytes` appends a `WAL_CONFIG_MARKER` via `raft_log_append` (replacing the `committed_config_tag` push/commit); derive committed config from the latest committed marker in the committed prefix; key the existing out-of-band config-push (`OPCode_replConfigPush`) to the marker's `{version_tag, sha256}`, hash-verify the held bytes against the committed marker before activating, and fence a member from serving under mismatched configuration (R7.5/R7.8–R7.10). The full entity-bearing upgrade snapshot is **out of scope** (`config-upgrade-snapshot`).
- `src/controller/commit.c` — no mechanism change; `on_commit_advanced` calls into the existing `commit_advance_locked`/`commit_set_writable_lsn` path with the library-supplied `commit_index`.

### New Files
- `src/controller/testsuite/walcompare.{c,h}` — cross-member WAL diff by `(lsn → term, pcrc)`; nonzero exit on committed-prefix divergence. Reuses the `waldump.c` reader/decoder (roadmap-planned tooling, `roadmap.md:55`).
- `src/controller/testsuite/test_term_safety_adapter.c` — unit/integration tests for the adapter callbacks (term stamping, `on_get_entry_term` read-back, truncate delegation, `on_commit_advanced` barrier trigger, marker append/derive).
- `src/tools/test-term-safety-chaos.sh` (or an `xce_test` script under `testsuite/`) — the R6 chaotic-multi-kill driver invoking `walcompare`.

> Each file keeps one responsibility: protocol stays in `raft.c` (untouched); `persist.c` = WAL I/O + term value; `ha.c`/`ha_convergence.c` = callback bodies + intake routing; `config_repl.c` = marker + out-of-band config-push byte delivery; `testsuite/` = regression. The File Structure Plan reflects the boundary: no protocol logic file is created or modified, and no upgrade-snapshot file is added (descoped to `config-upgrade-snapshot`).

## System Flows

### Replicated append → standby persist with the leader's term (R1, R2, R3)

```mermaid
sequenceDiagram
    participant P as Primary library (raft.c)
    participant PA as Primary adapter (this spec)
    participant SA as Standby adapter (this spec)
    participant S as Standby library (raft.c)
    Note over P: raft_log_append(value) — stamps s->term
    P->>PA: on_persist_entry(idx, term, entry)
    PA->>PA: persist_set_term(p, term) then WAL write hdr.term=term
    Note over P: raft_send_append_entries(peer, commit_index)
    P->>PA: on_send_append_entries(peer, prevIdx, prevTerm, leaderCommit, entry+entryTerm)
    PA-->>SA: repl envelope (carries prevLogTerm + entryTerm)
    Note over S: raft_on_append_entries_request()<br/>UpdateTerm FIRST (G5) then logOk (G2)
    S->>SA: on_get_entry_term(prevIdx)  %% logOk
    alt term conflict at idx (G3)
        S->>SA: on_truncate_from(idx) then on_persist_entry(idx, leader_term)
    else append
        S->>SA: on_persist_entry(idx, leader_term)  %% standby persists LEADER term
    end
    Note over SA: persist-before-ACK preserved -> WalAck only on durable
```

Flow-level decisions:
- The standby persists the **leader's** entry term (carried explicitly on the envelope), closing the `persist_wal_write_replicated` "best-effort `p->term`" hazard. `persist_set_term` keeps `p->term` aligned with the library term for the local-append path.
- Incoming heartbeats are 0-entry AppendEntries routed through the **same** `raft_on_append_entries_request`, so a higher-term heartbeat triggers `UpdateTerm` (G5) — the controller no longer needs `ha_consume_heartbeat_committed` to adopt term.
- The contiguity guard (bug-850) and persist-before-ACK (ha-commit-durability) are unchanged and sit beneath `on_persist_entry`.

### Commit advance → existing barrier; config marker (R4, R7)

```mermaid
sequenceDiagram
    participant L as Leader library (raft.c)
    participant A as Adapter (this spec)
    participant B as Sync commit barrier (ha-commit-durability, PRESERVED)
    Note over L: replication updated match_index[] (library)
    Note over L: raft_advance_commit() — current-term Figure-8 (G4)
    alt commit_index strictly advances
        L->>A: on_commit_advanced(commit_index)
        A->>B: feed safe commit position into existing barrier
        Note over B: majority self-cap at durable_lsn + persist-before-ACK<br/>+ own-term writable gate (UNCHANGED)
    else held (prev-term / no quorum)
        Note over L: no callback, barrier untouched
    end
    Note over A: committed config = latest WAL_CONFIG_MARKER<br/>within committed prefix (derived, not a separate index)
```

Flow-level decisions:
- `on_commit_advanced` **triggers** the barrier; it never performs durability itself. `ha_compute_safe_committed_lsn` is retired as the commit source (the library's current-term-gated `commit_index` replaces it), but the post-election truncate-to-safe at promotion and the durable self-cap remain (R5).
- A config change is an ordinary committed log entry: the `WAL_CONFIG_MARKER` commits through the same barrier and reconciles through the same `(term,lsn)` matching (R7.1/R7.3/R7.6/R7.11). The committed config is the latest committed marker (R7.7).

### Configuration bytes — out-of-band config-push, hash-verified against the committed marker (R7.5, R7.8–R7.10)

```mermaid
flowchart TB
    M["WAL_CONFIG_MARKER commits at term,lsn<br/>(version_tag + sha256, no bytes)"] --> NEED{"local bytes match<br/>marker sha256?"}
    NEED -->|no| PUSH["obtain bytes via existing config-push<br/>OPCode_replConfigPush, keyed to version_tag"]
    PUSH --> VERIFY["verify obtained bytes sha256 == marker hash"]
    NEED -->|yes| VERIFY
    VERIFY -->|match| ACTIVATE["activate configuration"]
    VERIFY -->|mismatch| FENCE["serve-fence: keep member non-serving<br/>under new config; retry (R7.9)"]
```

Flow-level decision: the marker (consensus) is the authority; the **bytes travel out-of-band on the existing config-push transport** (`OPCode_replConfigPush`), never in a WAL payload (R7.4). A member obtains and **hash-verifies the bytes against the committed marker before activating** (R7.8), fences itself on mismatch (R7.9), and on recovery activates the configuration referenced by the latest committed marker (R7.10). The full materialized-state **upgrade snapshot** (former R8) is **out of scope** — see `config-upgrade-snapshot`.

## Requirements Traceability

| Requirement | Summary | Adapter component(s) | Library/upstream realization | Flow |
|-------------|---------|----------------------|------------------------------|------|
| 1.1, 1.2 | record live/promotion term in entry | `on_persist_entry` + `persist_set_term`; `raft_log_append` (incl. own-term NOOP) | library stamps `s->term` | Append flow |
| 1.3, 1.4 | transmit + persist leader term on replication | `on_send_append_entries` carries entryTerm; standby `on_persist_entry` writes leader term | `raft_on_append_entries_request` | Append flow |
| 1.5 | recovery reads back per-entry term | `replay_wal` term index; `on_get_entry_term`; restore `initial_log_len`/last-log-term | library init restore | — |
| 1.6 | no zero/placeholder term while term valid | `persist_set_term` keeps `p->term` live | — | — |
| 1.7 | preserve 40-byte WAL layout (value fix) | `wal_entry_hdr_t` unchanged | — | — |
| 1.8 | legacy zero term = lowest-known for matching | `on_get_entry_term` maps legacy 0 → lowest-known | library `(term,index)` match tolerant | — |
| 2.1, 2.2 | persist won term before appending; not committed until majority | `raft_cb_persist`→`ha_persist_term`; own-term `WAL_NOOP` gate (`commit_set_writable_lsn`) | `BecomeLeader` + commit rule | Commit flow |
| 2.3, 2.4, 2.6 | no re-elect at stale term; adopt higher term on any RPC | heartbeat/repl intake → `raft_on_append_entries_request`; `on_persist` | `UpdateTerm` (G5) | Append flow |
| 2.5 | at most one leader advances committed per term | library current-term commit + freshness | `raft-commit-safety` G4/G6 | Commit flow |
| 3.1, 3.2 | reconcile by `(term,lsn)`; truncate divergent suffix | `ha_convergence` delegation → `on_truncate_from` | `raft_on_append_entries_request` (G3) | Append flow |
| 3.3 | never truncate committed prefix | library truncates only above match point; `on_truncate_from` honors it | library | — |
| 3.4 | reconciled prefix byte-identical to leader | standby persists leader entry+term | library | Append flow |
| 3.5, 3.6 | non-promotable retry; no false committed | keep loud non-promotable fence; do not report divergent as committed | `ha-commit-durability` fence preserved | — |
| 4.1, 4.5 | advance committed only on `(term,lsn)` quorum + current-term | `on_commit_advanced` feeds barrier with library `commit_index` | `raft_advance_commit` (G4) | Commit flow |
| 4.2, 4.3, 4.4 | committed entry byte-identical/preserved across leadership | library Leader Completeness + barrier durability | library + `ha-commit-durability` | Commit flow |
| 5.1 | clean single failover converges identically | unchanged promotion truncate-to-safe + library matching | `raft_cb_become_leader` | — |
| 5.2 | recovery-on-promotion + advertise-real-LSN (848/849) | `raft_cb_become_leader` + `raft_cb_vote_eligible` unchanged | preserved | — |
| 5.3 | contiguity guard, no forward gap (850) | `persist_wal_write_replicated` guard unchanged beneath `on_persist_entry` | preserved | — |
| 5.4 | no added steady-state ack latency | barrier path untouched; `on_commit_advanced` only changes source | preserved | — |
| 6.1, 6.2, 6.3 | chaotic regression asserts `(term,lsn,content)` prefix identity | `walcompare` + chaos script | — | — |
| 6.4 | no acked-committed job absent post-run | `expect-job $jobid` per acked submit | — | — |
| 6.5 | pre-fix repro fails, post-fix passes | negative-direction assertion (term=0 pre-fix) | — | — |
| 6.6 | config changes interleaved → same committed config ref | `walcompare` checks committed marker tag+hash | R7 derive | Config flow |
| 7.1–7.4, 7.6, 7.7, 7.11 | config as `(term,lsn)` marker, no bytes, term-stamped, reconciled, derived, majority-committed | `WAL_CONFIG_MARKER` via `raft_log_append`; marker-derived committed config | library log/commit/reconcile | Config marker flow |
| 7.5, 7.8, 7.9, 7.10 | bytes via out-of-band config-push; verify hash before activate; fence on mismatch; activate from latest committed marker on recovery | config-push byte delivery (`OPCode_replConfigPush`) keyed to marker `{version_tag, sha256}` + hash-verify-before-activate + serve-fence | existing config-push transport | Config bytes flow |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|-------|--------|--------------|--------------------------|-----------|
| WAL term adapter | Adapter | Stamp/read-back the library term in `hdr.term` | 1.1–1.8 | library callbacks (P0), `persist.c` (P0) | Service, State |
| AppendEntries intake/route | Adapter | Route heartbeat/repl through the library; term adoption | 2.3–2.6, 3.1–3.4 | `raft_on_append_entries_request` (P0), repl envelope (P0) | Service |
| Reconcile delegation | Adapter | Replace committed-boundary truncate with `on_truncate_from` | 3.1–3.6 | library (P0), `persist_truncate_wal_above` (P0), fence (P1) | Service |
| Commit-source switch | Adapter | Feed library `commit_index` into existing barrier | 4.1–4.5, 5.4 | `on_commit_advanced` (P0), `commit.c` barrier (P0) | Service |
| Clean-failover preservation | Adapter | Keep 848/849/850 behaviors unchanged | 5.1–5.4 | promotion/vote callbacks, contiguity guard (P0) | Service |
| Config marker | Adapter | `WAL_CONFIG_MARKER` in unified log; derive committed config | 7.1–7.4,7.6,7.7,7.11 | `raft_log_append` (P0), `config_repl.c` (P0) | Service, State |
| Config byte delivery | Adapter | Out-of-band config-push keyed to marker; hash-verify-before-activate; serve-fence on mismatch | 7.5,7.8,7.9,7.10 | config-push transport `OPCode_replConfigPush` (P0), `config_repl.c` (P0) | Service |
| Live regression | Test | Chaotic-multi-kill + `walcompare` | 6.1–6.6 | `xce_test`, `waldump`, `/xce-deploy-ha` (P0) | Test |

### Adapter Callback → Controller Implementation Site (the core seam table)

| Library callback / entry point (upstream) | Controller implementation site | Behavior |
|---|---|---|
| `on_persist_entry(ctx,index,term,entry,len)` | `ha.c` callback → `persist_set_term(p,term)` + `persist_wal_async`/`persist_wal_write_replicated` (`persist.c:588/726`) | Stamp `hdr.term=term` at LSN=index; persist-before-ACK preserved. |
| `on_get_entry_term(ctx,index)->term\|0` | `ha.c` callback → in-memory WAL term index from `replay_wal`/live append | Return recorded `hdr.term` for LSN; legacy 0 → lowest-known (R1.8); 0 if absent. |
| `on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)` | `ha.c` callback → `repl.c` envelope build (`typemap.h` extended fields) | Enqueue AppendEntries carrying `prevLogTerm` + entry `term`; `leaderCommit = commit_index`. |
| `on_truncate_from(ctx,index)` | `ha.c` callback → `persist_truncate_wal_above(p,index-1)` (`persist.c:2455`) | Delete LSN≥index; replaces `ha_rejoin_reconcile` committed-boundary heuristic. |
| `on_commit_advanced(ctx,commit_index)` | `ha.c` callback → `commit.c` existing barrier (`commit_advance_locked`/`commit_set_writable_lsn`) | Feed safe commit position; **trigger** barrier, do not replace it. |
| `raft_log_append(s,entry,len,&idx)` | primary append path in `ha.c`/`config_repl.c` (incl. `WAL_CONFIG_MARKER`) | Library stamps `s->term`, assigns LSN; drives `on_persist_entry`. |
| `raft_on_append_entries_request(...)` | heartbeat/repl intake in `ha.c` (replaces `ha_consume_heartbeat_committed` term handling) | Library does `UpdateTerm`+logOk+truncate/append via callbacks. |
| `raft_on_append_entries_reply(...)` | repl ACK intake in `ha.c` | Library updates `match_index[]`/`next_index[]`. |
| `raft_advance_commit(s)` | driven after repl ACK in `ha.c` | Library computes current-term-gated `commit_index`; may fire `on_commit_advanced`. |
| `raft_on_vote_request/pre_vote_request(...,last_log_term,...)` | existing vote callbacks in `ha.c`, extended | Controller decodes candidate `last_log_term` from wire; reads own via `on_get_entry_term(log_len)`. |

### WAL term adapter

| Field | Detail |
|-------|--------|
| Intent | Make `hdr.term` carry the library's election term on every entry and surface it on recovery |
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8 |

**Responsibilities & Constraints**
- `persist_set_term(persist_t *p, uint64_t term)` assigns `p->term` (the missing seam); called by `on_persist_entry` and on term adoption so no append stamps 0 while a valid term is in effect (R1.6). On-disk `wal_entry_hdr_t` is unchanged (40 bytes, R1.7).
- `replay_wal` retains each `hdr.term` in an in-memory per-LSN index so `on_get_entry_term` answers from recovered state (R1.5); a legacy entry with term 0 maps to the lowest-known term for matching (R1.8).
- Standbys persist the **leader's** entry term (carried on the envelope), not local `p->term` (R1.3/R1.4).

**Contracts**: Service [x] / State [x]

##### Service Interface
```c
/* Set the controller's live election term so subsequent WAL appends stamp it
 * into hdr.term (R1.1/R1.6). Called from on_persist_entry and on term adoption.
 * No on-disk layout change; value fix only (R1.7). */
void persist_set_term(persist_t *p, uint64_t term);

/* Recovered per-entry term for an LSN (1-based == index), feeding on_get_entry_term
 * and the library's init restore. Returns 0 when absent; legacy-0 is mapped to the
 * lowest-known term by the adapter for (term,lsn) matching (R1.8). */
uint64_t persist_entry_term(const persist_t *p, uint64_t lsn);
```
- Preconditions: `p` valid; `lsn` within `[1, current_lsn)`.
- Postconditions: `on_get_entry_term(lsn)` returns the recorded term; appends stamp the live term.
- Invariants: `hdr.term` never 0 while a valid term is in effect; 40-byte layout preserved.

### Config marker

| Field | Detail |
|-------|--------|
| Intent | Order config changes through the unified `(term,lsn)` log as a lightweight marker |
| Requirements | 7.1, 7.2, 7.3, 7.4, 7.6, 7.7, 7.11 |

**Responsibilities & Constraints**
- `do_upgrade_config_with_bytes` (`config_repl.c:1001-1163`) appends `WAL_CONFIG_MARKER` (`wal_config_marker_t = {version_tag, sha256[32]}`, **no bytes**, R7.2/R7.4) via `raft_log_append`, so it inherits term-stamping (R7.3), commit (R7.11), and `(term,lsn)` reconcile/truncate (R7.6).
- Committed config is derived from the **latest committed marker within the committed prefix** (R7.7); the separate `committed_config_tag` index is retired as authoritative (it may remain as a derived cache).

**Contracts**: Service [x] / State [x]

##### Service Interface
```c
/* Append a config-change marker (version_tag + sha256, no config bytes) to the
 * unified replicated log via raft_log_append. Returns RAFT_OK / the append rc.
 * The config bytes travel out-of-band via config-push (R7.5), never the marker. */
int config_append_marker(cluster_t *c, uint64_t version_tag,
                         const uint8_t sha256[32], uint64_t *out_lsn);

/* Derive the committed configuration reference from the latest WAL_CONFIG_MARKER
 * at or below committed_lsn (R7.7). */
int config_committed_marker(const cluster_t *c, uint64_t *version_tag,
                            uint8_t sha256_out[32]);
```
- Preconditions: leader for append; valid `sha256`.
- Postconditions: marker committed ⇒ committed config reference advances; reconcile truncates a divergent marker exactly like a job entry.
- Invariants: config bytes never enter a WAL payload (R7.4).

### Config byte delivery

| Field | Detail |
|-------|--------|
| Intent | Deliver config bytes out-of-band via the existing config-push transport; hash-verify against the committed marker before activation |
| Requirements | 7.5, 7.8, 7.9, 7.10 |

**Responsibilities & Constraints**
- Move the configuration bytes through the **existing out-of-band config-push transport** (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`), keyed to the committed marker's `{version_tag, sha256}` — never through a WAL payload (R7.4/R7.5).
- When a committed marker references bytes not yet held locally, obtain them via config-push and **verify the obtained bytes' sha256 against the marker hash before activating** (R7.8).
- If a member's locally held bytes do not match the latest committed marker's hash, **keep the member from serving under the new configuration and keep retrying** rather than activating mismatched config (R7.9) — the "controller never self-terminates" loud non-promotable fence, never `exit`/`abort`.
- On recovery, activate the configuration referenced by the **latest committed marker**, obtaining the bytes from out-of-band storage (R7.10).

**Contracts**: Service [x]

##### Service Interface
```c
/* Ensure the bytes for the latest committed config marker are held locally and
 * hash-verified against the marker, then activate. Obtains bytes via config-push
 * (OPCode_replConfigPush) when absent. Returns RAFT_OK on activate; a non-success
 * rc keeps the member fenced (non-serving under the new config) and retrying
 * (R7.8/R7.9/R7.10). */
int config_activate_committed(cluster_t *c);
```
- Preconditions: a committed `WAL_CONFIG_MARKER` exists in the committed prefix.
- Postconditions: on success the active config matches the marker's `{version_tag, sha256}`; on hash mismatch the member stays fenced and retries.
- Invariants: bytes are never sourced from a WAL payload; activation only after hash-verify against the committed marker.

> **Note**: the full materialized-state **upgrade snapshot** (embed entities + config in one artifact, install on every member so no member re-derives post-upgrade state — former R8) is **out of scope** here and lives in the independent **`config-upgrade-snapshot`** spec, which HARD-depends on `entity-bearing-snapshots`. This spec delivers config bytes via the interim out-of-band config-push path that `config-upgrade-snapshot` later supersedes for the upgrade case.

### Live regression (R6)

| Field | Detail |
|-------|--------|
| Intent | Reproduce bug-851 and assert cross-member `(term,lsn)` committed-prefix identity |
| Requirements | 6.1, 6.2, 6.3, 6.4, 6.5, 6.6 |

**Responsibilities & Constraints**
- `walcompare <dirA> <dirB> ...` diffs members by `(lsn → term, pcrc)`; nonzero exit on any committed-prefix divergence (R6.1/R6.2/R6.3), reusing `waldump.c`'s decoder.
- The chaos script drives repeated primary kills interleaved with `jobsub` + config upgrades via `/xce-deploy-ha` + `kill_node`/`start_node` + deterministic `--wal-sync-ms`, then asserts `expect-job $jobid` for every acked submit (R6.4) and `walcompare` identity incl. committed config marker (R6.6).
- R6.5: on **pre-fix** code (term=0) the repro must fail (`walcompare` detects divergence / `expect-job` loss); on fixed code it passes.

**Contracts**: Test [x]

## Data Models

### Domain Model
- **Log entry**: `(index==LSN, term, type, payload)` — the unified `(term,lsn)` log for both job state and config markers. Index 1-based == WAL LSN (library convention).
- **Config-change marker**: `wal_config_marker_t = {uint64_t version_tag, uint8_t sha256[32]}` — a reference, never the bytes.
- **Committed config**: derived value = the latest `WAL_CONFIG_MARKER` at or below `committed_lsn`.

### Logical Data Model
- `wal_entry_type` gains `WAL_CONFIG_MARKER = 50` (between `WAL_NOOP=40` and `WAL_CHECKPOINT=100`).
- `persist_t.term` becomes a written field via `persist_set_term`; a recovered per-LSN term index is added to `persist_t` (in-memory only).
- The repl envelope (`wal_catchup_entry_t`/`capnp_repl_envelope_t`) gains explicit `prev_log_index`, `prev_log_term`, and entry `term` fields on the AppendEntries path.
- **No on-disk WAL layout change** (R1.7): `wal_entry_hdr_t` stays 40 bytes with `term` at offset 0.

### Data Contracts & Integration
- **AppendEntries wire**: carries `(prevLogIndex, prevLogTerm, leaderCommit, entry, entryTerm)` — the standby persists `entryTerm`, not local `p->term`.
- **Vote wire**: carries `(lastLogTerm, lastLogIndex)`; the controller decodes `lastLogTerm` and passes it to the library; own value via `on_get_entry_term(log_len)`.
- **Config bytes**: delivered out-of-band via config-push (`OPCode_replConfigPush`), keyed to the marker's `{version_tag, sha256}`, hash-verified against the committed marker before activation; never in a WAL payload. The full materialized-state upgrade snapshot is out of scope (`config-upgrade-snapshot`).

## Error Handling

### Error Strategy
The adapter follows "controller never self-terminates" (`error-handling.md`, `ha-async-io.md`): callbacks return status; on a non-recoverable condition the controller enters a **loud, non-promotable fence and keeps retrying** — it never `exit`/`abort`. Persist-before-ACK and the durable self-cap are unchanged.

### Error Categories and Responses
- **Persist failure** (`on_persist_entry` rc≠0) → propagate; library does not advance the local log; standby does not ACK (persist-before-ACK preserved).
- **Truncate failure** (`on_truncate_from` rc≠0) → report a non-success request result so the leader retries; the member stays non-promotable until reconciled (R3.5).
- **Unreconcilable member** → loud permanent fence, non-promotable, keep retrying catch-up (R3.5/R5; never report false convergence, R3.6).
- **Config hash mismatch** (R7.9) → keep the member from serving under the new config; retry; do not activate mismatched config (`config_activate_committed` returns non-success; member stays fenced).
- **Config bytes absent for a committed marker** (R7.8) → obtain via config-push and hash-verify before activate; do not activate on a missing or mismatched payload.

### Monitoring
Reuse `raft_cb_trace` and existing INFO/ERROR logging for: term-stamp, term-adopt-on-AE, conflict-truncate, commit-advance-trigger, config-marker-commit, config-byte-verify, and the loud fences.

## Testing Strategy

TDD-mandatory (`testing.md`): tests in `src/controller/testsuite/` precede C changes; gate on the ANSI-stripped `:FAIL:` count; run final durability/regression validation on a real fsync-able filesystem.

### Unit Tests (adapter callbacks, in `test_term_safety_adapter.c`)
1. `on_persist_entry` stamps `hdr.term` from the library term (via `persist_set_term`) and `on_get_entry_term` reads it back; legacy-0 maps to lowest-known (R1.1–1.8).
2. AppendEntries intake routes a higher-term heartbeat through `raft_on_append_entries_request`, the library adopts the term, and `ha_persist_term` records it (R2.3–2.6 / G5).
3. `on_truncate_from` deletes the divergent suffix and the standby persists the leader entry+term; committed prefix untouched (R3.1–3.4 / G3).
4. `on_commit_advanced` feeds the library `commit_index` into the existing barrier and the WAIT entry releases only on durable majority; `ha_compute_safe_committed_lsn` no longer the commit source (R4.1/R4.5, R5.4).
5. `config_append_marker` appends `WAL_CONFIG_MARKER` (no bytes) and `config_committed_marker` derives committed config from the latest committed marker; a divergent marker truncates like a job entry (R7.1–7.7).

### Integration Tests
1. Clean single failover converges to byte-identical WALs incl. terms, preserving recovery-on-promotion + advertise-real-LSN + contiguity (R5.1–5.3).
2. Marker commit + reconcile under a forced leader change keeps every member on the same committed config reference (R6.6, R7.11).
3. Config byte delivery: a committed marker whose bytes are obtained via config-push activates only after the bytes' sha256 matches the marker hash; an injected hash mismatch keeps the member fenced (non-serving under the new config) and retrying (R7.8/R7.9/R7.10).

### E2E / Live Regression (R6 — the headline)
1. `xce_test` chaotic-multi-kill: 3-node cluster, repeated primary SIGKILL interleaved with `jobsub` and config upgrades under deterministic `--wal-sync-ms`; after the run `walcompare` asserts `(term,lsn,pcrc)` committed-prefix identity across all members and `expect-job $jobid` confirms no acked-committed job is absent (R6.1–6.4, R6.6).
2. Negative direction: the same script on **pre-fix** code (term=0) reproduces bug-851 (committed-prefix divergence / job loss); on fixed code it passes (R6.5).

## Resolved Scope Note (descope of former R8)

The former Requirement 8 (full entity-bearing configuration-upgrade snapshot) was a **blocking item** in the prior draft of this design: it required a full materialized-state snapshot at the change boundary, which depends on **entity-bearing snapshots** that do not exist yet (today snapshots are a 16-byte LSN sentinel; the `xzarray_freeze`/`thaw` machinery is built but dormant — the unstarted `entity-bearing-snapshots` backlog item).

**Resolution (2026-06-21): the materialized-state-consistency-on-upgrade work is descoped into the independent [`config-upgrade-snapshot`](../config-upgrade-snapshot/brief.md) spec, which HARD-depends on `entity-bearing-snapshots`. It is out of scope here.**

What this spec ships now (no snapshot dependency):
- R1–R6: per-entry term, monotonic term adoption, `(term,lsn)` reconcile, term-safe committed prefix, and the live chaotic-multi-kill regression — all realized on the library + existing transport.
- R7: the config-change **marker** (consensus, `(term,lsn)`, no bytes) in the unified log, with configuration **bytes** delivered via the existing **out-of-band config-push transport** (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. `config-upgrade-snapshot` later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land.

Constraints preserved by the descope: config bytes are never carried in the WAL marker (R7.2/R7.4), and no member re-derives committed state from a stale separate index — the marker is the consensus authority.