myxce/.kiro/specs/ha-term-safety/design.md

Design Document

Overview

Purpose: ha-term-safety is the controller-side integration + config + live-regression umbrella for the Raft TLA+ Conformance library series. It wires the XCE controller as a thin callback adapter to the protocol that raft-log-replication, raft-commit-safety, and raft-sim-conformance implement in src/lib/raft/raft.c, closing bug-851 (silent committed-prefix divergence under chaotic failover) without re-implementing any Raft protocol logic in the controller. (The related configuration-upgrade materialized-state divergence, bug-857, is descoped to config-upgrade-snapshot.)

Users: The XCE HA controller cluster (as Raft nodes), operators relying on "acknowledged ⇒ never lost across any failover pattern", and maintainers needing a live regression that reproduces and fences bug-851.

Impact: Today the replicated log, term propagation, reconcile, and commit are hand-rolled across persist.c/repl.c/ha.c/ha_convergence.c/config_repl.c and match by LSN only (term-blind). After this feature the controller delegates those decisions to the library through five new callbacks and three new entry points, stamps the real election term into the existing 40-byte WAL header, derives committed config from an in-log (term,lsn) marker instead of a separate index, and ships a live chaotic-multi-kill regression. Configuration bytes continue to move through the existing out-of-band config-push transport (OPCode_replConfigPush), hash-verified against the committed marker before activation. The Raft protocol itself (logOk, (term,index) truncation, current-term commit, freshness) is not in this spec — it is realized by the three library specs.

Goals

• Wire the controller adapter to the upstream callbacks verbatim (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced) and entry points (raft_log_append, raft_on_append_entries_request/reply, raft_advance_commit), stamping wal_entry_hdr_t.term from the library's term (R1) and routing incoming heartbeat/replication through the library so term adoption happens on every RPC (R2/R3).
• Replace the term-blind safe-commit (ha_compute_safe_committed_lsn) and committed-boundary reconcile (ha_rejoin_reconcile) with the library's commit_index (via on_commit_advanced) and (term,lsn) truncation (via on_truncate_from), preserving the synchronous commit barrier + persist-before-ACK durability and the clean-failover behaviors of bugs 848/849/850 (R4/R5).
• Order configuration changes through the same (term,lsn) log as a lightweight WAL_CONFIG_MARKER (version_tag + sha256, no bytes); derive committed config from the latest committed marker; deliver the configuration bytes via the existing out-of-band config-push transport, hash-verified against the committed marker before activation (R7).
• Ship a live xce_test chaotic-multi-kill regression asserting cross-member (term,lsn)-fingerprinted committed-prefix identity (R6).

Non-Goals

• The Raft protocol itself — logOk/prevLogTerm match, (term,index) conflict truncation, the Figure-8 current-term commit rule, (lastLogTerm,lastLogIndex) freshness, UpdateTerm-on-any-RPC — owned by raft-log-replication + raft-commit-safety; consumed verbatim.
• The commit-barrier / durability mechanism — the synchronous barrier, group-commit flusher, persist-before-ACK, reply gate, durable-floor freshness — owned by ha-commit-durability; this spec only changes the barrier's input, never its mechanism.
• The in-process conformance oracle — Election Safety / Log Matching / Leader Completeness / State Machine Safety in raft_sim.c — owned by raft-sim-conformance; the R6 live regression complements it, it does not duplicate it.
• Configuration-upgrade materialized-state consistency — establishing a full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state (former R8) — is descoped to the independent config-upgrade-snapshot spec, which HARD-depends on entity-bearing-snapshots. It is out of scope here; this spec delivers the config marker (consensus) plus out-of-band config-push byte delivery, not a full upgrade snapshot.
• Voting-set membership changes / joint consensus (ha-config-bootstrap); bugs 854/847/845/828 (tracked separately).

Boundary Commitments

This Spec Owns

• The controller callback-adapter bodies implementing the five upstream callbacks over the WAL / repl transport / commit barrier (and an in-process route for incoming AppendEntries/heartbeat).
• wal_entry_hdr_t.term value stamping: a persist_set_term seam so every append records the library's live term; reading hdr.term back on replay_wal and feeding it to on_get_entry_term + the library's restore (initial_log_len, last-log-term).
• ha_convergence delegation: replacing the committed-boundary truncate with routing the leader's AppendEntries through raft_on_append_entries_request + on_truncate_from.
• Commit-source switch: feeding the library's commit_index (via on_commit_advanced) into the existing synchronous barrier, retiring the term-blind ha_compute_safe_committed_lsn as the commit source.
• WAL_CONFIG_MARKER (R7): the new WAL entry type + append-via-raft_log_append + committed-config-derived-from-latest-committed-marker, retiring committed_config_tag as an authoritative index.
• Out-of-band config-push byte delivery + hash-verify-before-activate (R7): keying the existing config-push transport (OPCode_replConfigPush) to the committed marker's version_tag + sha256, verifying held bytes against the marker hash before activation, and fencing a member from serving under mismatched configuration.
• The live R6 regression: the chaotic-multi-kill xce_test script + the walcompare cross-member (term,lsn,pcrc) divergence tool.

Out of Boundary

• All Raft protocol decisions (library: raft-log-replication, raft-commit-safety).
• The commit barrier, durability/fsync, persist-before-ACK, durable-floor freshness mechanism (ha-commit-durability).
• The conformance oracle / four-invariant checkers (raft-sim-conformance).
• Configuration-upgrade materialized-state consistency — the full entity-bearing state snapshot at the change boundary (so no member re-derives post-upgrade state) and its xzarray_freeze/thaw + entity encode/decode payload — is descoped to the independent config-upgrade-snapshot spec (which HARD-depends on entity-bearing-snapshots). This spec ships the config marker (consensus) + out-of-band config-push byte delivery; it does not build the upgrade snapshot or entity serialization.
• Membership/quorum transitions (ha-config-bootstrap); the on-disk WAL byte layout (unchanged — value fix only).

Allowed Dependencies

• src/lib/raft/raft.{c,h} after the three upstream specs land: the five callbacks + entry points + raft_append_result_t + commit_index/next_index[]/match_index[] + (lastLogTerm,lastLogIndex) freshness, consumed verbatim.
• The existing controller infrastructure: persist.c WAL/recovery, repl.c transport, commit.c synchronous barrier + durability (ha-commit-durability), the out-of-band config-push transport (OPCode_replConfigPush, config_repl.c/ha.c), the ten already-wired election callbacks (s_real_cbs, ha.c:1309-1320).
• The test infrastructure: waldump.{c,h}, xce_test script primitives, /xce-deploy-ha.

Revalidation Triggers

Re-check this design if any of the following change:

• The signature/semantics of any upstream callback (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced) or entry point (raft_log_append, raft_send_append_entries, raft_on_append_entries_request/reply, raft_advance_commit), including raft_append_result_t.
• The 1-based-index ↔ WAL-LSN identity or the "term 0 == absent" convention.
• The addition of last_log_term to raft_on_vote_request/raft_on_pre_vote_request (the controller must plumb it onto the wire and read it from hdr.term on recovery).
• The commit_index restore convention (initial_commit_index) or initial_log_len.
• The config-push transport contract (OPCode_replConfigPush) or the marker's {version_tag, sha256} key — the byte delivery is hash-verified against the committed marker before activation.
• The synchronous commit barrier / persist-before-ACK contract (ha-commit-durability) — on_commit_advanced must continue to trigger it unchanged.
• The split of the configuration-upgrade materialized-state work into config-upgrade-snapshot — if that spec changes how it consumes the (term,lsn) config marker, re-check the marker contract here.

Architecture

Existing Architecture Analysis

The controller already drives the election half of the library through ten callbacks wired in s_real_cbs (ha.c:1309-1320): vote/PreVote send, freshness gate (raft_cb_vote_eligible, ha.c:299-360), promotion/demotion (raft_cb_become_leader/follower, ha.c:880-1242), and term persistence (raft_cb_persist → ha_persist_term, ha.c:1251-1294, 2441-2474). The controller uses only src/lib/raft/raft.c (no legacy src/controller/raft.c). The log half is hand-rolled and term-blind:

• Every WAL append stamps hdr.term = p->term (persist.c:588/726/1040) but p->term is never assigned → term 0 everywhere (G1). The 40-byte header already has the term field (wal_types.h:29-39) — a value fix, not a layout change.
• Incoming heartbeats/replication never adopt a higher term (ha_consume_heartbeat_committed drops stale only, ha.c:5013-5036) (G5).
• Reconcile truncates by committed-boundary, not (term,lsn) (ha_rejoin_reconcile, ha_convergence.c:295-366; persist_truncate_wal_above LSN-only, persist.c:2455-2565) (G3).
• Safe commit is computed term-blind from voter LSNs (ha_compute_safe_committed_lsn, ha.c:480-509) (G4); config commits through a separate committed_config_tag with no term (config_repl.c:941-979), with the bytes pushed out-of-band via OPCode_replConfigPush — the marker (consensus) will replace the term-blind index, while the byte transport is reused as-is.

The durability layer (ha-commit-durability) is correct and must be preserved: majority_candidate_locked self-caps at persist_durable_lsn (commit.c:79-138), persist-before-ACK on standbys (persist_wal_write_replicated), the pre-promotion eligibility fence (ha.c:933-951), advertise-real-durable-floor freshness, and the contiguity guard (persist.c:691-706, bug-850).

This design follows the same Ports-and-Adapters discipline already established by the election callbacks: the library decides; the controller performs I/O.

Architecture Pattern & Boundary Map

flowchart TB
    subgraph Library["src/lib/raft/raft.c — PROTOCOL OWNER (NOT this spec)"]
        APP["raft_log_append (ClientRequest)"]
        AEREQ["raft_on_append_entries_request<br/>UpdateTerm + logOk + truncate"]
        AEREP["raft_on_append_entries_reply"]
        ADV["raft_advance_commit<br/>current-term Figure-8"]
        VOTE["vote freshness lastLogTerm,lastLogIndex"]
    end
    subgraph Adapter["src/controller — THIN ADAPTER (this spec)"]
        PE["on_persist_entry<br/>persist_set_term + WAL write"]
        GET["on_get_entry_term<br/>WAL index / replay read-back"]
        SEND["on_send_append_entries<br/>repl envelope prevLogTerm+entryTerm"]
        TRUNC["on_truncate_from<br/>persist_truncate_wal_above"]
        COMMIT["on_commit_advanced<br/>TRIGGER existing sync barrier"]
        ROUTE["heartbeat/repl intake -> raft_on_append_entries_request<br/>(replaces ha_consume_heartbeat + ha_rejoin_reconcile)"]
    end
    subgraph Config["Config marker + out-of-band byte delivery (this spec)"]
        MARK["WAL_CONFIG_MARKER<br/>version_tag + sha256 via raft_log_append"]
        CPUSH["config-push byte delivery<br/>OPCode_replConfigPush, hash-verify vs marker before activate"]
    end
    subgraph Preserved["PRESERVED policy (other specs)"]
        BARRIER["sync commit barrier + persist-before-ACK<br/>ha-commit-durability"]
        CFGXPORT["existing config-push transport<br/>OPCode_replConfigPush"]
    end
    subgraph Test["Live regression (this spec)"]
        WC["walcompare term,lsn,pcrc"]
        SCRIPT["xce_test chaotic multi-kill"]
    end
    APP --> PE
    AEREQ --> GET & PE & TRUNC
    ROUTE --> AEREQ
    AEREP --> SEND
    ADV --> COMMIT
    COMMIT --> BARRIER
    MARK --> APP
    CPUSH --> CFGXPORT
    SCRIPT --> WC

Architecture Integration:

• Selected pattern: thin callback adapter (Ports-and-Adapters), identical to the election-callback pattern already in ha.c. The library owns protocol; the controller owns I/O.
• Domain boundaries: protocol decisions (append/truncate/commit/vote) in raft.c; durable I/O (WAL write, repl send, barrier, config-push transport) in the controller; the marker + out-of-band byte delivery in this spec; the configuration-upgrade materialized-state snapshot is descoped to config-upgrade-snapshot.
• Existing patterns preserved: s_real_cbs callback wiring, persist-before-ACK, the synchronous barrier, the pre-promotion fence, the contiguity guard, "controller never self-terminates".
• Steering compliance: ha-async-io.md (no blocking in iomgr/vtask; barrier on its own path), persistence.md (durable floor / acknowledged⇒durable preserved; never persist commit index), error-handling.md (status returns, fail-loud non-promotable fence, never exit/abort), testing.md (TDD; tests in testsuite/ before C changes).

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Protocol core	src/lib/raft/raft.c (gnu17, after 3 upstream specs)	Owns Raft log/commit/freshness	Consumed verbatim; not modified here
Adapter / WAL	src/controller/persist.c + wal_types.h	hdr.term stamping, recovery read-back, truncate	40-byte header unchanged; persist_set_term seam added
Adapter / transport	src/controller/repl.c, typemap.h	carry prevLogTerm + entry term on AppendEntries	extend repl envelope fields
Adapter / commit	src/controller/commit.c (ha-commit-durability)	barrier triggered by on_commit_advanced	mechanism unchanged
Config	src/controller/config_repl.c + wal_types.h	WAL_CONFIG_MARKER, marker-derived committed config	retires committed_config_tag index
Config bytes	existing config-push transport (OPCode_replConfigPush, config_repl.c/ha.c)	deliver bytes out-of-band; hash-verify vs marker before activate	transport reused as-is; no snapshot
Test	xce_test + waldump.{c,h} + /xce-deploy-ha	live chaotic regression + walcompare	~90% infra exists

File Structure Plan

Modified Files

• src/include/wal_types.h — add WAL_CONFIG_MARKER = 50 to wal_entry_type; add the marker payload struct wal_config_marker_t { uint64_t version_tag; uint8_t sha256[32]; } (no config bytes). No change to wal_entry_hdr_t (40 bytes preserved, R1.7).
• src/include/persist.h — declare persist_set_term(persist_t *p, uint64_t term); declare recovery accessors that surface per-entry term + restored log_len/last-log-term for the library init.
• src/controller/persist.c — implement persist_set_term (sets p->term); make replay_wal retain hdr.term per LSN (in-memory term index) so on_get_entry_term and library restore can read it; persist_truncate_wal_above reused unchanged by on_truncate_from.
• src/controller/repl.c + src/include/typemap.h — extend the AppendEntries-carrying envelope (wal_catchup_entry_t/capnp_repl_envelope_t) to carry prev_log_index, prev_log_term, and the entry's term explicitly, so a standby persists the leader's term (not local p->term).
• src/controller/ha.c — wire the five new callbacks into s_real_cbs (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced); add the heartbeat/replication intake that routes through raft_on_append_entries_request (replacing the term-blind ha_consume_heartbeat_committed term handling); drive raft_log_append on the primary append path; pass last_log_term on the vote wire and decode it on intake; retire ha_compute_safe_committed_lsn as the commit source (the library's commit_index drives the barrier via on_commit_advanced).
• src/controller/ha_convergence.c — replace ha_rejoin_reconcile's committed-boundary truncate with delegation: route the leader's AppendEntries through the library; on_truncate_from performs the truncate. Keep the non-promotable retry-fence behavior (R3.5/R5).
• src/controller/config_repl.c — do_upgrade_config_with_bytes appends a WAL_CONFIG_MARKER via raft_log_append (replacing the committed_config_tag push/commit); derive committed config from the latest committed marker in the committed prefix; key the existing out-of-band config-push (OPCode_replConfigPush) to the marker's {version_tag, sha256}, hash-verify the held bytes against the committed marker before activating, and fence a member from serving under mismatched configuration (R7.5/R7.8–R7.10). The full entity-bearing upgrade snapshot is out of scope (config-upgrade-snapshot).
• src/controller/commit.c — no mechanism change; on_commit_advanced calls into the existing commit_advance_locked/commit_set_writable_lsn path with the library-supplied commit_index.

New Files

• src/controller/testsuite/walcompare.{c,h} — cross-member WAL diff by (lsn → term, pcrc); nonzero exit on committed-prefix divergence. Reuses the waldump.c reader/decoder (roadmap-planned tooling, roadmap.md:55).
• src/controller/testsuite/test_term_safety_adapter.c — unit/integration tests for the adapter callbacks (term stamping, on_get_entry_term read-back, truncate delegation, on_commit_advanced barrier trigger, marker append/derive).
• src/tools/test-term-safety-chaos.sh (or an xce_test script under testsuite/) — the R6 chaotic-multi-kill driver invoking walcompare.

Each file keeps one responsibility: protocol stays in raft.c (untouched); persist.c = WAL I/O + term value; ha.c/ha_convergence.c = callback bodies + intake routing; config_repl.c = marker + out-of-band config-push byte delivery; testsuite/ = regression. The File Structure Plan reflects the boundary: no protocol logic file is created or modified, and no upgrade-snapshot file is added (descoped to config-upgrade-snapshot).

System Flows

Replicated append → standby persist with the leader's term (R1, R2, R3)

sequenceDiagram
    participant P as Primary library (raft.c)
    participant PA as Primary adapter (this spec)
    participant SA as Standby adapter (this spec)
    participant S as Standby library (raft.c)
    Note over P: raft_log_append(value) — stamps s->term
    P->>PA: on_persist_entry(idx, term, entry)
    PA->>PA: persist_set_term(p, term) then WAL write hdr.term=term
    Note over P: raft_send_append_entries(peer, commit_index)
    P->>PA: on_send_append_entries(peer, prevIdx, prevTerm, leaderCommit, entry+entryTerm)
    PA-->>SA: repl envelope (carries prevLogTerm + entryTerm)
    Note over S: raft_on_append_entries_request()<br/>UpdateTerm FIRST (G5) then logOk (G2)
    S->>SA: on_get_entry_term(prevIdx)  %% logOk
    alt term conflict at idx (G3)
        S->>SA: on_truncate_from(idx) then on_persist_entry(idx, leader_term)
    else append
        S->>SA: on_persist_entry(idx, leader_term)  %% standby persists LEADER term
    end
    Note over SA: persist-before-ACK preserved -> WalAck only on durable

Flow-level decisions:

• The standby persists the leader's entry term (carried explicitly on the envelope), closing the persist_wal_write_replicated "best-effort p->term" hazard. persist_set_term keeps p->term aligned with the library term for the local-append path.
• Incoming heartbeats are 0-entry AppendEntries routed through the same raft_on_append_entries_request, so a higher-term heartbeat triggers UpdateTerm (G5) — the controller no longer needs ha_consume_heartbeat_committed to adopt term.
• The contiguity guard (bug-850) and persist-before-ACK (ha-commit-durability) are unchanged and sit beneath on_persist_entry.

Commit advance → existing barrier; config marker (R4, R7)

sequenceDiagram
    participant L as Leader library (raft.c)
    participant A as Adapter (this spec)
    participant B as Sync commit barrier (ha-commit-durability, PRESERVED)
    Note over L: replication updated match_index[] (library)
    Note over L: raft_advance_commit() — current-term Figure-8 (G4)
    alt commit_index strictly advances
        L->>A: on_commit_advanced(commit_index)
        A->>B: feed safe commit position into existing barrier
        Note over B: majority self-cap at durable_lsn + persist-before-ACK<br/>+ own-term writable gate (UNCHANGED)
    else held (prev-term / no quorum)
        Note over L: no callback, barrier untouched
    end
    Note over A: committed config = latest WAL_CONFIG_MARKER<br/>within committed prefix (derived, not a separate index)

Flow-level decisions:

• on_commit_advanced triggers the barrier; it never performs durability itself. ha_compute_safe_committed_lsn is retired as the commit source (the library's current-term-gated commit_index replaces it), but the post-election truncate-to-safe at promotion and the durable self-cap remain (R5).
• A config change is an ordinary committed log entry: the WAL_CONFIG_MARKER commits through the same barrier and reconciles through the same (term,lsn) matching (R7.1/R7.3/R7.6/R7.11). The committed config is the latest committed marker (R7.7).

Configuration bytes — out-of-band config-push, hash-verified against the committed marker (R7.5, R7.8–R7.10)

flowchart TB
    M["WAL_CONFIG_MARKER commits at term,lsn<br/>(version_tag + sha256, no bytes)"] --> NEED{"local bytes match<br/>marker sha256?"}
    NEED -->|no| PUSH["obtain bytes via existing config-push<br/>OPCode_replConfigPush, keyed to version_tag"]
    PUSH --> VERIFY["verify obtained bytes sha256 == marker hash"]
    NEED -->|yes| VERIFY
    VERIFY -->|match| ACTIVATE["activate configuration"]
    VERIFY -->|mismatch| FENCE["serve-fence: keep member non-serving<br/>under new config; retry (R7.9)"]

Flow-level decision: the marker (consensus) is the authority; the bytes travel out-of-band on the existing config-push transport (OPCode_replConfigPush), never in a WAL payload (R7.4). A member obtains and hash-verifies the bytes against the committed marker before activating (R7.8), fences itself on mismatch (R7.9), and on recovery activates the configuration referenced by the latest committed marker (R7.10). The full materialized-state upgrade snapshot (former R8) is out of scope — see config-upgrade-snapshot.

Requirements Traceability

Requirement	Summary	Adapter component(s)	Library/upstream realization	Flow
1.1, 1.2	record live/promotion term in entry	on_persist_entry + persist_set_term; raft_log_append (incl. own-term NOOP)	library stamps s->term	Append flow
1.3, 1.4	transmit + persist leader term on replication	on_send_append_entries carries entryTerm; standby on_persist_entry writes leader term	raft_on_append_entries_request	Append flow
1.5	recovery reads back per-entry term	replay_wal term index; on_get_entry_term; restore initial_log_len/last-log-term	library init restore	—
1.6	no zero/placeholder term while term valid	persist_set_term keeps p->term live	—	—
1.7	preserve 40-byte WAL layout (value fix)	wal_entry_hdr_t unchanged	—	—
1.8	legacy zero term = lowest-known for matching	on_get_entry_term maps legacy 0 → lowest-known	library (term,index) match tolerant	—
2.1, 2.2	persist won term before appending; not committed until majority	raft_cb_persist→ha_persist_term; own-term WAL_NOOP gate (commit_set_writable_lsn)	BecomeLeader + commit rule	Commit flow
2.3, 2.4, 2.6	no re-elect at stale term; adopt higher term on any RPC	heartbeat/repl intake → raft_on_append_entries_request; on_persist	UpdateTerm (G5)	Append flow
2.5	at most one leader advances committed per term	library current-term commit + freshness	raft-commit-safety G4/G6	Commit flow
3.1, 3.2	reconcile by (term,lsn); truncate divergent suffix	ha_convergence delegation → on_truncate_from	raft_on_append_entries_request (G3)	Append flow
3.3	never truncate committed prefix	library truncates only above match point; on_truncate_from honors it	library	—
3.4	reconciled prefix byte-identical to leader	standby persists leader entry+term	library	Append flow
3.5, 3.6	non-promotable retry; no false committed	keep loud non-promotable fence; do not report divergent as committed	ha-commit-durability fence preserved	—
4.1, 4.5	advance committed only on (term,lsn) quorum + current-term	on_commit_advanced feeds barrier with library commit_index	raft_advance_commit (G4)	Commit flow
4.2, 4.3, 4.4	committed entry byte-identical/preserved across leadership	library Leader Completeness + barrier durability	library + ha-commit-durability	Commit flow
5.1	clean single failover converges identically	unchanged promotion truncate-to-safe + library matching	raft_cb_become_leader	—
5.2	recovery-on-promotion + advertise-real-LSN (848/849)	raft_cb_become_leader + raft_cb_vote_eligible unchanged	preserved	—
5.3	contiguity guard, no forward gap (850)	persist_wal_write_replicated guard unchanged beneath on_persist_entry	preserved	—
5.4	no added steady-state ack latency	barrier path untouched; on_commit_advanced only changes source	preserved	—
6.1, 6.2, 6.3	chaotic regression asserts (term,lsn,content) prefix identity	walcompare + chaos script	—	—
6.4	no acked-committed job absent post-run	expect-job $jobid per acked submit	—	—
6.5	pre-fix repro fails, post-fix passes	negative-direction assertion (term=0 pre-fix)	—	—
6.6	config changes interleaved → same committed config ref	walcompare checks committed marker tag+hash	R7 derive	Config flow
7.1–7.4, 7.6, 7.7, 7.11	config as (term,lsn) marker, no bytes, term-stamped, reconciled, derived, majority-committed	WAL_CONFIG_MARKER via raft_log_append; marker-derived committed config	library log/commit/reconcile	Config marker flow
7.5, 7.8, 7.9, 7.10	bytes via out-of-band config-push; verify hash before activate; fence on mismatch; activate from latest committed marker on recovery	config-push byte delivery (OPCode_replConfigPush) keyed to marker {version_tag, sha256} + hash-verify-before-activate + serve-fence	existing config-push transport	Config bytes flow

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
WAL term adapter	Adapter	Stamp/read-back the library term in hdr.term	1.1–1.8	library callbacks (P0), persist.c (P0)	Service, State
AppendEntries intake/route	Adapter	Route heartbeat/repl through the library; term adoption	2.3–2.6, 3.1–3.4	raft_on_append_entries_request (P0), repl envelope (P0)	Service
Reconcile delegation	Adapter	Replace committed-boundary truncate with on_truncate_from	3.1–3.6	library (P0), persist_truncate_wal_above (P0), fence (P1)	Service
Commit-source switch	Adapter	Feed library commit_index into existing barrier	4.1–4.5, 5.4	on_commit_advanced (P0), commit.c barrier (P0)	Service
Clean-failover preservation	Adapter	Keep 848/849/850 behaviors unchanged	5.1–5.4	promotion/vote callbacks, contiguity guard (P0)	Service
Config marker	Adapter	WAL_CONFIG_MARKER in unified log; derive committed config	7.1–7.4,7.6,7.7,7.11	raft_log_append (P0), config_repl.c (P0)	Service, State
Config byte delivery	Adapter	Out-of-band config-push keyed to marker; hash-verify-before-activate; serve-fence on mismatch	7.5,7.8,7.9,7.10	config-push transport OPCode_replConfigPush (P0), config_repl.c (P0)	Service
Live regression	Test	Chaotic-multi-kill + walcompare	6.1–6.6	xce_test, waldump, /xce-deploy-ha (P0)	Test

Adapter Callback → Controller Implementation Site (the core seam table)

Library callback / entry point (upstream)	Controller implementation site	Behavior
on_persist_entry(ctx,index,term,entry,len)	ha.c callback → persist_set_term(p,term) + persist_wal_async/persist_wal_write_replicated (persist.c:588/726)	Stamp hdr.term=term at LSN=index; persist-before-ACK preserved.
on_get_entry_term(ctx,index)->term|0	ha.c callback → in-memory WAL term index from replay_wal/live append	Return recorded hdr.term for LSN; legacy 0 → lowest-known (R1.8); 0 if absent.
on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)	ha.c callback → repl.c envelope build (typemap.h extended fields)	Enqueue AppendEntries carrying prevLogTerm + entry term; leaderCommit = commit_index.
on_truncate_from(ctx,index)	ha.c callback → persist_truncate_wal_above(p,index-1) (persist.c:2455)	Delete LSN≥index; replaces ha_rejoin_reconcile committed-boundary heuristic.
on_commit_advanced(ctx,commit_index)	ha.c callback → commit.c existing barrier (commit_advance_locked/commit_set_writable_lsn)	Feed safe commit position; trigger barrier, do not replace it.
raft_log_append(s,entry,len,&idx)	primary append path in ha.c/config_repl.c (incl. WAL_CONFIG_MARKER)	Library stamps s->term, assigns LSN; drives on_persist_entry.
raft_on_append_entries_request(...)	heartbeat/repl intake in ha.c (replaces ha_consume_heartbeat_committed term handling)	Library does UpdateTerm+logOk+truncate/append via callbacks.
raft_on_append_entries_reply(...)	repl ACK intake in ha.c	Library updates match_index[]/next_index[].
raft_advance_commit(s)	driven after repl ACK in ha.c	Library computes current-term-gated commit_index; may fire on_commit_advanced.
raft_on_vote_request/pre_vote_request(...,last_log_term,...)	existing vote callbacks in ha.c, extended	Controller decodes candidate last_log_term from wire; reads own via on_get_entry_term(log_len).

WAL term adapter

Field	Detail
Intent	Make hdr.term carry the library's election term on every entry and surface it on recovery
Requirements	1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8

Responsibilities & Constraints

• persist_set_term(persist_t *p, uint64_t term) assigns p->term (the missing seam); called by on_persist_entry and on term adoption so no append stamps 0 while a valid term is in effect (R1.6). On-disk wal_entry_hdr_t is unchanged (40 bytes, R1.7).
• replay_wal retains each hdr.term in an in-memory per-LSN index so on_get_entry_term answers from recovered state (R1.5); a legacy entry with term 0 maps to the lowest-known term for matching (R1.8).
• Standbys persist the leader's entry term (carried on the envelope), not local p->term (R1.3/R1.4).

Contracts: Service [x] / State [x]

Service Interface

/* Set the controller's live election term so subsequent WAL appends stamp it
 * into hdr.term (R1.1/R1.6). Called from on_persist_entry and on term adoption.
 * No on-disk layout change; value fix only (R1.7). */
void persist_set_term(persist_t *p, uint64_t term);

/* Recovered per-entry term for an LSN (1-based == index), feeding on_get_entry_term
 * and the library's init restore. Returns 0 when absent; legacy-0 is mapped to the
 * lowest-known term by the adapter for (term,lsn) matching (R1.8). */
uint64_t persist_entry_term(const persist_t *p, uint64_t lsn);

• Preconditions: p valid; lsn within [1, current_lsn).
• Postconditions: on_get_entry_term(lsn) returns the recorded term; appends stamp the live term.
• Invariants: hdr.term never 0 while a valid term is in effect; 40-byte layout preserved.

Config marker

Field	Detail
Intent	Order config changes through the unified (term,lsn) log as a lightweight marker
Requirements	7.1, 7.2, 7.3, 7.4, 7.6, 7.7, 7.11

Responsibilities & Constraints

• do_upgrade_config_with_bytes (config_repl.c:1001-1163) appends WAL_CONFIG_MARKER (wal_config_marker_t = {version_tag, sha256[32]}, no bytes, R7.2/R7.4) via raft_log_append, so it inherits term-stamping (R7.3), commit (R7.11), and (term,lsn) reconcile/truncate (R7.6).
• Committed config is derived from the latest committed marker within the committed prefix (R7.7); the separate committed_config_tag index is retired as authoritative (it may remain as a derived cache).

Contracts: Service [x] / State [x]

Service Interface

/* Append a config-change marker (version_tag + sha256, no config bytes) to the
 * unified replicated log via raft_log_append. Returns RAFT_OK / the append rc.
 * The config bytes travel out-of-band via config-push (R7.5), never the marker. */
int config_append_marker(cluster_t *c, uint64_t version_tag,
                         const uint8_t sha256[32], uint64_t *out_lsn);

/* Derive the committed configuration reference from the latest WAL_CONFIG_MARKER
 * at or below committed_lsn (R7.7). */
int config_committed_marker(const cluster_t *c, uint64_t *version_tag,
                            uint8_t sha256_out[32]);

• Preconditions: leader for append; valid sha256.
• Postconditions: marker committed ⇒ committed config reference advances; reconcile truncates a divergent marker exactly like a job entry.
• Invariants: config bytes never enter a WAL payload (R7.4).

Config byte delivery

Field	Detail
Intent	Deliver config bytes out-of-band via the existing config-push transport; hash-verify against the committed marker before activation
Requirements	7.5, 7.8, 7.9, 7.10

Responsibilities & Constraints

• Move the configuration bytes through the existing out-of-band config-push transport (OPCode_replConfigPush, config_repl.c/ha.c), keyed to the committed marker's {version_tag, sha256} — never through a WAL payload (R7.4/R7.5).
• When a committed marker references bytes not yet held locally, obtain them via config-push and verify the obtained bytes' sha256 against the marker hash before activating (R7.8).
• If a member's locally held bytes do not match the latest committed marker's hash, keep the member from serving under the new configuration and keep retrying rather than activating mismatched config (R7.9) — the "controller never self-terminates" loud non-promotable fence, never exit/abort.
• On recovery, activate the configuration referenced by the latest committed marker, obtaining the bytes from out-of-band storage (R7.10).

Contracts: Service [x]

Service Interface

/* Ensure the bytes for the latest committed config marker are held locally and
 * hash-verified against the marker, then activate. Obtains bytes via config-push
 * (OPCode_replConfigPush) when absent. Returns RAFT_OK on activate; a non-success
 * rc keeps the member fenced (non-serving under the new config) and retrying
 * (R7.8/R7.9/R7.10). */
int config_activate_committed(cluster_t *c);

• Preconditions: a committed WAL_CONFIG_MARKER exists in the committed prefix.
• Postconditions: on success the active config matches the marker's {version_tag, sha256}; on hash mismatch the member stays fenced and retries.
• Invariants: bytes are never sourced from a WAL payload; activation only after hash-verify against the committed marker.

Note

: the full materialized-state upgrade snapshot (embed entities + config in one artifact, install on every member so no member re-derives post-upgrade state — former R8) is out of scope here and lives in the independent config-upgrade-snapshot spec, which HARD-depends on entity-bearing-snapshots. This spec delivers config bytes via the interim out-of-band config-push path that config-upgrade-snapshot later supersedes for the upgrade case.

Live regression (R6)

Field	Detail
Intent	Reproduce bug-851 and assert cross-member (term,lsn) committed-prefix identity
Requirements	6.1, 6.2, 6.3, 6.4, 6.5, 6.6

Responsibilities & Constraints

• walcompare <dirA> <dirB> ... diffs members by (lsn → term, pcrc); nonzero exit on any committed-prefix divergence (R6.1/R6.2/R6.3), reusing waldump.c's decoder.
• The chaos script drives repeated primary kills interleaved with jobsub + config upgrades via /xce-deploy-ha + kill_node/start_node + deterministic --wal-sync-ms, then asserts expect-job $jobid for every acked submit (R6.4) and walcompare identity incl. committed config marker (R6.6).
• R6.5: on pre-fix code (term=0) the repro must fail (walcompare detects divergence / expect-job loss); on fixed code it passes.

Contracts: Test [x]

Data Models

Domain Model

• Log entry: (index==LSN, term, type, payload) — the unified (term,lsn) log for both job state and config markers. Index 1-based == WAL LSN (library convention).
• Config-change marker: wal_config_marker_t = {uint64_t version_tag, uint8_t sha256[32]} — a reference, never the bytes.
• Committed config: derived value = the latest WAL_CONFIG_MARKER at or below committed_lsn.

Logical Data Model

• wal_entry_type gains WAL_CONFIG_MARKER = 50 (between WAL_NOOP=40 and WAL_CHECKPOINT=100).
• persist_t.term becomes a written field via persist_set_term; a recovered per-LSN term index is added to persist_t (in-memory only).
• The repl envelope (wal_catchup_entry_t/capnp_repl_envelope_t) gains explicit prev_log_index, prev_log_term, and entry term fields on the AppendEntries path.
• No on-disk WAL layout change (R1.7): wal_entry_hdr_t stays 40 bytes with term at offset 0.

Data Contracts & Integration

• AppendEntries wire: carries (prevLogIndex, prevLogTerm, leaderCommit, entry, entryTerm) — the standby persists entryTerm, not local p->term.
• Vote wire: carries (lastLogTerm, lastLogIndex); the controller decodes lastLogTerm and passes it to the library; own value via on_get_entry_term(log_len).
• Config bytes: delivered out-of-band via config-push (OPCode_replConfigPush), keyed to the marker's {version_tag, sha256}, hash-verified against the committed marker before activation; never in a WAL payload. The full materialized-state upgrade snapshot is out of scope (config-upgrade-snapshot).

Error Handling

Error Strategy

The adapter follows "controller never self-terminates" (error-handling.md, ha-async-io.md): callbacks return status; on a non-recoverable condition the controller enters a loud, non-promotable fence and keeps retrying — it never exit/abort. Persist-before-ACK and the durable self-cap are unchanged.

Error Categories and Responses

• Persist failure (on_persist_entry rc≠0) → propagate; library does not advance the local log; standby does not ACK (persist-before-ACK preserved).
• Truncate failure (on_truncate_from rc≠0) → report a non-success request result so the leader retries; the member stays non-promotable until reconciled (R3.5).
• Unreconcilable member → loud permanent fence, non-promotable, keep retrying catch-up (R3.5/R5; never report false convergence, R3.6).
• Config hash mismatch (R7.9) → keep the member from serving under the new config; retry; do not activate mismatched config (config_activate_committed returns non-success; member stays fenced).
• Config bytes absent for a committed marker (R7.8) → obtain via config-push and hash-verify before activate; do not activate on a missing or mismatched payload.

Monitoring

Reuse raft_cb_trace and existing INFO/ERROR logging for: term-stamp, term-adopt-on-AE, conflict-truncate, commit-advance-trigger, config-marker-commit, config-byte-verify, and the loud fences.

Testing Strategy

TDD-mandatory (testing.md): tests in src/controller/testsuite/ precede C changes; gate on the ANSI-stripped :FAIL: count; run final durability/regression validation on a real fsync-able filesystem.

Unit Tests (adapter callbacks, in test_term_safety_adapter.c)

1. on_persist_entry stamps hdr.term from the library term (via persist_set_term) and on_get_entry_term reads it back; legacy-0 maps to lowest-known (R1.1–1.8).
2. AppendEntries intake routes a higher-term heartbeat through raft_on_append_entries_request, the library adopts the term, and ha_persist_term records it (R2.3–2.6 / G5).
3. on_truncate_from deletes the divergent suffix and the standby persists the leader entry+term; committed prefix untouched (R3.1–3.4 / G3).
4. on_commit_advanced feeds the library commit_index into the existing barrier and the WAIT entry releases only on durable majority; ha_compute_safe_committed_lsn no longer the commit source (R4.1/R4.5, R5.4).
5. config_append_marker appends WAL_CONFIG_MARKER (no bytes) and config_committed_marker derives committed config from the latest committed marker; a divergent marker truncates like a job entry (R7.1–7.7).

Integration Tests

1. Clean single failover converges to byte-identical WALs incl. terms, preserving recovery-on-promotion + advertise-real-LSN + contiguity (R5.1–5.3).
2. Marker commit + reconcile under a forced leader change keeps every member on the same committed config reference (R6.6, R7.11).
3. Config byte delivery: a committed marker whose bytes are obtained via config-push activates only after the bytes' sha256 matches the marker hash; an injected hash mismatch keeps the member fenced (non-serving under the new config) and retrying (R7.8/R7.9/R7.10).

E2E / Live Regression (R6 — the headline)

1. xce_test chaotic-multi-kill: 3-node cluster, repeated primary SIGKILL interleaved with jobsub and config upgrades under deterministic --wal-sync-ms; after the run walcompare asserts (term,lsn,pcrc) committed-prefix identity across all members and expect-job $jobid confirms no acked-committed job is absent (R6.1–6.4, R6.6).
2. Negative direction: the same script on pre-fix code (term=0) reproduces bug-851 (committed-prefix divergence / job loss); on fixed code it passes (R6.5).

Resolved Scope Note (descope of former R8)

The former Requirement 8 (full entity-bearing configuration-upgrade snapshot) was a blocking item in the prior draft of this design: it required a full materialized-state snapshot at the change boundary, which depends on entity-bearing snapshots that do not exist yet (today snapshots are a 16-byte LSN sentinel; the xzarray_freeze/thaw machinery is built but dormant — the unstarted entity-bearing-snapshots backlog item).

Resolution (2026-06-21): the materialized-state-consistency-on-upgrade work is descoped into the independent config-upgrade-snapshot spec, which HARD-depends on entity-bearing-snapshots. It is out of scope here.

What this spec ships now (no snapshot dependency):

• R1–R6: per-entry term, monotonic term adoption, (term,lsn) reconcile, term-safe committed prefix, and the live chaotic-multi-kill regression — all realized on the library + existing transport.
• R7: the config-change marker (consensus, (term,lsn), no bytes) in the unified log, with configuration bytes delivered via the existing out-of-band config-push transport (OPCode_replConfigPush), hash-verified against the committed marker before activation. config-upgrade-snapshot later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land.

Constraints preserved by the descope: config bytes are never carried in the WAL marker (R7.2/R7.4), and no member re-derives committed state from a stale separate index — the marker is the consensus authority.