myxce/.kiro/specs/ha-term-safety/research.md

Research & Design Decisions

Summary

• Feature: ha-term-safety
• Discovery Scope: Complex Integration (controller-side adapter + config marker + out-of-band config-push byte delivery + live regression umbrella for the Raft TLA+ Conformance library series; the configuration-upgrade materialized-state snapshot is descoped to config-upgrade-snapshot).
• Key Findings:
  • The WAL on-disk header already carries wal_entry_hdr_t.term at offset 0 of a 40-byte layout (src/include/wal_types.h:29-39). R1.7 ("preserve on-disk layout, value fix only") is satisfied by the format as-is. The defect is purely that persist_t.term (src/include/persist.h:160) is never assigned anywhere — every hdr.term = p->term site (persist.c:588, 726, 1040) therefore stamps 0. The fix is a persist_set_term seam fed by the library's term, plus reading hdr.term back on replay (replay_wal reads but discards it today, persist.c:~1500-1620).
  • The controller uses the new library src/lib/raft/raft.c exclusively (no legacy src/controller/raft.c exists); ten election callbacks are already wired via s_real_cbs (ha.c:1309-1320). The log/commit callbacks defined by the three upstream specs (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced) are not yet wired — that wiring is this spec's core deliverable.
  • G5 (term not adopted on the replication/heartbeat path) is confirmed in code: only raft_on_vote_request/reply route a higher term into raft state (raft.c:125-134 → raft_cb_persist → ha_persist_term, ha.c:1251-1294, 2441-2474). The heartbeat consume path (ha_consume_heartbeat_committed, ha.c:5013-5036) only drops stale heartbeats; it never adopts a higher term. With the library now owning UpdateTerm-on-any-RPC, the controller must route incoming heartbeat/replication RPCs through raft_on_append_entries_request.
  • G3 (reconcile is committed-boundary, not (term,lsn)) is confirmed: ha_rejoin_reconcile (ha_convergence.c:295-366) clamps the truncation cutoff to committed_lsn and truncates the suffix blindly via persist_truncate_wal_above (LSN-only, persist.c:2455-2565). It cannot detect same-LSN/different-term divergence. The library's raft_on_append_entries_request + on_truncate_from replace this decision.
  • R8 SNAPSHOT FEASIBILITY → RESOLVED BY DESCOPE (2026-06-21): the snapshot-feasibility investigation found that snapshots are a 16-byte LSN sentinel stub today ("XCESNAP\0" + checkpoint_lsn; write_snapshot persist.c:1068-1101, load_snapshot persist.c:1280-1307) carrying zero entity state; the COW machinery a full upgrade snapshot needs (xzarray_freeze/xzarray_thaw, zarray.c:695-757) is built and unit-tested but dormant (kvmgr.c:297-298 vfreeze = NULL /* wired in P0-2 */, zero production callers); the roadmap's "entity-bearing snapshots (Task 11)" (roadmap.md:43) is unstarted. Because the full configuration-upgrade materialized-state snapshot is a HARD dependency on entity-bearing-snapshots that does not exist, the former Requirement 8 was descoped into the independent config-upgrade-snapshot spec (which HARD-depends on entity-bearing-snapshots). ha-term-safety ships R1–R7 with no snapshot dependency: R7's config marker (consensus) is in-log, and the config bytes are delivered via the existing out-of-band config-push transport (OPCode_replConfigPush), hash-verified against the committed marker before activation. The config marker never depended on entity-bearing snapshots; only the descoped upgrade snapshot did.

Research Log

Upstream library callback contract (the seam this spec wires to)

• Context: This spec must delegate the Raft protocol to the library and wire the controller as a thin adapter, reusing the upstream callback names verbatim.
• Sources Consulted: .kiro/specs/raft-log-replication/design.md, .kiro/specs/raft-commit-safety/design.md, .kiro/specs/raft-sim-conformance/design.md, TLA-gap-analysis.md.
• Findings:
  • raft-log-replication owns: raft_log_append, raft_send_append_entries, raft_on_append_entries_request, raft_on_append_entries_reply, raft_append_result_t; callbacks on_persist_entry(ctx,index,term,entry,len), on_get_entry_term(ctx,index)->term|0, on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...), on_truncate_from(ctx,index); per-peer next_index[]/match_index[]; index 1-based == WAL LSN; term 0 == absent; UpdateTerm applied on every RPC (G5); conflict branch = truncate-then-append in one handler call (G3).
  • raft-commit-safety owns: raft_advance_commit(s) with the Figure-8 current-term rule (G4); on_commit_advanced(ctx,commit_index) fired only on strict advance — triggers the existing barrier, never replaces it; vote/PreVote freshness becomes (lastLogTerm,lastLogIndex) lexicographic (G6), with the candidate's last_log_term added to raft_on_vote_request/raft_on_pre_vote_request signatures.
  • raft-sim-conformance owns the in-process oracle (Election Safety / Log Matching / Leader Completeness / State Machine Safety) — the verification cross-reference; the live regression here (R6) complements it (the sim proves the library; the live test proves the controller adapter wiring).
• Implications: Every R1–R6 protocol invariant is realized by the library; ha-term-safety implements the adapter callback bodies + the term plumbing on the wire + the reconcile delegation + the live regression. The controller must NOT re-implement logOk, conflict truncation, the current-term commit rule, or freshness — those live in raft.c.

WAL term stamping & persistence seam (R1, R2)

• Sources Consulted: wal_types.h:29-59, persist.h:160, persist.c:588/726/1040/1500-1620/2455-2565, typemap.h:338-387, repl.c:239-311.
• Findings:
  • wal_entry_hdr_t is 40 bytes, _Static_assert-locked, term already present at offset 0 → R1.7 is free.
  • persist_t.term exists but grep "p->term =" / "->term =" returns no assignment in persist.c → every entry stamps term 0 (G1). Fix: a persist_set_term(p, term) seam called from the adapter when the library establishes/adopts a term.
  • Replication transport: the active wire format is wal_catchup_entry_t (typemap.h:338-351) which has no term field; the term currently rides implicitly inside the serialized header bytes. capnp_repl_envelope_t.term exists (typemap.h:380-387) but is a test-only mapped type. The adapter's on_send_append_entries must carry prevLogTerm + the entry's term explicitly so a standby persists the leader's term, not its local p->term (closing the persist_wal_write_replicated "best-effort p->term" hazard at persist.c:719-726).
  • replay_wal reads hdr.term but discards it; recovery must surface per-entry term to feed on_get_entry_term and restore initial_log_len/last-log-term for the library at init.
  • Next free WAL type id: WAL_NOOP=40, WAL_CHECKPOINT=100 → WAL_CONFIG_MARKER = 50 slots cleanly (R7).

Term adoption, reconcile, commit, promotion seams (R2, R3, R4, R5)

• Sources Consulted: ha.c:113-1320, 480-509, 880-1033, 5013-5036, 5338-5402, ha_convergence.c:240-384, commit.c:79-395, cluster.c:1200-1299, myapp.c standby branch, persist.c:691-706.
• Findings:
  • R2/G5: heartbeat/replication never adopt a higher term (ha_consume_heartbeat_committed only drops stale). Routing incoming heartbeat/repl through raft_on_append_entries_request makes the library do UpdateTerm; the adapter's on_persist (already ha_persist_term) durably records it. Monotonic-term durability before appending as leader (R2.1/2.2) is realized by the library's BecomeLeader + the own-term WAL_NOOP gate already present (raft_cb_become_leader arms commit_set_writable_lsn, ha.c:880-1033 / commit.c:383-395).
  • R3/G3: ha_rejoin_reconcile committed-boundary clamp replaced by routing the leader's AppendEntries through raft_on_append_entries_request; the library decides match/truncate/append and calls on_truncate_from (adapter → persist_truncate_wal_above) + on_persist_entry.
  • R4/G4: ha_compute_safe_committed_lsn (ha.c:480-509) is term-blind (collects voter LSNs/tags, no term). It is superseded by the library's commit_index (current-term-gated) delivered via on_commit_advanced, which triggers the existing synchronous barrier. The barrier and durability (majority_candidate_locked self-caps at persist_durable_lsn, commit.c:79-138; persist-before-ACK; wait_done_locked own-term gate) are PRESERVED AS POLICY (ha-commit-durability) — only the input (safe commit position) changes source.
  • R5: clean-failover behaviors to preserve — recovery-on-promotion + truncate-to-safe (raft_cb_become_leader, ha.c:917-919), pre-promotion eligibility fence (ha.c:933-951, bugs 848/849), advertise-real durable floor for freshness (raft_cb_vote_eligible, ha.c:299-360), and the contiguity guard refusing a forward gap (persist_wal_write_replicated, persist.c:691-706, bug-850). These remain; the library's (term,index) matching is strictly additive on top.

Config replication seam (R7)

• Sources Consulted: config_repl.c:941-979, 1001-1163, 1857-1944, ha.c:5338-5402, 4980-5036, ha_convergence.c:372-384.
• Findings: config commits today through a separate committed_config_tag index (push_config_to_standbys waits majority ACK then atomic_store committed_config_tag, config_repl.c:965-970), carried on heartbeats (ha.c:5343-5345), with no term and tag-based anti-entropy — entirely outside the WAL (term,lsn) log. This is the second parallel violation. R7 replaces it: a WAL_CONFIG_MARKER entry (version_tag + sha256, not bytes) appended via raft_log_append, ordered/committed/reconciled in the same (term,lsn) log; committed config derived from the latest committed marker within the committed prefix (replaces the separate index). Config bytes continue to travel via the existing out-of-band config-push transport (OPCode_replConfigPush), now keyed to the marker's {version_tag, sha256} and hash-verified against the committed marker before activation — never in the WAL record. (The reverted byte-delivery decision: the prior draft routed bytes through a full upgrade snapshot; that snapshot is descoped to config-upgrade-snapshot, so the byte transport reverts to the existing config-push path.)

Live regression infrastructure (R6)

• Sources Consulted: src/controller/testsuite/waldump.{c,h}, test.c:129-1760, .kiro/specs/ha-commit-durability/repro-851/, INVESTIGATION-failover-job-loss.md, test_commit_barrier_rollout.c, src/tools/deploy-ha.sh, test-commit-durability.sh, roadmap.md:55.
• Findings: Infrastructure is ~90% present.
  • waldump_path() (waldump.h:42) already decodes per-entry (LSN, TERM, TYPE, PLEN, PCRC) and validates the CRC chain, with a payload-CRC fingerprint for cross-member comparison and nonzero exit on gaps/crc-break.
  • xce_test script primitives exist with exit-code verdicts: jobsub (captures $jobid), wait-primary, expect-jobs N, expect-job $jobid, ctrlstatus, waldump, !shell escape, $?/$jobid vars, g_session_failed → process exit code (test.c:129-1760).
  • /xce-deploy-ha + deploy-ha.sh/start.sh launch a 3-node cluster; kill_node/start_node + deterministic --wal-sync-ms 30000 drive chaotic kills (test-commit-durability.sh:113-198).
  • Gaps to build: (a) waldump currently prints TERM=0 for all entries because of G1 — once R1 lands it will show real terms; the R6 test must assert per-LSN term identity across members; (b) a walcompare <dirA> <dirB> cross-member (term,lsn,pcrc) divergence diff (already on the roadmap as planned tooling, roadmap.md:55) is the missing assertion primitive. R6 builds the chaotic-multi-kill script + walcompare, reusing all existing primitives.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Thin callback adapter (chosen)	Controller implements the upstream callback ports over WAL/repl/commit; library owns all protocol decisions	Honors the TLA-gap-analysis DECISION; one consistency domain; no protocol logic duplicated in controller; sim already proves the library	Requires deleting/neutralizing the term-blind controller logic (ha_compute_safe_committed_lsn, committed-boundary reconcile) rather than extending it	Matches Ports-and-Adapters already established by the election callbacks
Patch the controller's split log (rejected)	Add (term,lsn) matching directly into ha_convergence.c + ha_compute_safe_committed_lsn	No library dependency	Re-implements raft.tla in the controller; two consistency domains; cannot be sim-verified; explicitly rejected by the 2026-06-21 DECISION	Rejected
Config bytes in WAL (rejected)	Carry config bytes in WAL_CONFIG_MARKER payload	Self-contained log	Bloats the log; R7.2/R7.4 forbid it	Rejected by requirements

Design Decisions

Decision: Unify R1–R6 protocol onto the library; controller is a thin adapter

• Context: TLA-gap-analysis DECISION (2026-06-21) mandates the library own the protocol; the controller adapts.
• Alternatives Considered: (1) thin adapter (chosen); (2) patch controller split-log (rejected — duplicates protocol, unverifiable).
• Selected Approach: Wire five new library callbacks to controller I/O sites; route incoming heartbeat/replication through raft_on_append_entries_request; drive append via raft_log_append; replace the term-blind safe-commit and committed-boundary reconcile with the library's commit_index/on_truncate_from. No protocol logic is added to the controller.
• Rationale: Single consistency domain; the deterministic sim already verifies the protocol, so the live regression need only prove the wiring.
• Trade-offs: Must neutralize existing term-blind controller paths (a deletion, not an extension); upstream signature changes are revalidation triggers.
• Follow-up: Confirm the upstream callback/entry-point signatures are stable before wiring (Revalidation Triggers below).

Decision: Generalize job-state and config-change onto one (term,lsn) replicated log (R7)

• Context: Config is a second parallel consistency domain (committed_config_tag, no term) exposed to the same divergence.
• Selected Approach: Record each config change as a lightweight WAL_CONFIG_MARKER (version_tag + sha256, no bytes) appended via raft_log_append so it inherits term-safety, reconcile, and commit. Derive committed config from the latest committed marker; retire the separate committed_config_tag commit index.
• Rationale: One log, one reconcile, one commit rule — the Generalization lens: config change is a special case of "an ordered committed log entry".
• Trade-offs: Config bytes are delivered out-of-band via the existing config-push transport (OPCode_replConfigPush), hash-verified against the committed marker before activation — no snapshot dependency. During migration, committed_config_tag becomes a derived value, not an authoritative index.
• Follow-up: R7's marker is implementable now (no entity-bearing snapshots needed); the config bytes ride the existing config-push install path. The full materialized-state upgrade snapshot is descoped to config-upgrade-snapshot — see the resolved decision below.

Decision (RESOLVED 2026-06-21): former R8 descoped into config-upgrade-snapshot; R7 reverts to out-of-band config-push byte delivery

• Context: The former Requirement 8 required a full entity-bearing state snapshot embedding the new config, installed on every member, so no member replays pre-upgrade WAL under the new config (the bug-857 hazard).
• Finding (the prior blocker): Snapshots are a 16-byte LSN sentinel (persist.c:1068-1101/1280-1307); the entity-serialization machinery (xzarray_freeze/thaw) is dormant (kvmgr.c:297); roadmap "entity-bearing snapshots (Task 11)" is unstarted (roadmap.md:43). The full upgrade snapshot is therefore a HARD dependency on entity-bearing-snapshots, which does not exist.
• Alternatives Considered:
  1. Make entity-bearing-snapshots a hard prerequisite and sequence the upgrade snapshot after it, inside this spec.
  2. Carry config bytes in the WAL marker — violates R7.2/R7.4.
  3. Per-node replay under new config — the exact bug-857 hazard.
  4. Descope the materialized-state-on-upgrade work into an independent spec (selected) — split the entity-snapshot blocker out of ha-term-safety so R1–R7 can ship now.
• Selected Approach: The configuration-upgrade materialized-state consistency (former R8) is descoped into the independent config-upgrade-snapshot spec, which HARD-depends on entity-bearing-snapshots. ha-term-safety retains R7's consensus half (the in-log (term,lsn) config marker) and reverts the configuration-byte delivery to the existing out-of-band config-push transport (OPCode_replConfigPush), keyed to the marker's {version_tag, sha256} and hash-verified against the committed marker before activation. No snapshot is built or depended on here. config-upgrade-snapshot later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land.
• Rationale: Do not paper over an infeasible requirement, and do not block the implementable R1–R7 (protocol term-safety + config marker) on an unstarted entity-serialization backlog. The marker + commit + reconcile (R7) never required entities; only the materialized-state install did. The byte delivery falls back to the already-working config-push path.
• Trade-offs: ha-term-safety no longer delivers the full upgrade-snapshot guarantee (now owned by config-upgrade-snapshot); the interim config-push byte delivery is superseded for the upgrade case once that spec lands.
• Follow-up: config-upgrade-snapshot brief + requirements written (../config-upgrade-snapshot/{brief.md,requirements.md}); it anchors on this spec's (term,lsn) config marker and consumes entity-bearing-snapshots.

Decision: Preserve the commit barrier + durability + clean-failover behaviors as policy

• Context: R4 and R5 must not regress ha-commit-durability (bugs 661/828) or bugs 848/849/850.
• Selected Approach: on_commit_advanced only triggers the existing synchronous barrier; majority_candidate_locked self-cap, persist-before-ACK, the pre-promotion fence, advertise-real-durable-floor freshness, and the contiguity guard are unchanged. The library supplies the safe commit position; the durable act stays in the controller.
• Rationale: The barrier/durability is verified by tests + the Protocol Verification Checklist (not by raft.tla, which assumes a durable log); changing source-of-truth for the commit position must not change the durability mechanism.
• Trade-offs: None functional; R4.x latency for the steady state is unchanged (R5.4) because the barrier path is untouched.

Risks & Mitigations

• Former R8 infeasible without entity-bearing snapshots → RESOLVED by descoping the configuration-upgrade materialized-state work into the independent config-upgrade-snapshot spec (HARD-depends on entity-bearing-snapshots); ha-term-safety ships R1–R7 with out-of-band config-push byte delivery and no snapshot dependency.
• Upstream signature drift (the three library specs are generated but not yet implemented) → Listed as Revalidation Triggers; the adapter wires to the exact published signatures; if a signature changes, re-check the adapter table.
• Neutralizing term-blind controller paths could regress clean failover → R5 acceptance criteria + the existing durability tests are the guard; the live R6 regression asserts both chaotic AND clean convergence.
• waldump shows TERM=0 pre-fix → expected (G1); R6 asserts non-zero, per-LSN-consistent terms post-fix, and the negative direction (pre-fix repro must fail) per R6.5.
• Migration of committed_config_tag to marker-derived → derive-not-store; on first boot after upgrade, a cluster with no marker treats config as term-0/lowest-known (R1.8 analogue) until the first marker commits.

References

• .kiro/specs/ha-term-safety/requirements.md — R1–R7 (former R8 descoped 2026-06-21).
• .kiro/specs/config-upgrade-snapshot/{brief.md,requirements.md} — the descoped configuration-upgrade materialized-state snapshot (HARD-depends on entity-bearing-snapshots).
• .kiro/specs/ha-term-safety/TLA-gap-analysis.md — gaps G1–G6 + the library-owns-protocol DECISION.
• .kiro/specs/raft-log-replication/design.md, .kiro/specs/raft-commit-safety/design.md, .kiro/specs/raft-sim-conformance/design.md — upstream callback contracts (consumed verbatim).
• .kiro/steering/persistence.md, ha-async-io.md, roadmap.md:13-21,43,55 — durability policy, process-lifecycle invariant, Raft conformance series, Task 11 backlog, planned walcompare tooling.
• .kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md + repro-851/ — bug-851/857 evidence and the regression baseline.