From 8a25c090a9d3dac568c9f96d01be6b3edc29b6b4 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 22 Jun 2026 01:25:01 +0000 Subject: [PATCH] docs(ha-term-safety): approve design + generate implementation tasks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design validated GO (3 task-time refinements, no redesign). Generated tasks.md: 5 major groups, 17 sub-tasks, covering R1-R7. The 3 review issues are pinned as explicit tasks: 1.2 (RequestVote wire schema — restores the controller build), 1.3 (legacy-zero-term matching rule), 2.4 (library commit_index vs own-term WAL_NOOP double-gating resolution). Awaiting tasks approval before /kiro-impl. --- .kiro/specs/ha-term-safety/spec.json | 9 +- .kiro/specs/ha-term-safety/tasks.md | 137 +++++++++++++++++++++++++++ 2 files changed, 142 insertions(+), 4 deletions(-) create mode 100644 .kiro/specs/ha-term-safety/tasks.md diff --git a/.kiro/specs/ha-term-safety/spec.json b/.kiro/specs/ha-term-safety/spec.json index b189bc41..5385a36f 100644 --- a/.kiro/specs/ha-term-safety/spec.json +++ b/.kiro/specs/ha-term-safety/spec.json @@ -3,7 +3,7 @@ "created_at": "2026-06-21T09:34:18Z", "updated_at": "2026-06-21T12:00:00Z", "language": "en", - "phase": "design-generated", + "phase": "tasks-generated", "approvals": { "requirements": { "generated": true, @@ -11,12 +11,13 @@ }, "design": { "generated": true, - "approved": false + "approved": true }, "tasks": { - "generated": false, + "generated": true, "approved": false } }, - "ready_for_implementation": false + "ready_for_implementation": false, + "updated_at_tasks": "2026-06-21T18:00:00Z" } \ No newline at end of file diff --git a/.kiro/specs/ha-term-safety/tasks.md b/.kiro/specs/ha-term-safety/tasks.md new file mode 100644 index 00000000..1cd7e792 --- /dev/null +++ b/.kiro/specs/ha-term-safety/tasks.md @@ -0,0 +1,137 @@ +# Implementation Plan + +> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/controller/testsuite/` (e.g. `test_term_safety_adapter.c`) that fails before the controller change and passes after. Build/run via `/xce-build controller` (builds lib + controller + `controller_unit_test`, ASAN). +> +> **Spec boundary** (applies to ALL tasks): this spec is the **controller-side thin callback adapter + config marker + live regression** over the raft library (`src/lib/raft/raft.{c,h}`), consumed **verbatim**. It does NOT re-implement any Raft protocol decision (logOk, `(term,index)` truncation, current-term commit, freshness, UpdateTerm) — those are owned by `raft-log-replication` + `raft-commit-safety` (both landed, library-side). It does NOT change the commit-barrier / persist-before-ACK durability mechanism (`ha-commit-durability`) — `on_commit_advanced` only TRIGGERS it. The on-disk `wal_entry_hdr_t` (40 bytes) is unchanged — term is a value fix, not a layout change (R1.7). The full entity-bearing config-upgrade snapshot is OUT of scope (`config-upgrade-snapshot`). +> +> **Depends (HARD): `raft-log-replication` + `raft-commit-safety`** — needs the five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`), the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), `commit_index`/`next_index[]`/`match_index[]`, and the extended `(last_log_term, last_log_index)` vote signatures already present in `raft.{c,h}`. +> +> **Build-restoring note:** the upstream vote-signature extension currently breaks the controller build (`cluster.c` vote callers use the old arity). Task 1.2 restores the build; until 1.2 lands, `/xce-build controller` will not compile (`/xce-build lib` stays green). + +## 1. Foundation: wire format, WAL term seam, recovery read-back + +- [ ] 1.1 Extend the AppendEntries replication envelope to carry the Raft term fields + - Add `prev_log_index`, `prev_log_term`, and the entry's `term` to the AppendEntries-carrying repl envelope so a standby persists the leader's term, not a local substitute; regenerate the Cap'n Proto bindings + - Observable completion: a replicated entry's envelope round-trips `(prev_log_index, prev_log_term, entry_term)` across send/decode (unit-asserted on a built envelope); existing replication messages still decode + - _Requirements: 1.3, 1.4_ + - _Boundary: repl transport / typemap (Cap'n Proto)_ + +- [ ] 1.2 Extend the RequestVote/PreVote wire with `(last_log_term, last_log_index)` and restore the controller vote call sites + - Add `last_log_term` (and `last_log_index`) to the RequestVote/PreVote wire message; regenerate bindings; update the controller's vote and pre-vote intake to decode the candidate's tuple and pass it to the extended library `raft_on_vote_request`/`raft_on_pre_vote_request`, sourcing the voter's own tuple via the entry-term read-back at `log_len` + - Observable completion: `/xce-build controller` COMPILES AND LINKS again (the upstream signature break is resolved); a vote intake decodes a candidate `last_log_term` and the library applies the logOk freshness gate + - _Requirements: 2.3, 2.4, 2.6_ + - _Boundary: repl/vote wire (Cap'n Proto), controller vote intake_ + - _Depends: 1.1_ + +- [ ] 1.3 Add the WAL term seam, recovery read-back, and the legacy-zero-term matching rule + - Implement the `persist_set_term` seam (sets the controller's live election term) and a per-LSN recovered term index built during WAL replay, plus an accessor returning the recorded term for an LSN (0 when absent); preserve the 40-byte on-disk header (value fix only) + - Pin the legacy-zero-term rule (REVIEW ISSUE 2): a pre-feature entry carrying term 0 maps to a single, precisely-defined "lowest-known term" for `(term,lsn)` matching so recovery never fails and a legacy entry is never spuriously treated as divergent from another legacy entry; document and test the exact mapping + - Observable completion: after replaying a WAL with mixed real and legacy-0 terms, the accessor returns the recorded term for real entries and the defined lowest-known term for legacy-0 entries; header size unchanged + - _Requirements: 1.1, 1.5, 1.6, 1.7, 1.8_ + - _Boundary: persist (WAL I/O + term value)_ + +## 2. Core: the five callback adapter bodies + +- [ ] 2.1 Implement `on_persist_entry` over the WAL write path + - Stamp `hdr.term` from the library-supplied term (via `persist_set_term`) at the given 1-based index == LSN, through the existing async/replicated WAL write; preserve persist-before-ACK and the bug-850 contiguity guard beneath it + - Observable completion: a library append/accept persists an entry whose recovered `hdr.term` equals the library term; persist failure propagates a non-zero rc and does not ACK + - _Requirements: 1.1, 1.2, 1.3, 1.4_ + - _Depends: 1.3_ + +- [ ] 2.2 Implement `on_get_entry_term` and `on_truncate_from` + - `on_get_entry_term` answers from the recovered/live per-LSN term index (legacy-0 rule from 1.3); `on_truncate_from` deletes entries at index..end via the existing LSN truncate, replacing the committed-boundary heuristic + - Observable completion: the library reads back a stamped entry term via the callback; a truncate-from call removes the local suffix at and above the index and leaves the prefix intact + - _Requirements: 1.5, 1.8, 3.1, 3.2_ + - _Depends: 1.3_ + +- [ ] 2.3 Implement `on_send_append_entries` over the repl envelope + - Build and enqueue an AppendEntries on the repl transport carrying `prev_log_index`, `prev_log_term`, the entry's `term`, and `leader_commit = commit_index`; a heartbeat carries zero entries + - Observable completion: emitting to a peer produces an envelope with the correct prev tuple and entry term; a 0-entry heartbeat is emitted when there is nothing to send + - _Requirements: 1.3, 3.1_ + - _Depends: 1.1_ + +- [ ] 2.4 Implement `on_commit_advanced` as a trigger into the existing barrier, and resolve the commit-gate ownership + - Feed the library-supplied `commit_index` into the existing synchronous commit barrier (`commit_advance_locked`/`commit_set_writable_lsn`) as the commit SOURCE; the barrier mechanism (majority self-cap at durable floor, persist-before-ACK) is unchanged + - Resolve double-gating (REVIEW ISSUE 3): state and implement whether the controller's existing own-term `WAL_NOOP` writable gate is now redundant with the library's current-term commit rule (and removed) or retained as a complementary durability gate; ensure the two cannot deadlock (commit held by both) or leave a gap + - Observable completion: a strict library commit advance releases the barrier's WAIT only on durable majority; the term-blind safe-commit is no longer the commit source; a unit test pins the own-term-gate decision + - _Requirements: 2.2, 4.1, 4.5, 5.4_ + - _Depends: 1.3_ + +- [ ] 2.5 Wire the five callbacks into the controller callback set and the library init restore + - Add the five callbacks to the controller's `s_real_cbs`; on `raft_init` restore `initial_log_len`, `initial_commit_index`, and the recovered last-log term/index from the WAL so the library boots with the durable floor + - Observable completion: a controller node initializes the raft library with all required callbacks non-NULL and restored indices matching its recovered WAL; existing election callbacks remain wired + - _Requirements: 1.5, 2.1_ + - _Depends: 2.1, 2.2, 2.3, 2.4_ + +## 3. Integration: drive the library entry points on the controller paths + +- [ ] 3.1 Drive `raft_log_append` on the primary append path + - Route the primary's job-state append (and the own-term promotion no-op) through `raft_log_append` so the library stamps the term and assigns the LSN, firing `on_persist_entry` + - Observable completion: a primary `jobsub`-driven append goes through the library and lands in the WAL with the live term stamped; the promotion no-op records the newly established term + - _Requirements: 1.1, 1.2, 2.1_ + - _Depends: 2.5_ + +- [ ] 3.2 Route heartbeat/replication intake through `raft_on_append_entries_request` (replaces term-blind intake + committed-boundary reconcile) + - Replace the term-handling in `ha_consume_heartbeat_committed` and the committed-boundary truncate in `ha_rejoin_reconcile` with routing every incoming heartbeat/replication (0- or 1-entry AppendEntries) through the library so `UpdateTerm` (G5), logOk (G2), and `(term,lsn)` conflict-truncate (G3) happen via the callbacks; keep the loud non-promotable retry fence for an unreconcilable member + - Observable completion: a higher-term heartbeat makes the node adopt the term and step down (term recorded); a divergent same-LSN tail is truncated and reconverged; an unreconcilable member stays non-promotable and keeps retrying (no false convergence) + - _Requirements: 2.3, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_ + - _Boundary: ha intake + ha_convergence_ + - _Depends: 2.5_ + +- [ ] 3.3 Drive `raft_on_append_entries_reply` and `raft_advance_commit` on the ACK path + - On a replication ACK, route it through `raft_on_append_entries_reply` to update `match_index[]`/`next_index[]`, then drive `raft_advance_commit` so the library recomputes the current-term-gated commit and may fire `on_commit_advanced` + - Observable completion: a quorum ACK advances the library `commit_index` only when the agreed entry is current-term, which then triggers the barrier; a stale-term reply is dropped + - _Requirements: 4.1, 4.5_ + - _Depends: 2.5, 3.2_ + +## 4. Configuration ordering as a unified `(term,lsn)` marker (R7) + +- [ ] 4.1 Add the `WAL_CONFIG_MARKER` entry type and reference payload + - Add the new WAL entry type and its marker payload carrying only `{version_tag, sha256}` (no config bytes); no on-disk header layout change + - Observable completion: a config marker can be encoded/decoded with its version tag and 32-byte hash and carries no configuration bytes + - _Requirements: 7.2, 7.4_ + - _Boundary: wal_types / config_repl_ + +- [ ] 4.2 Append config markers via the unified log and derive committed config from the latest committed marker + - Append a `WAL_CONFIG_MARKER` through `raft_log_append` (inheriting term-stamp, commit, and `(term,lsn)` reconcile/truncate); derive the committed configuration reference from the latest committed marker within the committed prefix, retiring the separate `committed_config_tag` as the authoritative index + - Observable completion: committing a config change advances the derived committed-config reference; a divergent config marker is truncated by reconcile exactly like a job entry + - _Requirements: 7.1, 7.3, 7.6, 7.7, 7.11_ + - _Depends: 3.1_ + +- [ ] 4.3 Deliver config bytes out-of-band, hash-verify before activate, fence on mismatch + - Key the existing out-of-band config-push transport to the committed marker's `{version_tag, sha256}`; obtain bytes when not held locally, verify their hash against the committed marker before activating; keep a member non-serving under the new configuration (loud fence, keep retrying) on mismatch; on recovery activate the configuration referenced by the latest committed marker + - Observable completion: a committed marker whose bytes match the hash activates; an injected hash mismatch keeps the member fenced and retrying (never activates mismatched config); config bytes never enter a WAL payload + - _Requirements: 7.5, 7.8, 7.9, 7.10_ + - _Depends: 4.2_ + +## 5. Verification: clean-failover preservation, tooling, and the live chaotic regression + +- [ ] 5.1 Adapter callback unit/integration tests + - Cover: term stamp + read-back incl. legacy-0 mapping; higher-term heartbeat adoption via the intake route; truncate-then-persist of a divergent suffix with committed prefix untouched; `on_commit_advanced` releasing the barrier only on durable majority; config marker append + committed-config derive + divergent-marker truncation + - Observable completion: the adapter test binary exercises every callback and the marker path and passes under ASAN + - _Requirements: 1.1, 1.5, 1.8, 2.3, 3.1, 3.2, 4.1, 7.1, 7.6, 7.7_ + - _Depends: 3.3, 4.2_ + +- [ ] 5.2 Clean single-failover preservation integration test + - Assert a single clean failover (one primary killed, one standby promoted, old primary restarted) converges all members to a byte-identical WAL including terms, preserving recovery-on-promotion + advertise-real-LSN (848/849) and the contiguity guard (850); no added steady-state ack latency + - Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green + - _Requirements: 5.1, 5.2, 5.3, 5.4_ + - _Depends: 3.3_ + +- [ ] 5.3 Cross-member `walcompare` divergence tool + - Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder + - Observable completion: the tool returns success on identical member WALs and fails (with the diverging LSN) on a hand-divergent pair + - _Requirements: 6.1, 6.2, 6.3_ + - _Boundary: testsuite/walcompare_ + +- [ ] 5.4 Live chaotic-multi-kill regression (R6 headline) + - Drive a 3-node cluster through repeated primary kills interleaved with `jobsub` and config upgrades under deterministic WAL sync; after the run assert `(term,lsn,pcrc)` committed-prefix identity across all members via `walcompare`, that every acked-committed job is present on every surviving member, and that every member converges to the same committed config reference; confirm the script reproduces bug-851 on pre-fix code (term 0) and passes on the fixed code + - Observable completion: the regression passes on the fixed code and fails on pre-fix code; no acked-committed job is absent; committed config reference identical across members + - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 7.11_ + - _Depends: 5.2, 5.3, 4.3_ + +- [ ] 5.5 Full controller build + suite green under ASAN + - Run `/xce-build controller`; the controller compiles and links (build restored) and all controller unit/integration tests plus the adapter tests pass under ASAN with no regression in election/durability behavior + - Observable completion: controller test binary exits 0; ASAN clean; clean-failover and durability suites still green + - _Requirements: 1.1, 2.3, 3.1, 4.1, 5.1, 6.1, 7.1_ + - _Depends: 5.1, 5.2, 5.4_