docs(ha-term-safety): approve design + generate implementation tasks
Design validated GO (3 task-time refinements, no redesign). Generated tasks.md: 5 major groups, 17 sub-tasks, covering R1-R7. The 3 review issues are pinned as explicit tasks: 1.2 (RequestVote wire schema — restores the controller build), 1.3 (legacy-zero-term matching rule), 2.4 (library commit_index vs own-term WAL_NOOP double-gating resolution). Awaiting tasks approval before /kiro-impl.
This commit is contained in:
parent
91eb9ca3fa
commit
8a25c090a9
2 changed files with 142 additions and 4 deletions
|
|
@ -3,7 +3,7 @@
|
|||
"created_at": "2026-06-21T09:34:18Z",
|
||||
"updated_at": "2026-06-21T12:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "design-generated",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
@ -11,12 +11,13 @@
|
|||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"generated": true,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
"ready_for_implementation": false,
|
||||
"updated_at_tasks": "2026-06-21T18:00:00Z"
|
||||
}
|
||||
137
.kiro/specs/ha-term-safety/tasks.md
Normal file
137
.kiro/specs/ha-term-safety/tasks.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Implementation Plan
|
||||
|
||||
> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/controller/testsuite/` (e.g. `test_term_safety_adapter.c`) that fails before the controller change and passes after. Build/run via `/xce-build controller` (builds lib + controller + `controller_unit_test`, ASAN).
|
||||
>
|
||||
> **Spec boundary** (applies to ALL tasks): this spec is the **controller-side thin callback adapter + config marker + live regression** over the raft library (`src/lib/raft/raft.{c,h}`), consumed **verbatim**. It does NOT re-implement any Raft protocol decision (logOk, `(term,index)` truncation, current-term commit, freshness, UpdateTerm) — those are owned by `raft-log-replication` + `raft-commit-safety` (both landed, library-side). It does NOT change the commit-barrier / persist-before-ACK durability mechanism (`ha-commit-durability`) — `on_commit_advanced` only TRIGGERS it. The on-disk `wal_entry_hdr_t` (40 bytes) is unchanged — term is a value fix, not a layout change (R1.7). The full entity-bearing config-upgrade snapshot is OUT of scope (`config-upgrade-snapshot`).
|
||||
>
|
||||
> **Depends (HARD): `raft-log-replication` + `raft-commit-safety`** — needs the five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`), the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), `commit_index`/`next_index[]`/`match_index[]`, and the extended `(last_log_term, last_log_index)` vote signatures already present in `raft.{c,h}`.
|
||||
>
|
||||
> **Build-restoring note:** the upstream vote-signature extension currently breaks the controller build (`cluster.c` vote callers use the old arity). Task 1.2 restores the build; until 1.2 lands, `/xce-build controller` will not compile (`/xce-build lib` stays green).
|
||||
|
||||
## 1. Foundation: wire format, WAL term seam, recovery read-back
|
||||
|
||||
- [ ] 1.1 Extend the AppendEntries replication envelope to carry the Raft term fields
|
||||
- Add `prev_log_index`, `prev_log_term`, and the entry's `term` to the AppendEntries-carrying repl envelope so a standby persists the leader's term, not a local substitute; regenerate the Cap'n Proto bindings
|
||||
- Observable completion: a replicated entry's envelope round-trips `(prev_log_index, prev_log_term, entry_term)` across send/decode (unit-asserted on a built envelope); existing replication messages still decode
|
||||
- _Requirements: 1.3, 1.4_
|
||||
- _Boundary: repl transport / typemap (Cap'n Proto)_
|
||||
|
||||
- [ ] 1.2 Extend the RequestVote/PreVote wire with `(last_log_term, last_log_index)` and restore the controller vote call sites
|
||||
- Add `last_log_term` (and `last_log_index`) to the RequestVote/PreVote wire message; regenerate bindings; update the controller's vote and pre-vote intake to decode the candidate's tuple and pass it to the extended library `raft_on_vote_request`/`raft_on_pre_vote_request`, sourcing the voter's own tuple via the entry-term read-back at `log_len`
|
||||
- Observable completion: `/xce-build controller` COMPILES AND LINKS again (the upstream signature break is resolved); a vote intake decodes a candidate `last_log_term` and the library applies the logOk freshness gate
|
||||
- _Requirements: 2.3, 2.4, 2.6_
|
||||
- _Boundary: repl/vote wire (Cap'n Proto), controller vote intake_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 1.3 Add the WAL term seam, recovery read-back, and the legacy-zero-term matching rule
|
||||
- Implement the `persist_set_term` seam (sets the controller's live election term) and a per-LSN recovered term index built during WAL replay, plus an accessor returning the recorded term for an LSN (0 when absent); preserve the 40-byte on-disk header (value fix only)
|
||||
- Pin the legacy-zero-term rule (REVIEW ISSUE 2): a pre-feature entry carrying term 0 maps to a single, precisely-defined "lowest-known term" for `(term,lsn)` matching so recovery never fails and a legacy entry is never spuriously treated as divergent from another legacy entry; document and test the exact mapping
|
||||
- Observable completion: after replaying a WAL with mixed real and legacy-0 terms, the accessor returns the recorded term for real entries and the defined lowest-known term for legacy-0 entries; header size unchanged
|
||||
- _Requirements: 1.1, 1.5, 1.6, 1.7, 1.8_
|
||||
- _Boundary: persist (WAL I/O + term value)_
|
||||
|
||||
## 2. Core: the five callback adapter bodies
|
||||
|
||||
- [ ] 2.1 Implement `on_persist_entry` over the WAL write path
|
||||
- Stamp `hdr.term` from the library-supplied term (via `persist_set_term`) at the given 1-based index == LSN, through the existing async/replicated WAL write; preserve persist-before-ACK and the bug-850 contiguity guard beneath it
|
||||
- Observable completion: a library append/accept persists an entry whose recovered `hdr.term` equals the library term; persist failure propagates a non-zero rc and does not ACK
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.4_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 2.2 Implement `on_get_entry_term` and `on_truncate_from`
|
||||
- `on_get_entry_term` answers from the recovered/live per-LSN term index (legacy-0 rule from 1.3); `on_truncate_from` deletes entries at index..end via the existing LSN truncate, replacing the committed-boundary heuristic
|
||||
- Observable completion: the library reads back a stamped entry term via the callback; a truncate-from call removes the local suffix at and above the index and leaves the prefix intact
|
||||
- _Requirements: 1.5, 1.8, 3.1, 3.2_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 2.3 Implement `on_send_append_entries` over the repl envelope
|
||||
- Build and enqueue an AppendEntries on the repl transport carrying `prev_log_index`, `prev_log_term`, the entry's `term`, and `leader_commit = commit_index`; a heartbeat carries zero entries
|
||||
- Observable completion: emitting to a peer produces an envelope with the correct prev tuple and entry term; a 0-entry heartbeat is emitted when there is nothing to send
|
||||
- _Requirements: 1.3, 3.1_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2.4 Implement `on_commit_advanced` as a trigger into the existing barrier, and resolve the commit-gate ownership
|
||||
- Feed the library-supplied `commit_index` into the existing synchronous commit barrier (`commit_advance_locked`/`commit_set_writable_lsn`) as the commit SOURCE; the barrier mechanism (majority self-cap at durable floor, persist-before-ACK) is unchanged
|
||||
- Resolve double-gating (REVIEW ISSUE 3): state and implement whether the controller's existing own-term `WAL_NOOP` writable gate is now redundant with the library's current-term commit rule (and removed) or retained as a complementary durability gate; ensure the two cannot deadlock (commit held by both) or leave a gap
|
||||
- Observable completion: a strict library commit advance releases the barrier's WAIT only on durable majority; the term-blind safe-commit is no longer the commit source; a unit test pins the own-term-gate decision
|
||||
- _Requirements: 2.2, 4.1, 4.5, 5.4_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 2.5 Wire the five callbacks into the controller callback set and the library init restore
|
||||
- Add the five callbacks to the controller's `s_real_cbs`; on `raft_init` restore `initial_log_len`, `initial_commit_index`, and the recovered last-log term/index from the WAL so the library boots with the durable floor
|
||||
- Observable completion: a controller node initializes the raft library with all required callbacks non-NULL and restored indices matching its recovered WAL; existing election callbacks remain wired
|
||||
- _Requirements: 1.5, 2.1_
|
||||
- _Depends: 2.1, 2.2, 2.3, 2.4_
|
||||
|
||||
## 3. Integration: drive the library entry points on the controller paths
|
||||
|
||||
- [ ] 3.1 Drive `raft_log_append` on the primary append path
|
||||
- Route the primary's job-state append (and the own-term promotion no-op) through `raft_log_append` so the library stamps the term and assigns the LSN, firing `on_persist_entry`
|
||||
- Observable completion: a primary `jobsub`-driven append goes through the library and lands in the WAL with the live term stamped; the promotion no-op records the newly established term
|
||||
- _Requirements: 1.1, 1.2, 2.1_
|
||||
- _Depends: 2.5_
|
||||
|
||||
- [ ] 3.2 Route heartbeat/replication intake through `raft_on_append_entries_request` (replaces term-blind intake + committed-boundary reconcile)
|
||||
- Replace the term-handling in `ha_consume_heartbeat_committed` and the committed-boundary truncate in `ha_rejoin_reconcile` with routing every incoming heartbeat/replication (0- or 1-entry AppendEntries) through the library so `UpdateTerm` (G5), logOk (G2), and `(term,lsn)` conflict-truncate (G3) happen via the callbacks; keep the loud non-promotable retry fence for an unreconcilable member
|
||||
- Observable completion: a higher-term heartbeat makes the node adopt the term and step down (term recorded); a divergent same-LSN tail is truncated and reconverged; an unreconcilable member stays non-promotable and keeps retrying (no false convergence)
|
||||
- _Requirements: 2.3, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
|
||||
- _Boundary: ha intake + ha_convergence_
|
||||
- _Depends: 2.5_
|
||||
|
||||
- [ ] 3.3 Drive `raft_on_append_entries_reply` and `raft_advance_commit` on the ACK path
|
||||
- On a replication ACK, route it through `raft_on_append_entries_reply` to update `match_index[]`/`next_index[]`, then drive `raft_advance_commit` so the library recomputes the current-term-gated commit and may fire `on_commit_advanced`
|
||||
- Observable completion: a quorum ACK advances the library `commit_index` only when the agreed entry is current-term, which then triggers the barrier; a stale-term reply is dropped
|
||||
- _Requirements: 4.1, 4.5_
|
||||
- _Depends: 2.5, 3.2_
|
||||
|
||||
## 4. Configuration ordering as a unified `(term,lsn)` marker (R7)
|
||||
|
||||
- [ ] 4.1 Add the `WAL_CONFIG_MARKER` entry type and reference payload
|
||||
- Add the new WAL entry type and its marker payload carrying only `{version_tag, sha256}` (no config bytes); no on-disk header layout change
|
||||
- Observable completion: a config marker can be encoded/decoded with its version tag and 32-byte hash and carries no configuration bytes
|
||||
- _Requirements: 7.2, 7.4_
|
||||
- _Boundary: wal_types / config_repl_
|
||||
|
||||
- [ ] 4.2 Append config markers via the unified log and derive committed config from the latest committed marker
|
||||
- Append a `WAL_CONFIG_MARKER` through `raft_log_append` (inheriting term-stamp, commit, and `(term,lsn)` reconcile/truncate); derive the committed configuration reference from the latest committed marker within the committed prefix, retiring the separate `committed_config_tag` as the authoritative index
|
||||
- Observable completion: committing a config change advances the derived committed-config reference; a divergent config marker is truncated by reconcile exactly like a job entry
|
||||
- _Requirements: 7.1, 7.3, 7.6, 7.7, 7.11_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 4.3 Deliver config bytes out-of-band, hash-verify before activate, fence on mismatch
|
||||
- Key the existing out-of-band config-push transport to the committed marker's `{version_tag, sha256}`; obtain bytes when not held locally, verify their hash against the committed marker before activating; keep a member non-serving under the new configuration (loud fence, keep retrying) on mismatch; on recovery activate the configuration referenced by the latest committed marker
|
||||
- Observable completion: a committed marker whose bytes match the hash activates; an injected hash mismatch keeps the member fenced and retrying (never activates mismatched config); config bytes never enter a WAL payload
|
||||
- _Requirements: 7.5, 7.8, 7.9, 7.10_
|
||||
- _Depends: 4.2_
|
||||
|
||||
## 5. Verification: clean-failover preservation, tooling, and the live chaotic regression
|
||||
|
||||
- [ ] 5.1 Adapter callback unit/integration tests
|
||||
- Cover: term stamp + read-back incl. legacy-0 mapping; higher-term heartbeat adoption via the intake route; truncate-then-persist of a divergent suffix with committed prefix untouched; `on_commit_advanced` releasing the barrier only on durable majority; config marker append + committed-config derive + divergent-marker truncation
|
||||
- Observable completion: the adapter test binary exercises every callback and the marker path and passes under ASAN
|
||||
- _Requirements: 1.1, 1.5, 1.8, 2.3, 3.1, 3.2, 4.1, 7.1, 7.6, 7.7_
|
||||
- _Depends: 3.3, 4.2_
|
||||
|
||||
- [ ] 5.2 Clean single-failover preservation integration test
|
||||
- Assert a single clean failover (one primary killed, one standby promoted, old primary restarted) converges all members to a byte-identical WAL including terms, preserving recovery-on-promotion + advertise-real-LSN (848/849) and the contiguity guard (850); no added steady-state ack latency
|
||||
- Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.4_
|
||||
- _Depends: 3.3_
|
||||
|
||||
- [ ] 5.3 Cross-member `walcompare` divergence tool
|
||||
- Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder
|
||||
- Observable completion: the tool returns success on identical member WALs and fails (with the diverging LSN) on a hand-divergent pair
|
||||
- _Requirements: 6.1, 6.2, 6.3_
|
||||
- _Boundary: testsuite/walcompare_
|
||||
|
||||
- [ ] 5.4 Live chaotic-multi-kill regression (R6 headline)
|
||||
- Drive a 3-node cluster through repeated primary kills interleaved with `jobsub` and config upgrades under deterministic WAL sync; after the run assert `(term,lsn,pcrc)` committed-prefix identity across all members via `walcompare`, that every acked-committed job is present on every surviving member, and that every member converges to the same committed config reference; confirm the script reproduces bug-851 on pre-fix code (term 0) and passes on the fixed code
|
||||
- Observable completion: the regression passes on the fixed code and fails on pre-fix code; no acked-committed job is absent; committed config reference identical across members
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 7.11_
|
||||
- _Depends: 5.2, 5.3, 4.3_
|
||||
|
||||
- [ ] 5.5 Full controller build + suite green under ASAN
|
||||
- Run `/xce-build controller`; the controller compiles and links (build restored) and all controller unit/integration tests plus the adapter tests pass under ASAN with no regression in election/durability behavior
|
||||
- Observable completion: controller test binary exits 0; ASAN clean; clean-failover and durability suites still green
|
||||
- _Requirements: 1.1, 2.3, 3.1, 4.1, 5.1, 6.1, 7.1_
|
||||
- _Depends: 5.1, 5.2, 5.4_
|
||||
Loading…
Add table
Add a link
Reference in a new issue