From 9f530f5742e7d12402612f7306b3029d8ce06904 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 7 Jul 2026 08:45:41 +0000 Subject: [PATCH] docs: fix unchecked group-header checkboxes across 8 fully-implemented specs controller-status, ha-commit-engine, ha-heartbeat-committed-propagation, ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and raft-ha-core-safety each had every numbered sub-task marked [x] but their group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real unchecked sub-tasks in any of the eight files. No task content changed. --- .kiro/specs/controller-status/tasks.md | 12 ++++++------ .kiro/specs/ha-commit-engine/tasks.md | 12 ++++++------ .../ha-heartbeat-committed-propagation/tasks.md | 6 +++--- .kiro/specs/ha-prevote/tasks.md | 12 ++++++------ .kiro/specs/ha-single-leader-role/tasks.md | 12 ++++++------ .kiro/specs/ha-wal-replication/tasks.md | 8 ++++---- .kiro/specs/libxce-failover/tasks.md | 10 +++++----- .kiro/specs/raft-ha-core-safety/tasks.md | 14 +++++++------- 8 files changed, 43 insertions(+), 43 deletions(-) diff --git a/.kiro/specs/controller-status/tasks.md b/.kiro/specs/controller-status/tasks.md index ca71669..4a896e3 100644 --- a/.kiro/specs/controller-status/tasks.md +++ b/.kiro/specs/controller-status/tasks.md @@ -3,14 +3,14 @@ > TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor. > The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2). -- [ ] 1. Foundation: status reply schema + codegen +- [x] 1. Foundation: status reply schema + codegen - [x] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema - Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count. - Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory. - Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean. - _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_ -- [ ] 2. Controller: status handler + any-role dispatch +- [x] 2. Controller: status handler + any-role dispatch - [x] 2.1 Implement the controller status handler - Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role. - Route the new opcode to this handler in the controller request dispatch switch. @@ -26,7 +26,7 @@ - _Boundary: ha_dispatch_gate_ - _Depends: 2.1_ -- [ ] 3. libxce client API +- [x] 3. libxce client API - [x] 3.1 Add the failover-aware any-node status call - Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success. - TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure. @@ -42,7 +42,7 @@ - _Boundary: libxce client_ - _Depends: 3.1_ -- [ ] 4. Test client command + wait-primary integration +- [x] 4. Test client command + wait-primary integration - [x] 4.1 Add the `ctrlstatus` client command - Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by ()" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query. - Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure. @@ -56,14 +56,14 @@ - _Boundary: cmd_wait_primary_ - _Depends: 3.2_ -- [ ] 5. Validation (live cluster) +- [x] 5. Validation (live cluster) - [x] 5.1 End-to-end status + authoritative wait-primary on a live cluster - Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by " framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves. - Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks. - _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_ - _Depends: 4.1, 4.2_ -- [ ] 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11) +- [x] 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11) > Sequential, single request path (schema → typemap → `ctrlstatus_fill` → render → E2E); no `(P)`. Read-only reporting only — the live-fresh-committed heartbeat change is OUT of boundary (route to ha-heartbeat-liveness / ha-repl-hardening). - [x] 6.1 Add the responder snapshot timestamp field to the schema + hand-authored typemap - Append a responder snapshot-time field to the status payload at the next free ordinal (after the peer count), carrying the responder's wall-clock at sample time as epoch milliseconds (append-only — never reorder existing field ordinals). Add the matching field to the hand-authored `xce_ctrlstatus_t` typedef in `src/include/typemap.h` and verify it field-for-field against the regenerated codec (per the 1.1/2.1 learning: `$C.mapname` types are NOT auto-generated here). diff --git a/.kiro/specs/ha-commit-engine/tasks.md b/.kiro/specs/ha-commit-engine/tasks.md index 12a2e13..f5ec28d 100644 --- a/.kiro/specs/ha-commit-engine/tasks.md +++ b/.kiro/specs/ha-commit-engine/tasks.md @@ -3,7 +3,7 @@ > TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. All new tests live in `testsuite/test_commit_engine.c` (brief-scoped single test file) plus targeted updates to `test_repl_majority.c` — this shared-file reality is why no `(P)` markers are applied: the test file and `repl.c` are touched by multiple groups, so the plan is sequential. > Verify exact line numbers with grep before editing (research.md table has the verified anchors; files drift). -- [ ] 1. Commit engine core: per-LSN evaluation and commit-index integrity +- [x] 1. Commit engine core: per-LSN evaluation and commit-index integrity - [x] 1.1 Write RED tests for per-LSN majority evaluation and commit-index integrity - Create `testsuite/test_commit_engine.c` against a stub `persist_t` + `replicator_t` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (meson.build); create `src/controller/include/commit.h` with the umbrella-verbatim interface (types + prototypes only) so tests compile - Tests: with n_peers=3, a standby ACK for LSN M does not commit entry N>M (kills `acked_lsn > 0` counting); ACK watermark >= N from one standby plus self commits N; cumulative/out-of-order watermarks advance `committed_lsn` to the majority median, never past the highest appended LSN, never backwards; one far-lagging standby does not block majority commit; RETAIN mode returns 0 immediately without blocking @@ -17,7 +17,7 @@ - Observable completion: all task 1.1 tests PASS; engine is the only `committed_lsn` writer in `commit.c` paths - _Requirements: 1.2, 2.1, 2.2, 2.3, 5.1_ -- [ ] 2. WAIT mode, self-quorum, zero-peer rule, admission cap, own-term gate storage +- [x] 2. WAIT mode, self-quorum, zero-peer rule, admission cap, own-term gate storage - [x] 2.1 Write RED tests for blocking-wait semantics and admission control - Tests in `test_commit_engine.c`: WAIT blocks until majority ACK then returns 0; WAIT returns `-ETIMEDOUT` at deadline with `committed_lsn` unmoved past the entry; n_peers=1 → WAIT returns 0 immediately and `committed_lsn` advances to the entry; n_peers=3 with zero standby ACKs → `-ETIMEDOUT`, never success (short-circuit regression); with n_vcpus=8 and reserved=2, a 7th concurrent WAIT gets `-XCE_ERR_NOT_READY` while the 6 admitted waits proceed; cap floor `max(1, n_vcpus - reserved)` holds for tiny n_vcpus; while the cap is saturated, `commit_on_ack` still advances `committed_lsn` and completes admitted waiters; `writable_lsn` unset (0) → no gating; `commit_set_writable_lsn(G)` with G > entry → waiter completes only once `committed_lsn >= G` - Observable completion: new cases FAIL (red) against the task-1 engine @@ -29,7 +29,7 @@ - Observable completion: all task 2.1 tests PASS; cap is expressed only relative to the `n_vcpus` init argument (no absolute constant other than the reserved headroom) - _Requirements: 1.1, 4.3, 4.4, 6.3, 7.1, 7.2, 7.3_ -- [ ] 3. Membership-change re-evaluation +- [x] 3. Membership-change re-evaluation - [x] 3.1 Write RED tests for membership re-evaluation (completed jointly with 3.2 — RED unachievable, see Implementation Notes) - Tests in `test_commit_engine.c`: a catch-up standby's ACK watermark does not count toward a pending entry; after clearing `in_catch_up` and calling `commit_on_membership_change`, the pending waiter completes if the standby's watermark covers it; on standby removal that makes majority unreachable, a pending WAIT never returns success and fails with `-ETIMEDOUT` at its deadline; `commit_on_membership_change` returns without blocking - Observable completion: new cases FAIL (red) @@ -40,7 +40,7 @@ - Observable completion: all task 3.1 tests PASS - _Requirements: 4.1, 4.2_ -- [ ] 4. Quorum-accounting fix: catch-up exclusion at the dispatch gate +- [x] 4. Quorum-accounting fix: catch-up exclusion at the dispatch gate - [x] 4.1 Write RED tests for write-admission quorum counting - Tests: `repl_active_standby_count` counts only slots with `ep != NULL && !in_catch_up`; `ha_has_write_quorum` with n_peers=3 returns false when the only connected standby is in catch-up, true with one active standby, false with zero connected; after simulated reconnect (registry restored) the gate accepts again with no reset call (stateless re-evaluation); n_peers=1 always true - Declare `repl_active_standby_count` in src/include/repl.h so tests compile @@ -52,7 +52,7 @@ - Observable completion: all task 4.1 tests PASS; existing dispatch-gate tests still green - _Requirements: 3.1, 3.2, 3.3, 6.1, 6.2_ -- [ ] 5. Integration: wire the engine behind the job-submit sync path +- [x] 5. Integration: wire the engine behind the job-submit sync path - [x] 5.1 Write RED integration tests for the ACK feed and the sync-append path - Tests in `test_commit_engine.c`: `repl_on_ack` with a wired engine updates `acked_lsn` AND triggers commit evaluation (majority → `committed_lsn` advances, waiter wakes); `repl_start_catchup`/`repl_complete_catchup`/`repl_remove_standby`/`repl_add_standby` trigger re-evaluation; `persist_wal_sync_append` with wired engine returns 0 only on per-entry majority, returns `-ETIMEDOUT` when standbys' `acked_lsn` covers only an EARLIER entry (stale-ACK regression — connectivity/old ACKs never substitute), leaves `committed_lsn` unmoved on timeout, and for n_peers=1 returns 0 with `committed_lsn` advanced (replaces the old `-ENODEV` degraded path) - Update `testsuite/test_repl_majority.c` walAck cases to wire an engine and assert per-LSN semantics; configAck cases byte-identical @@ -72,7 +72,7 @@ - Observable completion: all task 5.1 tests PASS, including stale-ACK regression and single-node `committed_lsn` advancement; job submit on a wired cluster confirms only on per-entry majority - _Requirements: 1.1, 1.2, 1.3, 2.1, 4.4, 6.3_ -- [ ] 6. Validation: full regression and dishonest-path audit +- [x] 6. Validation: full regression and dishonest-path audit - [x] 6.1 Run the full controller regression and audit the legacy paths - Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix any regression introduced by the short-circuit removal or the `committed_lsn` ownership move (existing persist/repl/ha suites) - Audit: grep confirms no production (engine-wired) walAck path reaches the `acked_lsn > 0` counting or the `n_active == 0 → success` branch; both survive only under the deprecated `ce == NULL` test fallback diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md b/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md index 0fbcddd..f9486d1 100644 --- a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md +++ b/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md @@ -3,7 +3,7 @@ > TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor. > The work runs along a single request path (atomic migration → libxce sender → primary stamp / standby consume → vote-floor verification → E2E), so tasks are sequential by design; no `(P)` markers. The two foundation tasks (1.1 atomic migration, 1.2 sender) touch independent files but are kept ordered for review simplicity. The HA-critical change is the standby consume + term-guard (task 2.2) and its vote-floor interaction (task 3.2). -- [ ] 1. Foundation: atomic state + heartbeat sender +- [x] 1. Foundation: atomic state + heartbeat sender - [x] 1.1 Migrate `last_known_committed_config_tag` to `_Atomic` - Change `ha->last_known_committed_config_tag` from a plain `uint64_t` to `_Atomic uint64_t` (`src/controller/include/ha.h`) and update its existing writers — `cluster_apply_ctrlhello_reply` (`ha_convergence.c`) and the election safe-commit path (`ha.c`) — to `atomic_store_explicit(..., release)`, and any reader (incl. the controller-status `ctrlstatus_fill`) to `atomic_load_explicit(..., acquire)`. Pure type/access migration — no behavior change. - Observable: the controller suite builds clean and passes unchanged; every read/write of the field uses atomic accessors (no plain access remains). @@ -16,7 +16,7 @@ - _Requirements: 1.1_ - _Boundary: libxce sender_ -- [ ] 2. Standby consume (reconcile-free, term-guarded, monotonic) +- [x] 2. Standby consume (reconcile-free, term-guarded, monotonic) - [x] 2.1 Add the reconcile-free monotonic updater `ha_observe_heartbeat_committed` - Add `ha_observe_heartbeat_committed(cluster_t* c, uint64_t committed_lsn, uint64_t committed_config_tag)` (`ha.c`, declared in `ha.h`): advance `last_known_committed_lsn` only when the carried LSN is strictly greater (monotonic; atomic), advancing `last_known_committed_config_tag` together with it; no-op when `<=` current or zero. It must NOT call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state. - TDD: unit tests assert advance-on-higher (LSN + config tag updated), no-op on equal/lower/zero, config tag never regresses independently, and — critically — that the call performs no WAL truncation / no reconcile and mutates nothing beyond `last_known_committed_*`. @@ -32,7 +32,7 @@ - _Boundary: cluster.c OP_nop handler_ - _Depends: 2.1_ -- [ ] 3. Primary produce + safety verification +- [x] 3. Primary produce + safety verification - [x] 3.1 Stamp committed + term on the primary heartbeat - In `primary_heartbeat_fn` (`ha.c`), replace the per-standby `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, atomic_load(ha->term), atomic_load(persist.committed_lsn), atomic_load(ha->committed_config_tag))`. Preserve the `XCE_TESTING` `ha_nop_req_hook` seam and the existing heartbeat send-failure `WARN` and `last_push_time` bookkeeping. - TDD/observable: a test (or the `ha_nop_req_hook` seam) confirms the primary's heartbeat now carries its current committed LSN / committed config tag / term; the send-failure and liveness-eviction behavior of `primary_heartbeat_fn` is unchanged. diff --git a/.kiro/specs/ha-prevote/tasks.md b/.kiro/specs/ha-prevote/tasks.md index d144249..fa4f978 100644 --- a/.kiro/specs/ha-prevote/tasks.md +++ b/.kiro/specs/ha-prevote/tasks.md @@ -2,7 +2,7 @@ > TDD is mandatory (project rule): for each implementation sub-task, add the failing test in the relevant `testsuite/` first (RED), then implement to green. Dependency direction: capnp/types → raft library → controller adapter → dispatch/transport. The real-vote state machine is reused unchanged after PreVote success. -- [ ] 1. Foundation: wire schema and raft-library PreVote scaffolding +- [x] 1. Foundation: wire schema and raft-library PreVote scaffolding - [x] 1.1 Add PreVote opcodes to the election wire protocol - Add two opcodes for the PreVote request and PreVote reply, reusing the existing vote request/reply payload structures and their message union arms (no new payload structs). - Regenerate the generated bindings and confirm the build produces the new opcode constants while existing vote opcode ordinals are unchanged. @@ -15,7 +15,7 @@ - _Requirements: 1.2, 6.2_ - _Boundary: raft pre-vote functions_ -- [ ] 2. Core: raft-library pure PreVote decision and tally (test-first, no side effects) +- [x] 2. Core: raft-library pure PreVote decision and tally (test-first, no side effects) - [x] 2.1 Implement the voter-side PreVote decision - RED first: add raft unit tests asserting grant for a fresh, higher-or-equal-term candidate when the eligibility callback allows it, and deny on stale term or when the eligibility callback rejects. - Reuse the existing vote-eligibility callback (primary-deny, fresh-heartbeat stickiness, catch-up deny) and let the dispatcher apply freshness beforehand, exactly as the real-vote path does. @@ -38,7 +38,7 @@ - _Boundary: raft pre-vote functions_ - _Depends: 1.2_ -- [ ] 3. Core: controller voter-side PreVote dispatch (test-first) +- [x] 3. Core: controller voter-side PreVote dispatch (test-first) - [x] 3.1 Answer incoming PreVote requests with stickiness + freshness, no mutation - RED first: extend the vote-gate tests so a PreVote from a fresh candidate is denied while a leader is believed alive (primary role, or standby with fresh heartbeat, or catch-up) and granted once stickiness lapses and freshness passes. - Compute the freshness floor and run the eligibility gate before the raft decision, reusing the real-vote dispatch shape; reply with the voter's unchanged term plus the grant bit and freshness fields; log every denial with candidate id and reason. @@ -47,7 +47,7 @@ - _Boundary: PreVote dispatch_ - _Depends: 1.1, 2.1_ -- [ ] 4. Core: controller candidate-side orchestration and trigger restructuring (test-first) +- [x] 4. Core: controller candidate-side orchestration and trigger restructuring (test-first) - [x] 4.1 (P) Run the PreVote round and branch on its outcome - RED first: add controller tests driving the reply handler through the existing vote-reply hook (no real transport): majority grant → real election started; no majority → node stays standby with term unchanged and the liveness watchdog re-armed; round timeout treated as a loss. - Owns the candidate orchestration in the controller: start the round (no persist, no role change), arm a round timeout bounded by the existing election-probe deadline, and the PreVote-reply handler that feeds the raft tally and branches — on a win perform the existing truncate + electing-transition + start-real-election sequence (unchanged real-election path); on loss log campaign abandonment. The send path emits the PreVote request opcode from task 1.1; transport-side reply routing is wired separately in task 5.1. @@ -64,7 +64,7 @@ - _Boundary: PreVote orchestration (trigger path)_ - _Depends: 4.1_ -- [ ] 5. Integration: transport wiring and end-to-end scenarios +- [x] 5. Integration: transport wiring and end-to-end scenarios - [x] 5.1 Wire the PreVote reply opcode case to the candidate handler - Add the transport dispatch case for the incoming PreVote-reply opcode so it routes to the reply handler built in 4.1 (replacing the test hook in production), paralleling the real-vote reply dispatch case. - Observable: a PreVote reply arriving over the real transport reaches the candidate's tally and can drive the round to a decision end-to-end (not just via the test hook). @@ -84,7 +84,7 @@ - _Requirements: 5.1, 5.4, 4.2_ - _Depends: 4.1, 4.2, 5.1_ -- [ ] 6. Validation: safety invariants and regression guard +- [x] 6. Validation: safety invariants and regression guard - [x] 6.1 Assert Raft safety invariants hold with PreVote - Add raft-library invariant assertions that the PreVote functions mutate no persistent state (Election Safety and term monotonicity preserved), and record the TLA+/verification-checklist consistency argument (the real-vote path is unchanged, so the higher-term-adopt failover and the bug-339/bug-340 fixes are preserved). - Observable: invariant tests pass and the existing failover/rejoin regression tests remain green with PreVote enabled. diff --git a/.kiro/specs/ha-single-leader-role/tasks.md b/.kiro/specs/ha-single-leader-role/tasks.md index aa2cf38..fd2f768 100644 --- a/.kiro/specs/ha-single-leader-role/tasks.md +++ b/.kiro/specs/ha-single-leader-role/tasks.md @@ -4,7 +4,7 @@ > **Hard upstream prerequisite**: ha-commit-engine AND ha-commit-barrier-rollout must be implemented and green first — the promotion path calls `commit_set_writable_lsn` on the engine in `cluster_t.commit`, and the universal barrier + dispatch-gate fact allowlist must be in force before the transitional state is removed (umbrella hard ordering). > Verify exact line numbers with grep before editing — research.md §3 has the 2026-06-11 verified anchor table, but barrier-rollout landing first WILL shift ha.c line numbers. -- [ ] 1. WAL_NOOP entry type (additive foundation) +- [x] 1. WAL_NOOP entry type (additive foundation) - [x] 1.1 Write RED tests for the no-op entry - Create `testsuite/test_ha_role_lifecycle.c` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (src/controller/meson.build:74-131); first cases cover the entry type only - Tests: `persist_wal_async(p, WAL_NOOP, NULL, 0)` appends a header-only entry (payload_len 0) and broadcasts it (stub replicator observes the entry); `apply_wal_entry` on a WAL_NOOP advances `current_lsn` past the entry's LSN and touches no entity state (jobmgr/queue/worker untouched); boot `replay_wal` over a segment containing WAL_NOOP replays cleanly with the same no-effect guarantee; `wal_classify(WAL_NOOP, 0) == WAL_MODE_RETAIN` (keeps barrier-rollout's totality test green) @@ -17,7 +17,7 @@ - Observable completion: all task 1.1 tests PASS; barrier-rollout's classification totality test still PASSES - _Requirements: 2.1_ -- [ ] 2. Promotion rewrite: direct PRIMARY + own-term gate +- [x] 2. Promotion rewrite: direct PRIMARY + own-term gate - [x] 2.1 Write RED lifecycle tests for direct promotion and the own-term gate - Tests in `test_ha_role_lifecycle.c`: driving the become-leader path yields `ha->role == CTRL_PRIMARY` directly (no intermediate value observable; `ha_is_primary()` true immediately); `ha_should_redirect()` returns 0 the instant promotion returns, before any standby ACK; persistence hooks are registered by promotion (hooksys observes the three persist hook fns; `hooks_registered == 1`); promotion arms NO delaymgr watchdog (delaymgr task-count seam unchanged across promotion); with a real engine (n_peers=3, stub registry): promotion appends WAL_NOOP at lsn=N, registers it RETAIN, and `commit_set_writable_lsn(N)` is in effect — a `commit_submit(WAIT)` for entry N+1 does NOT complete while ACKs cover N+1 but not N, and completes once `committed_lsn >= N`; with the gate unmet a WAIT submission resolves only as barrier success, `-ETIMEDOUT`, or `-XCE_ERR_NOT_READY` — never via any promotion-side path; n_peers=1: gate opens immediately at promotion (noop self-committed), WAIT returns 0 as before - Observable completion: new cases FAIL (red) against the current COMMITTING-entering `raft_cb_become_leader` @@ -30,14 +30,14 @@ - Observable completion: all task 2.1 tests PASS; promotion never blocks (returns without waiting for the noop to commit) - _Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3_ -- [ ] 3. Legacy XCE_TESTING election path symmetry +- [x] 3. Legacy XCE_TESTING election path symmetry - [x] 3.1 Update ha_elect_evaluate_and_signal and its direct tests (RED → GREEN in one pass) - Retarget existing legacy-path expectations first (test_ha_start_election_async.c, the evaluation cases in test_ha_role.c): quorum win → role `CTRL_PRIMARY` directly, no watchdog armed — run to confirm RED against current code - In the CTRL_PRIMARY branch of `ha_elect_evaluate_and_signal` (ha.c:2247-2300 anchor): replace the COMMITTING store (2248-2252) with the direct `CTRL_PRIMARY` store, delete the `repl_ack_count` reset (2255) and the watchdog-arm block (2275-2299); keep the config-push/ctrlhello XCE_TESTING hooks; route promotion side effects through the same shared sequence (noop+gate tail guarded on persist/engine availability, as in production) - Observable completion: retargeted legacy-path tests PASS; lifecycle test for symmetry PASSES (both entry points produce identical role/hook/heartbeat/no-watchdog state) - _Requirements: 1.1, 1.5_ -- [ ] 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates) +- [x] 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates) - [x] 4.1 Delete the force-promotion watchdog and ack/convergence machinery - Delete from ha.c: `committing_timeout_fn` (3080-3121), `committing_timeout_fn_for_test` (2950-2990), `committing_timeout_ctx_t` type + fwd decl (64-69, 83), the cancel block in `raft_cb_become_follower` (640-648); `ha_check_majority_convergence` (1295-1353) and its heartbeat-reply call site (2710-2715); `ha_repl_ack_received` (3123-3141) — its `OPCode_replCatchUpDone` branch in `repl_ack_decode_and_dispatch` (2659-2661) becomes an INFO-logged no-op (opcode stays gate-allowed; `repl_complete_catchup` untouched); `repl_ack_count` init at ha.c:835 - Delete declarations from ha.h: `ha_check_majority_convergence` (396-412), `ha_repl_ack_received` (626-630) @@ -52,7 +52,7 @@ - Observable completion: barrier-rollout's gate tests (PRIMARY fact allowlist under no-quorum) still PASS; vote-gate tests PASS with PRIMARY-only denial; suite green - _Requirements: 1.2, 1.3, 1.4_ -- [ ] 5. Test-fixture sweep and final symbol removal +- [x] 5. Test-fixture sweep and final symbol removal - [x] 5.1 Retarget the remaining COMMITTING-referencing fixtures - Update expectations COMMITTING→PRIMARY (census: research.md §3): `test_ha_role.c` (37 refs), `test_ha_rdhup_integration.c` (20), `test_ha_repl_push.c` (14), `test_ha_startup.c` (12), `test_ha_connection_reuse.c` (11), `test_ha_start_election_async.c` (7), `test_ha_vote_gate.c` (6, residual from 4.2); delete the 11 `cfg.committing_timeout_ms = 10000;` initializers in `test_config_repl.c` - Each updated test asserts the new contract (direct PRIMARY, no transitional observations), not stubbed compatibility @@ -65,7 +65,7 @@ - Observable completion: clean build with zero warnings about the removed symbols; full controller testsuite + ASAN green via `/xce-build controller` - _Requirements: 1.2_ -- [ ] 6. Documentation sync and grep verification +- [x] 6. Documentation sync and grep verification - [x] 6.1 Synchronize steering and contributor docs to the three-state model - `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING) - `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob) diff --git a/.kiro/specs/ha-wal-replication/tasks.md b/.kiro/specs/ha-wal-replication/tasks.md index 07dde2a..d1bcf45 100644 --- a/.kiro/specs/ha-wal-replication/tasks.md +++ b/.kiro/specs/ha-wal-replication/tasks.md @@ -6,7 +6,7 @@ --- -- [ ] 1. repl_broadcast TCP send +- [x] 1. repl_broadcast TCP send - [x] 1.1 Fill in repl_broadcast send path - Remove `(void)payload; (void)payload_len;` and the `pending_ack_count++` counter. - For each non-catch-up standby (`s->ep != NULL, !in_catch_up`), build a single-entry `wal_catchup_entry_t` with `lsn=hdr->lsn`, `entry_type=hdr->type`, `raw_payload=payload`, `raw_payload_len=payload_len`. @@ -25,7 +25,7 @@ --- -- [ ] 2. HA→replicator standby registration bridge +- [x] 2. HA→replicator standby registration bridge - [x] 2.1 Register standbys on connect - In `raft_cb_become_leader` (ha.c), after promoting vote eps to `ha->standby_endpoints`: for each promoted ep, call `repl_add_standby(&c->persist.repl, ep, peer_host, peer_port)`. - In `on_standby_connected` (ha.c), after ctrlhello is sent: call `repl_add_standby` with the late-joiner's endpoint, host, and port. @@ -49,7 +49,7 @@ --- -- [ ] 3. Standby WAL ACK reply +- [x] 3. Standby WAL ACK reply - [x] 3.1 Send WalAckReply after WAL entry applied - In `ha_apply_wal_catchup_entry` (ha.c), after `ha_apply_single_entry` succeeds for an entry: construct a `WalAckReply { term=ha->term, lsn=ent->lsn, success=1 }` and send via `xce_reply` to the endpoint that delivered the entry. - The endpoint is not directly available in `ha_apply_wal_catchup_entry` — it's called from `cluster.c:handle_wal_catchup_entry` which has the ep. Pass `ep` as a new parameter: `ha_apply_wal_catchup_entry(ha, p, entry, ep)`. @@ -70,7 +70,7 @@ --- -- [ ] 4. Primary ACK handler → advance committed_lsn +- [x] 4. Primary ACK handler → advance committed_lsn - [x] 4.1 Handle incoming WalAckReply on primary - In `repl_ack_decode_and_dispatch` (ha.c:2506), add case for `RequestReply_data_walackreplymsg`. - Extract `lsn` and `success` from `WalAckReply`. If `success`, call `repl_on_ack(&c->persist.repl, ep, lsn)`. diff --git a/.kiro/specs/libxce-failover/tasks.md b/.kiro/specs/libxce-failover/tasks.md index 8d5dfa3..3454d0e 100644 --- a/.kiro/specs/libxce-failover/tasks.md +++ b/.kiro/specs/libxce-failover/tasks.md @@ -1,6 +1,6 @@ # Implementation Plan -- [ ] 1. Foundation: failover state, captured-reply fields, and cleanup +- [x] 1. Foundation: failover state, captured-reply fields, and cleanup - [x] 1.1 Establish shared failover state and reply-capture fields - Expose the client context to a new client-internal header so the failover helper can operate on it - Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot @@ -14,7 +14,7 @@ - Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist - _Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4_ -- [ ] 2. Core: status classification, endpoint rotation, reply-path capture +- [x] 2. Core: status classification, endpoint rotation, reply-path capture - [x] 2.1 Implement status classification - Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal - Never classify redirect, standby, no-primary, or not-ready as success @@ -38,7 +38,7 @@ - _Boundary: RedirectCapture, ReplyWaitWake_ - _Depends: 1.1_ -- [ ] 3. Core: the retry/failover decision helper +- [x] 3. Core: the retry/failover decision helper - [x] 3.1 Implement the decision loop (classify, rotate, backoff, budget) - Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff - Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code @@ -55,7 +55,7 @@ - _Depends: 3.1_ - _Boundary: xcecli_retry_next_ -- [ ] 4. Core: reconnect/rejoin hook +- [x] 4. Core: reconnect/rejoin hook - [x] 4.1 Add the reconnect/rejoin hook mechanism - Provide a client API to register a reconnect callback and argument, stored in the failover state - Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result @@ -63,7 +63,7 @@ - _Requirements: 1.2_ - _Depends: 3.1, 3.2_ -- [ ] 5. Integration: wrap the client ops; controller redirect address +- [x] 5. Integration: wrap the client ops; controller redirect address - [x] 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock - Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration - Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index a1165c8..01d262c 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -6,7 +6,7 @@ > **Verify anchors before editing** (files drift): the design and `research.md` carry 2026-06-27 file:line anchors; re-grep before each edit. > **Root cause (task 1.3, DONE + verified):** the term-blind hypothesis is **refuted**. The real defect is **cross-term catch-up payload loss** — `ha_stream_wal_to_standby` silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Phase 2 targets the streamer + payload-aware guard (`research.md` §"Task 1.3"). A green synthetic unit test over a still-red soak does not satisfy 1.1/8.1 — closure requires the real-fs chaotic soak (7.2) clean. -- [ ] 1. Foundation: deterministic chaotic harness and live-divergence reproduction +- [x] 1. Foundation: deterministic chaotic harness and live-divergence reproduction - [x] 1.1 Extend the divergence detector to a machine-checkable per-member report - Extend `walcompare` to emit a parseable per-member `(lsn -> term, payload-crc, present/MISSING)` report over the committed prefix, and classify each divergence as committed vs uncommitted - Observable completion: `walcompare` prints a parseable per-member report and a committed-vs-uncommitted classification; the existing soak still detects the round-3 LSN divergence through it @@ -38,7 +38,7 @@ - _Boundary: Acceptance Soak Gate, Deterministic Chaotic Harness_ - _Depends: 1.2_ -- [ ] 2. Committed-prefix divergence fix: cross-term catch-up payload integrity + payload-aware guard +- [x] 2. Committed-prefix divergence fix: cross-term catch-up payload integrity + payload-aware guard - _Root cause (task 1.3, verified): the cross-term catch-up streamer silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Term and ingest-path are correct — the payload is lost._ - [x] 2.1 RED: payload-aware persistence guard tests - Add cases: an empty-payload (`PLEN=0`) entry written at an existing LSN N followed by the correct full-payload entry must end with the FULL payload (today the `(term,lsn)`-only idempotency guard silently no-ops the corrective write); the reverse (full then empty at the same `(term,lsn)`) must NOT degrade the stored entry to empty; an identical `(term, payload-crc)` re-write is a safe idempotent no-op @@ -66,7 +66,7 @@ - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - _Depends: 2.2, 2.3_ -- [ ] 3. Snapshot-boundary reconciliation +- [x] 3. Snapshot-boundary reconciliation - [x] 3.1 RED: snapshot post-install reconcile test - Add a case where a divergent pre-snapshot entry must be reconciled against the installed checkpoint's `(term, lsn)` boundary (today snapshot install sets applied-lsn without a `(term, lsn)` reconcile) - Observable completion: the case FAILS on the baseline because no `(term, lsn)` reconcile occurs at the snapshot boundary @@ -80,7 +80,7 @@ - _Boundary: Snapshot Boundary Reconcile_ - _Depends: 2.4_ -- [ ] 4. Standby reconvergence completeness +- [x] 4. Standby reconvergence completeness - _Root cause (Phase-4/5 investigation, `research.md`): a recovered standby holding a divergent uncommitted tail above the new leader's committed point (former-primary shape, e.g. ctrl-2 with 12-15 at an old term while leader committed 11) never reaches `ae_already_have` — it loops on `ae_reject_logmismatch`, its control channel flaps, and it never rejoins. A short-suffix standby (ctrl-3) DOES reconverge, so the backtrack works for small divergence but not the former-primary tail._ - [x] 4.1 RED: divergent-tail reconvergence + apply-gap gate tests - Primary case: construct a recovered standby with a MULTI-entry divergent uncommitted tail above the leader's committed point and assert it reconverges to follower — truncates the conflicting suffix and reaches `ae_already_have` (no permanent `ae_reject_logmismatch` loop). Secondary case: a rejoining node with an injected catch-up apply gap must not be reported converged/healthy until `applied_lsn` and `log_len` equal the leader's committed boundary @@ -95,7 +95,7 @@ - _Boundary: Standby Reconvergence_ - _Depends: 2.4_ -- [ ] 5. Election liveness and safety under churn +- [x] 5. Election liveness and safety under churn - _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._ - [x] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests - Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`) @@ -182,7 +182,7 @@ - _Boundary: catch-up backfill + election-liveness tail (5.7 family)_ - _Depends: 5.9_ -- [ ] 6. Integration: single-seam invariant end-to-end +- [x] 6. Integration: single-seam invariant end-to-end - [x] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path - Observable completion: an end-to-end test (or instrumented soak round) shows leader and standby hold identical `(term, payload-crc)` for every replicated and cross-term-caught-up LSN; `walcompare --report` clean across members @@ -196,7 +196,7 @@ - _Requirements: 1.1, 1.2, 2.4, 3.1_ - _Depends: 2.4, 3.2, 4.2_ -- [ ] 7. Validation: regression and acceptance soak gate +- [x] 7. Validation: regression and acceptance soak gate - [x] 7.1 Full controller suite under ASAN with the new harness green - Build and run the full controller unit suite under ASAN (suite-by-suite if the aggregate hits the known 300s cap); fix any regression from the ingest/persist/election changes - Observable completion: the full suite is green under ASAN and every `test_ha_core_safety` case passes with no ASAN errors