forked from lei.chen/myxce
docs: fix unchecked group-header checkboxes across 8 fully-implemented specs
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation, ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and raft-ha-core-safety each had every numbered sub-task marked [x] but their group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real unchecked sub-tasks in any of the eight files. No task content changed.
This commit is contained in:
parent
78fe058242
commit
9f530f5742
8 changed files with 43 additions and 43 deletions
|
|
@ -3,14 +3,14 @@
|
|||
> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
|
||||
> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).
|
||||
|
||||
- [ ] 1. Foundation: status reply schema + codegen
|
||||
- [x] 1. Foundation: status reply schema + codegen
|
||||
- [x] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema
|
||||
- Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
|
||||
- Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
|
||||
- Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
|
||||
- _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_
|
||||
|
||||
- [ ] 2. Controller: status handler + any-role dispatch
|
||||
- [x] 2. Controller: status handler + any-role dispatch
|
||||
- [x] 2.1 Implement the controller status handler
|
||||
- Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
|
||||
- Route the new opcode to this handler in the controller request dispatch switch.
|
||||
|
|
@ -26,7 +26,7 @@
|
|||
- _Boundary: ha_dispatch_gate_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 3. libxce client API
|
||||
- [x] 3. libxce client API
|
||||
- [x] 3.1 Add the failover-aware any-node status call
|
||||
- Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
|
||||
- TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
|
||||
|
|
@ -42,7 +42,7 @@
|
|||
- _Boundary: libxce client_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 4. Test client command + wait-primary integration
|
||||
- [x] 4. Test client command + wait-primary integration
|
||||
- [x] 4.1 Add the `ctrlstatus` client command
|
||||
- Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
|
||||
- Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
|
||||
|
|
@ -56,14 +56,14 @@
|
|||
- _Boundary: cmd_wait_primary_
|
||||
- _Depends: 3.2_
|
||||
|
||||
- [ ] 5. Validation (live cluster)
|
||||
- [x] 5. Validation (live cluster)
|
||||
- [x] 5.1 End-to-end status + authoritative wait-primary on a live cluster
|
||||
- Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
|
||||
- Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
|
||||
- _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_
|
||||
- _Depends: 4.1, 4.2_
|
||||
|
||||
- [ ] 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11)
|
||||
- [x] 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11)
|
||||
> Sequential, single request path (schema → typemap → `ctrlstatus_fill` → render → E2E); no `(P)`. Read-only reporting only — the live-fresh-committed heartbeat change is OUT of boundary (route to ha-heartbeat-liveness / ha-repl-hardening).
|
||||
- [x] 6.1 Add the responder snapshot timestamp field to the schema + hand-authored typemap
|
||||
- Append a responder snapshot-time field to the status payload at the next free ordinal (after the peer count), carrying the responder's wall-clock at sample time as epoch milliseconds (append-only — never reorder existing field ordinals). Add the matching field to the hand-authored `xce_ctrlstatus_t` typedef in `src/include/typemap.h` and verify it field-for-field against the regenerated codec (per the 1.1/2.1 learning: `$C.mapname` types are NOT auto-generated here).
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. All new tests live in `testsuite/test_commit_engine.c` (brief-scoped single test file) plus targeted updates to `test_repl_majority.c` — this shared-file reality is why no `(P)` markers are applied: the test file and `repl.c` are touched by multiple groups, so the plan is sequential.
|
||||
> Verify exact line numbers with grep before editing (research.md table has the verified anchors; files drift).
|
||||
|
||||
- [ ] 1. Commit engine core: per-LSN evaluation and commit-index integrity
|
||||
- [x] 1. Commit engine core: per-LSN evaluation and commit-index integrity
|
||||
- [x] 1.1 Write RED tests for per-LSN majority evaluation and commit-index integrity
|
||||
- Create `testsuite/test_commit_engine.c` against a stub `persist_t` + `replicator_t` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (meson.build); create `src/controller/include/commit.h` with the umbrella-verbatim interface (types + prototypes only) so tests compile
|
||||
- Tests: with n_peers=3, a standby ACK for LSN M does not commit entry N>M (kills `acked_lsn > 0` counting); ACK watermark >= N from one standby plus self commits N; cumulative/out-of-order watermarks advance `committed_lsn` to the majority median, never past the highest appended LSN, never backwards; one far-lagging standby does not block majority commit; RETAIN mode returns 0 immediately without blocking
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
- Observable completion: all task 1.1 tests PASS; engine is the only `committed_lsn` writer in `commit.c` paths
|
||||
- _Requirements: 1.2, 2.1, 2.2, 2.3, 5.1_
|
||||
|
||||
- [ ] 2. WAIT mode, self-quorum, zero-peer rule, admission cap, own-term gate storage
|
||||
- [x] 2. WAIT mode, self-quorum, zero-peer rule, admission cap, own-term gate storage
|
||||
- [x] 2.1 Write RED tests for blocking-wait semantics and admission control
|
||||
- Tests in `test_commit_engine.c`: WAIT blocks until majority ACK then returns 0; WAIT returns `-ETIMEDOUT` at deadline with `committed_lsn` unmoved past the entry; n_peers=1 → WAIT returns 0 immediately and `committed_lsn` advances to the entry; n_peers=3 with zero standby ACKs → `-ETIMEDOUT`, never success (short-circuit regression); with n_vcpus=8 and reserved=2, a 7th concurrent WAIT gets `-XCE_ERR_NOT_READY` while the 6 admitted waits proceed; cap floor `max(1, n_vcpus - reserved)` holds for tiny n_vcpus; while the cap is saturated, `commit_on_ack` still advances `committed_lsn` and completes admitted waiters; `writable_lsn` unset (0) → no gating; `commit_set_writable_lsn(G)` with G > entry → waiter completes only once `committed_lsn >= G`
|
||||
- Observable completion: new cases FAIL (red) against the task-1 engine
|
||||
|
|
@ -29,7 +29,7 @@
|
|||
- Observable completion: all task 2.1 tests PASS; cap is expressed only relative to the `n_vcpus` init argument (no absolute constant other than the reserved headroom)
|
||||
- _Requirements: 1.1, 4.3, 4.4, 6.3, 7.1, 7.2, 7.3_
|
||||
|
||||
- [ ] 3. Membership-change re-evaluation
|
||||
- [x] 3. Membership-change re-evaluation
|
||||
- [x] 3.1 Write RED tests for membership re-evaluation (completed jointly with 3.2 — RED unachievable, see Implementation Notes)
|
||||
- Tests in `test_commit_engine.c`: a catch-up standby's ACK watermark does not count toward a pending entry; after clearing `in_catch_up` and calling `commit_on_membership_change`, the pending waiter completes if the standby's watermark covers it; on standby removal that makes majority unreachable, a pending WAIT never returns success and fails with `-ETIMEDOUT` at its deadline; `commit_on_membership_change` returns without blocking
|
||||
- Observable completion: new cases FAIL (red)
|
||||
|
|
@ -40,7 +40,7 @@
|
|||
- Observable completion: all task 3.1 tests PASS
|
||||
- _Requirements: 4.1, 4.2_
|
||||
|
||||
- [ ] 4. Quorum-accounting fix: catch-up exclusion at the dispatch gate
|
||||
- [x] 4. Quorum-accounting fix: catch-up exclusion at the dispatch gate
|
||||
- [x] 4.1 Write RED tests for write-admission quorum counting
|
||||
- Tests: `repl_active_standby_count` counts only slots with `ep != NULL && !in_catch_up`; `ha_has_write_quorum` with n_peers=3 returns false when the only connected standby is in catch-up, true with one active standby, false with zero connected; after simulated reconnect (registry restored) the gate accepts again with no reset call (stateless re-evaluation); n_peers=1 always true
|
||||
- Declare `repl_active_standby_count` in src/include/repl.h so tests compile
|
||||
|
|
@ -52,7 +52,7 @@
|
|||
- Observable completion: all task 4.1 tests PASS; existing dispatch-gate tests still green
|
||||
- _Requirements: 3.1, 3.2, 3.3, 6.1, 6.2_
|
||||
|
||||
- [ ] 5. Integration: wire the engine behind the job-submit sync path
|
||||
- [x] 5. Integration: wire the engine behind the job-submit sync path
|
||||
- [x] 5.1 Write RED integration tests for the ACK feed and the sync-append path
|
||||
- Tests in `test_commit_engine.c`: `repl_on_ack` with a wired engine updates `acked_lsn` AND triggers commit evaluation (majority → `committed_lsn` advances, waiter wakes); `repl_start_catchup`/`repl_complete_catchup`/`repl_remove_standby`/`repl_add_standby` trigger re-evaluation; `persist_wal_sync_append` with wired engine returns 0 only on per-entry majority, returns `-ETIMEDOUT` when standbys' `acked_lsn` covers only an EARLIER entry (stale-ACK regression — connectivity/old ACKs never substitute), leaves `committed_lsn` unmoved on timeout, and for n_peers=1 returns 0 with `committed_lsn` advanced (replaces the old `-ENODEV` degraded path)
|
||||
- Update `testsuite/test_repl_majority.c` walAck cases to wire an engine and assert per-LSN semantics; configAck cases byte-identical
|
||||
|
|
@ -72,7 +72,7 @@
|
|||
- Observable completion: all task 5.1 tests PASS, including stale-ACK regression and single-node `committed_lsn` advancement; job submit on a wired cluster confirms only on per-entry majority
|
||||
- _Requirements: 1.1, 1.2, 1.3, 2.1, 4.4, 6.3_
|
||||
|
||||
- [ ] 6. Validation: full regression and dishonest-path audit
|
||||
- [x] 6. Validation: full regression and dishonest-path audit
|
||||
- [x] 6.1 Run the full controller regression and audit the legacy paths
|
||||
- Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix any regression introduced by the short-circuit removal or the `committed_lsn` ownership move (existing persist/repl/ha suites)
|
||||
- Audit: grep confirms no production (engine-wired) walAck path reaches the `acked_lsn > 0` counting or the `n_active == 0 → success` branch; both survive only under the deprecated `ce == NULL` test fallback
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
|
||||
> The work runs along a single request path (atomic migration → libxce sender → primary stamp / standby consume → vote-floor verification → E2E), so tasks are sequential by design; no `(P)` markers. The two foundation tasks (1.1 atomic migration, 1.2 sender) touch independent files but are kept ordered for review simplicity. The HA-critical change is the standby consume + term-guard (task 2.2) and its vote-floor interaction (task 3.2).
|
||||
|
||||
- [ ] 1. Foundation: atomic state + heartbeat sender
|
||||
- [x] 1. Foundation: atomic state + heartbeat sender
|
||||
- [x] 1.1 Migrate `last_known_committed_config_tag` to `_Atomic`
|
||||
- Change `ha->last_known_committed_config_tag` from a plain `uint64_t` to `_Atomic uint64_t` (`src/controller/include/ha.h`) and update its existing writers — `cluster_apply_ctrlhello_reply` (`ha_convergence.c`) and the election safe-commit path (`ha.c`) — to `atomic_store_explicit(..., release)`, and any reader (incl. the controller-status `ctrlstatus_fill`) to `atomic_load_explicit(..., acquire)`. Pure type/access migration — no behavior change.
|
||||
- Observable: the controller suite builds clean and passes unchanged; every read/write of the field uses atomic accessors (no plain access remains).
|
||||
|
|
@ -16,7 +16,7 @@
|
|||
- _Requirements: 1.1_
|
||||
- _Boundary: libxce sender_
|
||||
|
||||
- [ ] 2. Standby consume (reconcile-free, term-guarded, monotonic)
|
||||
- [x] 2. Standby consume (reconcile-free, term-guarded, monotonic)
|
||||
- [x] 2.1 Add the reconcile-free monotonic updater `ha_observe_heartbeat_committed`
|
||||
- Add `ha_observe_heartbeat_committed(cluster_t* c, uint64_t committed_lsn, uint64_t committed_config_tag)` (`ha.c`, declared in `ha.h`): advance `last_known_committed_lsn` only when the carried LSN is strictly greater (monotonic; atomic), advancing `last_known_committed_config_tag` together with it; no-op when `<=` current or zero. It must NOT call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state.
|
||||
- TDD: unit tests assert advance-on-higher (LSN + config tag updated), no-op on equal/lower/zero, config tag never regresses independently, and — critically — that the call performs no WAL truncation / no reconcile and mutates nothing beyond `last_known_committed_*`.
|
||||
|
|
@ -32,7 +32,7 @@
|
|||
- _Boundary: cluster.c OP_nop handler_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 3. Primary produce + safety verification
|
||||
- [x] 3. Primary produce + safety verification
|
||||
- [x] 3.1 Stamp committed + term on the primary heartbeat
|
||||
- In `primary_heartbeat_fn` (`ha.c`), replace the per-standby `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, atomic_load(ha->term), atomic_load(persist.committed_lsn), atomic_load(ha->committed_config_tag))`. Preserve the `XCE_TESTING` `ha_nop_req_hook` seam and the existing heartbeat send-failure `WARN` and `last_push_time` bookkeeping.
|
||||
- TDD/observable: a test (or the `ha_nop_req_hook` seam) confirms the primary's heartbeat now carries its current committed LSN / committed config tag / term; the send-failure and liveness-eviction behavior of `primary_heartbeat_fn` is unchanged.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> TDD is mandatory (project rule): for each implementation sub-task, add the failing test in the relevant `testsuite/` first (RED), then implement to green. Dependency direction: capnp/types → raft library → controller adapter → dispatch/transport. The real-vote state machine is reused unchanged after PreVote success.
|
||||
|
||||
- [ ] 1. Foundation: wire schema and raft-library PreVote scaffolding
|
||||
- [x] 1. Foundation: wire schema and raft-library PreVote scaffolding
|
||||
- [x] 1.1 Add PreVote opcodes to the election wire protocol
|
||||
- Add two opcodes for the PreVote request and PreVote reply, reusing the existing vote request/reply payload structures and their message union arms (no new payload structs).
|
||||
- Regenerate the generated bindings and confirm the build produces the new opcode constants while existing vote opcode ordinals are unchanged.
|
||||
|
|
@ -15,7 +15,7 @@
|
|||
- _Requirements: 1.2, 6.2_
|
||||
- _Boundary: raft pre-vote functions_
|
||||
|
||||
- [ ] 2. Core: raft-library pure PreVote decision and tally (test-first, no side effects)
|
||||
- [x] 2. Core: raft-library pure PreVote decision and tally (test-first, no side effects)
|
||||
- [x] 2.1 Implement the voter-side PreVote decision
|
||||
- RED first: add raft unit tests asserting grant for a fresh, higher-or-equal-term candidate when the eligibility callback allows it, and deny on stale term or when the eligibility callback rejects.
|
||||
- Reuse the existing vote-eligibility callback (primary-deny, fresh-heartbeat stickiness, catch-up deny) and let the dispatcher apply freshness beforehand, exactly as the real-vote path does.
|
||||
|
|
@ -38,7 +38,7 @@
|
|||
- _Boundary: raft pre-vote functions_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [ ] 3. Core: controller voter-side PreVote dispatch (test-first)
|
||||
- [x] 3. Core: controller voter-side PreVote dispatch (test-first)
|
||||
- [x] 3.1 Answer incoming PreVote requests with stickiness + freshness, no mutation
|
||||
- RED first: extend the vote-gate tests so a PreVote from a fresh candidate is denied while a leader is believed alive (primary role, or standby with fresh heartbeat, or catch-up) and granted once stickiness lapses and freshness passes.
|
||||
- Compute the freshness floor and run the eligibility gate before the raft decision, reusing the real-vote dispatch shape; reply with the voter's unchanged term plus the grant bit and freshness fields; log every denial with candidate id and reason.
|
||||
|
|
@ -47,7 +47,7 @@
|
|||
- _Boundary: PreVote dispatch_
|
||||
- _Depends: 1.1, 2.1_
|
||||
|
||||
- [ ] 4. Core: controller candidate-side orchestration and trigger restructuring (test-first)
|
||||
- [x] 4. Core: controller candidate-side orchestration and trigger restructuring (test-first)
|
||||
- [x] 4.1 (P) Run the PreVote round and branch on its outcome
|
||||
- RED first: add controller tests driving the reply handler through the existing vote-reply hook (no real transport): majority grant → real election started; no majority → node stays standby with term unchanged and the liveness watchdog re-armed; round timeout treated as a loss.
|
||||
- Owns the candidate orchestration in the controller: start the round (no persist, no role change), arm a round timeout bounded by the existing election-probe deadline, and the PreVote-reply handler that feeds the raft tally and branches — on a win perform the existing truncate + electing-transition + start-real-election sequence (unchanged real-election path); on loss log campaign abandonment. The send path emits the PreVote request opcode from task 1.1; transport-side reply routing is wired separately in task 5.1.
|
||||
|
|
@ -64,7 +64,7 @@
|
|||
- _Boundary: PreVote orchestration (trigger path)_
|
||||
- _Depends: 4.1_
|
||||
|
||||
- [ ] 5. Integration: transport wiring and end-to-end scenarios
|
||||
- [x] 5. Integration: transport wiring and end-to-end scenarios
|
||||
- [x] 5.1 Wire the PreVote reply opcode case to the candidate handler
|
||||
- Add the transport dispatch case for the incoming PreVote-reply opcode so it routes to the reply handler built in 4.1 (replacing the test hook in production), paralleling the real-vote reply dispatch case.
|
||||
- Observable: a PreVote reply arriving over the real transport reaches the candidate's tally and can drive the round to a decision end-to-end (not just via the test hook).
|
||||
|
|
@ -84,7 +84,7 @@
|
|||
- _Requirements: 5.1, 5.4, 4.2_
|
||||
- _Depends: 4.1, 4.2, 5.1_
|
||||
|
||||
- [ ] 6. Validation: safety invariants and regression guard
|
||||
- [x] 6. Validation: safety invariants and regression guard
|
||||
- [x] 6.1 Assert Raft safety invariants hold with PreVote
|
||||
- Add raft-library invariant assertions that the PreVote functions mutate no persistent state (Election Safety and term monotonicity preserved), and record the TLA+/verification-checklist consistency argument (the real-vote path is unchanged, so the higher-term-adopt failover and the bug-339/bug-340 fixes are preserved).
|
||||
- Observable: invariant tests pass and the existing failover/rejoin regression tests remain green with PreVote enabled.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
> **Hard upstream prerequisite**: ha-commit-engine AND ha-commit-barrier-rollout must be implemented and green first — the promotion path calls `commit_set_writable_lsn` on the engine in `cluster_t.commit`, and the universal barrier + dispatch-gate fact allowlist must be in force before the transitional state is removed (umbrella hard ordering).
|
||||
> Verify exact line numbers with grep before editing — research.md §3 has the 2026-06-11 verified anchor table, but barrier-rollout landing first WILL shift ha.c line numbers.
|
||||
|
||||
- [ ] 1. WAL_NOOP entry type (additive foundation)
|
||||
- [x] 1. WAL_NOOP entry type (additive foundation)
|
||||
- [x] 1.1 Write RED tests for the no-op entry
|
||||
- Create `testsuite/test_ha_role_lifecycle.c` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (src/controller/meson.build:74-131); first cases cover the entry type only
|
||||
- Tests: `persist_wal_async(p, WAL_NOOP, NULL, 0)` appends a header-only entry (payload_len 0) and broadcasts it (stub replicator observes the entry); `apply_wal_entry` on a WAL_NOOP advances `current_lsn` past the entry's LSN and touches no entity state (jobmgr/queue/worker untouched); boot `replay_wal` over a segment containing WAL_NOOP replays cleanly with the same no-effect guarantee; `wal_classify(WAL_NOOP, 0) == WAL_MODE_RETAIN` (keeps barrier-rollout's totality test green)
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
- Observable completion: all task 1.1 tests PASS; barrier-rollout's classification totality test still PASSES
|
||||
- _Requirements: 2.1_
|
||||
|
||||
- [ ] 2. Promotion rewrite: direct PRIMARY + own-term gate
|
||||
- [x] 2. Promotion rewrite: direct PRIMARY + own-term gate
|
||||
- [x] 2.1 Write RED lifecycle tests for direct promotion and the own-term gate
|
||||
- Tests in `test_ha_role_lifecycle.c`: driving the become-leader path yields `ha->role == CTRL_PRIMARY` directly (no intermediate value observable; `ha_is_primary()` true immediately); `ha_should_redirect()` returns 0 the instant promotion returns, before any standby ACK; persistence hooks are registered by promotion (hooksys observes the three persist hook fns; `hooks_registered == 1`); promotion arms NO delaymgr watchdog (delaymgr task-count seam unchanged across promotion); with a real engine (n_peers=3, stub registry): promotion appends WAL_NOOP at lsn=N, registers it RETAIN, and `commit_set_writable_lsn(N)` is in effect — a `commit_submit(WAIT)` for entry N+1 does NOT complete while ACKs cover N+1 but not N, and completes once `committed_lsn >= N`; with the gate unmet a WAIT submission resolves only as barrier success, `-ETIMEDOUT`, or `-XCE_ERR_NOT_READY` — never via any promotion-side path; n_peers=1: gate opens immediately at promotion (noop self-committed), WAIT returns 0 as before
|
||||
- Observable completion: new cases FAIL (red) against the current COMMITTING-entering `raft_cb_become_leader`
|
||||
|
|
@ -30,14 +30,14 @@
|
|||
- Observable completion: all task 2.1 tests PASS; promotion never blocks (returns without waiting for the noop to commit)
|
||||
- _Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3_
|
||||
|
||||
- [ ] 3. Legacy XCE_TESTING election path symmetry
|
||||
- [x] 3. Legacy XCE_TESTING election path symmetry
|
||||
- [x] 3.1 Update ha_elect_evaluate_and_signal and its direct tests (RED → GREEN in one pass)
|
||||
- Retarget existing legacy-path expectations first (test_ha_start_election_async.c, the evaluation cases in test_ha_role.c): quorum win → role `CTRL_PRIMARY` directly, no watchdog armed — run to confirm RED against current code
|
||||
- In the CTRL_PRIMARY branch of `ha_elect_evaluate_and_signal` (ha.c:2247-2300 anchor): replace the COMMITTING store (2248-2252) with the direct `CTRL_PRIMARY` store, delete the `repl_ack_count` reset (2255) and the watchdog-arm block (2275-2299); keep the config-push/ctrlhello XCE_TESTING hooks; route promotion side effects through the same shared sequence (noop+gate tail guarded on persist/engine availability, as in production)
|
||||
- Observable completion: retargeted legacy-path tests PASS; lifecycle test for symmetry PASSES (both entry points produce identical role/hook/heartbeat/no-watchdog state)
|
||||
- _Requirements: 1.1, 1.5_
|
||||
|
||||
- [ ] 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates)
|
||||
- [x] 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates)
|
||||
- [x] 4.1 Delete the force-promotion watchdog and ack/convergence machinery
|
||||
- Delete from ha.c: `committing_timeout_fn` (3080-3121), `committing_timeout_fn_for_test` (2950-2990), `committing_timeout_ctx_t` type + fwd decl (64-69, 83), the cancel block in `raft_cb_become_follower` (640-648); `ha_check_majority_convergence` (1295-1353) and its heartbeat-reply call site (2710-2715); `ha_repl_ack_received` (3123-3141) — its `OPCode_replCatchUpDone` branch in `repl_ack_decode_and_dispatch` (2659-2661) becomes an INFO-logged no-op (opcode stays gate-allowed; `repl_complete_catchup` untouched); `repl_ack_count` init at ha.c:835
|
||||
- Delete declarations from ha.h: `ha_check_majority_convergence` (396-412), `ha_repl_ack_received` (626-630)
|
||||
|
|
@ -52,7 +52,7 @@
|
|||
- Observable completion: barrier-rollout's gate tests (PRIMARY fact allowlist under no-quorum) still PASS; vote-gate tests PASS with PRIMARY-only denial; suite green
|
||||
- _Requirements: 1.2, 1.3, 1.4_
|
||||
|
||||
- [ ] 5. Test-fixture sweep and final symbol removal
|
||||
- [x] 5. Test-fixture sweep and final symbol removal
|
||||
- [x] 5.1 Retarget the remaining COMMITTING-referencing fixtures
|
||||
- Update expectations COMMITTING→PRIMARY (census: research.md §3): `test_ha_role.c` (37 refs), `test_ha_rdhup_integration.c` (20), `test_ha_repl_push.c` (14), `test_ha_startup.c` (12), `test_ha_connection_reuse.c` (11), `test_ha_start_election_async.c` (7), `test_ha_vote_gate.c` (6, residual from 4.2); delete the 11 `cfg.committing_timeout_ms = 10000;` initializers in `test_config_repl.c`
|
||||
- Each updated test asserts the new contract (direct PRIMARY, no transitional observations), not stubbed compatibility
|
||||
|
|
@ -65,7 +65,7 @@
|
|||
- Observable completion: clean build with zero warnings about the removed symbols; full controller testsuite + ASAN green via `/xce-build controller`
|
||||
- _Requirements: 1.2_
|
||||
|
||||
- [ ] 6. Documentation sync and grep verification
|
||||
- [x] 6. Documentation sync and grep verification
|
||||
- [x] 6.1 Synchronize steering and contributor docs to the three-state model
|
||||
- `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING)
|
||||
- `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob)
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
|
||||
---
|
||||
|
||||
- [ ] 1. repl_broadcast TCP send
|
||||
- [x] 1. repl_broadcast TCP send
|
||||
- [x] 1.1 Fill in repl_broadcast send path
|
||||
- Remove `(void)payload; (void)payload_len;` and the `pending_ack_count++` counter.
|
||||
- For each non-catch-up standby (`s->ep != NULL, !in_catch_up`), build a single-entry `wal_catchup_entry_t` with `lsn=hdr->lsn`, `entry_type=hdr->type`, `raw_payload=payload`, `raw_payload_len=payload_len`.
|
||||
|
|
@ -25,7 +25,7 @@
|
|||
|
||||
---
|
||||
|
||||
- [ ] 2. HA→replicator standby registration bridge
|
||||
- [x] 2. HA→replicator standby registration bridge
|
||||
- [x] 2.1 Register standbys on connect
|
||||
- In `raft_cb_become_leader` (ha.c), after promoting vote eps to `ha->standby_endpoints`: for each promoted ep, call `repl_add_standby(&c->persist.repl, ep, peer_host, peer_port)`.
|
||||
- In `on_standby_connected` (ha.c), after ctrlhello is sent: call `repl_add_standby` with the late-joiner's endpoint, host, and port.
|
||||
|
|
@ -49,7 +49,7 @@
|
|||
|
||||
---
|
||||
|
||||
- [ ] 3. Standby WAL ACK reply
|
||||
- [x] 3. Standby WAL ACK reply
|
||||
- [x] 3.1 Send WalAckReply after WAL entry applied
|
||||
- In `ha_apply_wal_catchup_entry` (ha.c), after `ha_apply_single_entry` succeeds for an entry: construct a `WalAckReply { term=ha->term, lsn=ent->lsn, success=1 }` and send via `xce_reply` to the endpoint that delivered the entry.
|
||||
- The endpoint is not directly available in `ha_apply_wal_catchup_entry` — it's called from `cluster.c:handle_wal_catchup_entry` which has the ep. Pass `ep` as a new parameter: `ha_apply_wal_catchup_entry(ha, p, entry, ep)`.
|
||||
|
|
@ -70,7 +70,7 @@
|
|||
|
||||
---
|
||||
|
||||
- [ ] 4. Primary ACK handler → advance committed_lsn
|
||||
- [x] 4. Primary ACK handler → advance committed_lsn
|
||||
- [x] 4.1 Handle incoming WalAckReply on primary
|
||||
- In `repl_ack_decode_and_dispatch` (ha.c:2506), add case for `RequestReply_data_walackreplymsg`.
|
||||
- Extract `lsn` and `success` from `WalAckReply`. If `success`, call `repl_on_ack(&c->persist.repl, ep, lsn)`.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Implementation Plan
|
||||
|
||||
- [ ] 1. Foundation: failover state, captured-reply fields, and cleanup
|
||||
- [x] 1. Foundation: failover state, captured-reply fields, and cleanup
|
||||
- [x] 1.1 Establish shared failover state and reply-capture fields
|
||||
- Expose the client context to a new client-internal header so the failover helper can operate on it
|
||||
- Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
|
||||
|
|
@ -14,7 +14,7 @@
|
|||
- Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
|
||||
- _Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 2. Core: status classification, endpoint rotation, reply-path capture
|
||||
- [x] 2. Core: status classification, endpoint rotation, reply-path capture
|
||||
- [x] 2.1 Implement status classification
|
||||
- Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
|
||||
- Never classify redirect, standby, no-primary, or not-ready as success
|
||||
|
|
@ -38,7 +38,7 @@
|
|||
- _Boundary: RedirectCapture, ReplyWaitWake_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3. Core: the retry/failover decision helper
|
||||
- [x] 3. Core: the retry/failover decision helper
|
||||
- [x] 3.1 Implement the decision loop (classify, rotate, backoff, budget)
|
||||
- Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
|
||||
- Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
|
||||
|
|
@ -55,7 +55,7 @@
|
|||
- _Depends: 3.1_
|
||||
- _Boundary: xcecli_retry_next_
|
||||
|
||||
- [ ] 4. Core: reconnect/rejoin hook
|
||||
- [x] 4. Core: reconnect/rejoin hook
|
||||
- [x] 4.1 Add the reconnect/rejoin hook mechanism
|
||||
- Provide a client API to register a reconnect callback and argument, stored in the failover state
|
||||
- Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
|
||||
|
|
@ -63,7 +63,7 @@
|
|||
- _Requirements: 1.2_
|
||||
- _Depends: 3.1, 3.2_
|
||||
|
||||
- [ ] 5. Integration: wrap the client ops; controller redirect address
|
||||
- [x] 5. Integration: wrap the client ops; controller redirect address
|
||||
- [x] 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock
|
||||
- Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
|
||||
- Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
> **Verify anchors before editing** (files drift): the design and `research.md` carry 2026-06-27 file:line anchors; re-grep before each edit.
|
||||
> **Root cause (task 1.3, DONE + verified):** the term-blind hypothesis is **refuted**. The real defect is **cross-term catch-up payload loss** — `ha_stream_wal_to_standby` silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Phase 2 targets the streamer + payload-aware guard (`research.md` §"Task 1.3"). A green synthetic unit test over a still-red soak does not satisfy 1.1/8.1 — closure requires the real-fs chaotic soak (7.2) clean.
|
||||
|
||||
- [ ] 1. Foundation: deterministic chaotic harness and live-divergence reproduction
|
||||
- [x] 1. Foundation: deterministic chaotic harness and live-divergence reproduction
|
||||
- [x] 1.1 Extend the divergence detector to a machine-checkable per-member report
|
||||
- Extend `walcompare` to emit a parseable per-member `(lsn -> term, payload-crc, present/MISSING)` report over the committed prefix, and classify each divergence as committed vs uncommitted
|
||||
- Observable completion: `walcompare` prints a parseable per-member report and a committed-vs-uncommitted classification; the existing soak still detects the round-3 LSN divergence through it
|
||||
|
|
@ -38,7 +38,7 @@
|
|||
- _Boundary: Acceptance Soak Gate, Deterministic Chaotic Harness_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [ ] 2. Committed-prefix divergence fix: cross-term catch-up payload integrity + payload-aware guard
|
||||
- [x] 2. Committed-prefix divergence fix: cross-term catch-up payload integrity + payload-aware guard
|
||||
- _Root cause (task 1.3, verified): the cross-term catch-up streamer silently ships `PLEN=0` on a short `pread`/`malloc`-fail, and the `(term,lsn)`-only persist guard is payload-blind. Term and ingest-path are correct — the payload is lost._
|
||||
- [x] 2.1 RED: payload-aware persistence guard tests
|
||||
- Add cases: an empty-payload (`PLEN=0`) entry written at an existing LSN N followed by the correct full-payload entry must end with the FULL payload (today the `(term,lsn)`-only idempotency guard silently no-ops the corrective write); the reverse (full then empty at the same `(term,lsn)`) must NOT degrade the stored entry to empty; an identical `(term, payload-crc)` re-write is a safe idempotent no-op
|
||||
|
|
@ -66,7 +66,7 @@
|
|||
- _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_
|
||||
- _Depends: 2.2, 2.3_
|
||||
|
||||
- [ ] 3. Snapshot-boundary reconciliation
|
||||
- [x] 3. Snapshot-boundary reconciliation
|
||||
- [x] 3.1 RED: snapshot post-install reconcile test
|
||||
- Add a case where a divergent pre-snapshot entry must be reconciled against the installed checkpoint's `(term, lsn)` boundary (today snapshot install sets applied-lsn without a `(term, lsn)` reconcile)
|
||||
- Observable completion: the case FAILS on the baseline because no `(term, lsn)` reconcile occurs at the snapshot boundary
|
||||
|
|
@ -80,7 +80,7 @@
|
|||
- _Boundary: Snapshot Boundary Reconcile_
|
||||
- _Depends: 2.4_
|
||||
|
||||
- [ ] 4. Standby reconvergence completeness
|
||||
- [x] 4. Standby reconvergence completeness
|
||||
- _Root cause (Phase-4/5 investigation, `research.md`): a recovered standby holding a divergent uncommitted tail above the new leader's committed point (former-primary shape, e.g. ctrl-2 with 12-15 at an old term while leader committed 11) never reaches `ae_already_have` — it loops on `ae_reject_logmismatch`, its control channel flaps, and it never rejoins. A short-suffix standby (ctrl-3) DOES reconverge, so the backtrack works for small divergence but not the former-primary tail._
|
||||
- [x] 4.1 RED: divergent-tail reconvergence + apply-gap gate tests
|
||||
- Primary case: construct a recovered standby with a MULTI-entry divergent uncommitted tail above the leader's committed point and assert it reconverges to follower — truncates the conflicting suffix and reaches `ae_already_have` (no permanent `ae_reject_logmismatch` loop). Secondary case: a rejoining node with an injected catch-up apply gap must not be reported converged/healthy until `applied_lsn` and `log_len` equal the leader's committed boundary
|
||||
|
|
@ -95,7 +95,7 @@
|
|||
- _Boundary: Standby Reconvergence_
|
||||
- _Depends: 2.4_
|
||||
|
||||
- [ ] 5. Election liveness and safety under churn
|
||||
- [x] 5. Election liveness and safety under churn
|
||||
- _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._
|
||||
- [x] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests
|
||||
- Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`)
|
||||
|
|
@ -182,7 +182,7 @@
|
|||
- _Boundary: catch-up backfill + election-liveness tail (5.7 family)_
|
||||
- _Depends: 5.9_
|
||||
|
||||
- [ ] 6. Integration: single-seam invariant end-to-end
|
||||
- [x] 6. Integration: single-seam invariant end-to-end
|
||||
- [x] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
|
||||
- Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
|
||||
- Observable completion: an end-to-end test (or instrumented soak round) shows leader and standby hold identical `(term, payload-crc)` for every replicated and cross-term-caught-up LSN; `walcompare --report` clean across members
|
||||
|
|
@ -196,7 +196,7 @@
|
|||
- _Requirements: 1.1, 1.2, 2.4, 3.1_
|
||||
- _Depends: 2.4, 3.2, 4.2_
|
||||
|
||||
- [ ] 7. Validation: regression and acceptance soak gate
|
||||
- [x] 7. Validation: regression and acceptance soak gate
|
||||
- [x] 7.1 Full controller suite under ASAN with the new harness green
|
||||
- Build and run the full controller unit suite under ASAN (suite-by-suite if the aggregate hits the known 300s cap); fix any regression from the ingest/persist/election changes
|
||||
- Observable completion: the full suite is green under ASAN and every `test_ha_core_safety` case passes with no ASAN errors
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue