fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node rotation) can leave a standby log-conflicted after a rejoin. Fixing this needed three compounding layers, each unit-green but still failing the identical live scenario until the next layer landed: 1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft library's nextIndex backoff and immediately repl_start_catchup's the standby at the corrected position, instead of the 16-strike silent eviction with no catch-up driver after it. Correct from the start, but live soak still failed identically: nothing ever produced the NACK this code was waiting for. 2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of silently dropping the reject. ha_send_wal_ack_reply had hardcoded success=1 unconditionally -- the only WalAckReply sender in the controller -- making (1)'s NACK-routing correct but unreachable dead code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe the NACK without a live endpoint. Live-confirmed the NACK now reaches the primary -- but the scenario still failed for a third reason. 3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Widens the STREAM_WAL ceiling (and the convergence check gating it) to the leader's own log tail, matching raft.tla's real AppendEntries semantics (leaderCommit is advisory, never a send-cap), while reusing bug-1478's exact guard so this doesn't reopen that trap (never widen onto a trailing local-only checkpoint's own unstreamable LSN). Verification: controller_unit_test 1053/1053 PASS, clean twice, including new regression tests for all three layers (test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its pre-fix behavior before the real fix was restored. Three successive live run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the final run passed end-to-end -- all 3 force_elect_node rounds succeeded (ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never previously reached), zero materialized-state divergence, zero job loss. Unblocks config-upgrade-snapshot task 8.6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
aecd8766f4
commit
34ab78d72d
14 changed files with 1256 additions and 377 deletions
|
|
@ -1,25 +1,23 @@
|
|||
{
|
||||
"feature_name": "config-upgrade-snapshot",
|
||||
"created_at": "2026-06-21T11:30:00Z",
|
||||
"updated_at": "2026-06-21T11:30:00Z",
|
||||
"updated_at": "2026-07-29T04:10:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-generated",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": [
|
||||
"bug-1418 (apply-lag promotion-gate liveness deadlock) — decided 2026-07-21 as prepare work for this spec: fix before/alongside implementation. Contaminates this spec's own Requirement 6 chaotic-multi-kill soak signal, and its graceful-degrade fix pattern is the template Requirement 3's activation gate must mirror. No spec currently owns it — scope as task 0 / a small prerequisite fix. See project memory project_bug1418_prework_decision.md."
|
||||
]
|
||||
"ready_for_implementation": true,
|
||||
"blocked_on": []
|
||||
}
|
||||
|
|
|
|||
|
|
@ -121,13 +121,13 @@
|
|||
- _Requirements: 6.2_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover
|
||||
- [x] 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover
|
||||
- Drive a configuration upgrade while repeatedly killing and restarting the primary, and assert every member's materialized state and active configuration are identical afterward
|
||||
- Assert a host's administratively-set status specifically survives the same scenario, not merely inferred from the aggregate state check
|
||||
- Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members
|
||||
- _Requirements: 4.1, 4.3, 6.1, 6.4_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Blocked: bug-1477 — pre-existing, out-of-spec-boundary HA bootstrap/election defect (`myapp.c`'s `skip_election` fast path, ~343-425, plus `ha.c`'s unconditional `phase2_done=1` latch across all 4 retry call sites — `raft_cb_become_follower` 2000-2012, `raft_cb_become_leader` 1772-1783, `ha_elect_evaluate_and_signal` 5609-5629/5699-5710; introduced commits `20e59d0b`/`8f0d2d55`, ~1 month before this spec). bug-1472 (the PRIOR blocker on this task, worker-control replication) is now fixed and verified (commit `7547a8f1`); rebuilding and re-running `--config-upgrade` on top of that fix surfaced a NEW, unrelated defect: a node that finds "leader already known" at boot (`skip_election`) makes exactly one `cluster_init_phase2()` attempt, which can lose the race against config becoming available and return `-EINVAL` — and since none of the 4 retry call sites ever fire for a node that never itself contests an election, that node's `commit_engine_init` (and therefore `apply_committed_entry`/`persist_checkpoint_marker_triggered`, config-upgrade-snapshot's own task-5 mechanism) never runs, for the rest of that process's life. The member's config still "activates" cosmetically via the independent `handle_repl_config_push` RPC path, masking the defect until a checkpoint-identity check catches it: that member permanently lacks the marker-triggered checkpoint every other member takes at the same LSN, and since `WAL_CHECKPOINT` is local-only/non-replicated by design, it can never reconstruct one — a permanent `ae_reject_logmismatch` catch-up wedge, distinct from bug-1475's (already-fixed) transient race. Completely general, unrelated to config upgrades specifically — any HA bootstrap timing that lands a node on `skip_election` before its config is available is exposed the same way. See `.wolf/buglog.json` bug-1477 for full forensic detail (log/grep/git-blame evidence, no unit-level repro built — the multi-node startup-stagger timing isn't cleanly unit-testable in isolation). Unblocks when bug-1477 is fixed by whoever owns HA bootstrap/election (add a retry path for a `skip_election` node whose first `cluster_init_phase2` attempt fails, and gate the `phase2_done=1` latch on that call's return value) — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
|
||||
- **Unblocked 2026-07-29 — bug-1479 fully fixed and live-verified.** `run-ha-soak.sh --config-upgrade` now reports a clean PASS through all 3 `force_elect_node` rounds (see Implementation Notes bullet "Task 8.6 — bug-1479 fully fixed 2026-07-29" below for the full account). Prior blockers bug-1472/bug-1477/bug-1479 all fixed and verified in sequence; no further blocker remains.
|
||||
|
||||
- [x] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
|
||||
- Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart
|
||||
|
|
@ -164,6 +164,10 @@
|
|||
- **Still open, blocking a clean PASS — bug-1471**: a separate, much narrower 1-byte checkpoint divergence in a host-admin-status frame (not a config-activation issue), found independently by two concurrent sessions working this task and root-cause-not-yet-isolated. Also noted but deliberately **not** triggered by the current soak scenario: a deeper, genuinely separate gap where no live mechanism exists anywhere in this codebase to add a *new* queue to an already-running member's runtime `qmgr` without a restart (`config_activate_committed`/`load_primary_config` refresh `myconf->cluster_config` but nothing syncs that into the live `qmgr`) — the soak's upgrade-target config change was deliberately scoped to modifying an *existing* queue's field rather than adding one, specifically to test what Requirements 4.1-4.3/6.1/6.4 actually ask for without tripping over this separate, deeper architectural gap. Both are flagged in `.wolf/buglog.json` for whoever next picks up bug-1471 (or a dedicated follow-up for the queue-creation gap) — task 8.6's own checkbox stays unchecked until the soak reports a clean PASS.
|
||||
- **Process note**: this task was interrupted by the same monthly-spend-limit condition hit earlier in this session (task 6.1), across three separate dispatch/resume cycles. Each resumption correctly checked live `git status`/`diff` before acting rather than blindly redoing work, and the final state was independently re-verified end-to-end (build + full 1025-test suite + file-integrity spot check) before being reviewed and committed — no corruption or duplicated work resulted, but it's worth flagging that this class of interruption can leave multiple agent instances transiently active in the same worktree.
|
||||
|
||||
- **Task 8.6 re-run 2026-07-29 (bug-1477 fix reconfirmed; NEW blocker bug-1479 reached, still open):** rebuilt `xce_controller`/`xce_test`/`controller_unit_test` fresh against the working tree's bug-1479 fix attempt (`ha_raft_on_repl_nack`/`repl_get_catchup_join_lsn` in `ha.c`/`ha.h`/`repl.c`/`repl.h`, 1048/1048 `controller_unit_test` green) and re-ran `run-ha-soak.sh --config-upgrade` end to end (deploy preserved, not deleted, at `~/tmp/xce_ha_bug1479_verify`). The chaotic 3-round primary rotation, the live config-upgrade commit, and the checkpoint-identity/config-identity/host-admin-status-survival checks ALL PASSED cleanly a second time (confirms bug-1477 stays fixed). The run then failed in this task's own dedicated end-of-round `force_elect_node` loop (the freshness-fence maneuver bullet 3 of task 8.6's Requirement 6.4 verification): `force-elect ctrl-1` no-op'd correctly, but `force-elect ctrl-2` hard-failed — after killing spoiler `ctrl-3`, `submit_jobs` against `{ctrl-1,ctrl-2}` got 0/3 acked across all 5 retry attempts. Root-caused (log-correlated across both nodes, not guessed): this is **bug-1479 still not fully fixed** — the fix's new primary-side NACK-routing code is correct but unreachable, because the standby's own AppendEntries-reject path (`ha_apply_wal_catchup_entry`/`ha_apply_term_entry_via_library`, `ha.c` ~3230-3232/3320-3321/3373-3374) sends no reply at all on reject, and `ha_send_wal_ack_reply` (the only `WalAckReply` sender in the controller) hardcodes `success=1` — there is no production path that ever sends a NACK. See `.wolf/buglog.json` bug-1479 (occurrence 2, updated 2026-07-29) for the full live-log evidence trail. This is a pre-existing HA-replication defect, same family as bug-1466/1471/1472/1474/1475/1477/1478, out of THIS spec's boundary (task 8.6 only exercises the `force_elect_node` mechanism that exposes it, same relationship the task already had to bug-1477/bug-1472) — task 8.6's checkbox stays unchecked until it is fixed and a fresh `run-ha-soak.sh --config-upgrade` reports a clean PASS through all 3 `force_elect_node` rounds. New follow-up spec created to track the fix plus a broader systematic check for siblings of this same gap: `.kiro/specs/raft-adapter-conformance/` (a `raft.tla` action-by-action conformance pass over the controller's Raft adapter — `ha.c`/`repl.c`/`config_repl.c` — distinct in scope from the existing `raft-sim-conformance` spec, which validates only `src/lib/raft/raft_sim.c`, the in-process library harness, and explicitly excludes this controller-adapter wire code).
|
||||
|
||||
- **Task 8.6 — bug-1479 fully fixed 2026-07-29 (task now `[x]`, clean live PASS):** closing the gap identified in the re-run above needed TWO more compounding fixes beyond the primary-side NACK routing, each verified with its own dedicated live soak re-run because the first two looked complete after a clean unit-test pass yet still failed the identical live scenario for a NEW reason each time. **Fix A (standby-side NACK sending):** `ha_apply_term_entry_via_library`'s logOk-reject path (the one live-reachable reject site; the other two in `ha_apply_wal_catchup_entry` are documented dormant/unreachable with real senders) now calls a new `ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success)` with `success=0` instead of silently dropping the reject — `ha_send_wal_ack_reply` had hardcoded `success=1` unconditionally, the ONLY `WalAckReply` sender anywhere in the controller, which made the primary's `ha_raft_on_repl_nack` unreachable dead code no matter how correct it was. New `ha_wal_ack_send_hook` (ha.h, XCE_TESTING) lets a unit test observe the NACK without a live endpoint (`test_bug1479_ae_reject_sends_nack_not_silence`, `test_config_repl.c`). Live re-run (`~/tmp/xce_ha_bug1479_verify_v2`) confirmed NACKs now genuinely reach the primary (`ctrl-1.log`: "WAL NACK from standby ... success=0 — marking catch-up") — but the SAME force-elect ctrl-2 scenario still failed, because of gap B. **Fix B (catch-up stream ceiling):** `ha_stream_wal_to_standby`'s catch-up range was hard-bounded at `committed_lsn`, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) — a self-referential deadlock invisible from either side's code in isolation. New pure function `ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn)` (ha.c/ha.h), wired into `ha_dispatch_repl_action_for_standby`, widens the STREAM_WAL ceiling (and the `ha_evaluate_repl_action` convergence check gating it) to the leader's own log tail instead of just `committed_lsn` — matching `raft.tla`'s real AppendEntries semantics (`leaderCommit` is advisory to the follower, never a send-cap) — while re-using bug-1478's exact guard (never widen onto a trailing local-only checkpoint's own unstreamable LSN slot) so this fix did not reopen that one. 4 new pure-function tests in `test_ha_role.c` (`test_compute_repl_stream_ceiling_lsn_*`). Both fixes RED-confirmed individually (each reverted to its pre-fix behavior in turn, re-run, confirmed the predicted failure, restored) before being combined. **Final verification (`~/tmp/xce_ha_bug1479_verify_v3`, not deleted):** `controller_unit_test` 1053/1053 PASS twice clean; live `run-ha-soak.sh --config-upgrade` PASSED end-to-end — all 3 `force_elect_node` rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7, never previously reached), final verdict "PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members." `.kiro/specs/raft-adapter-conformance/`'s concrete seed defect (missing NACK reply) is now fixed here; that spec's broader `raft.tla` action-by-action conformance audit remains valid, separate future work (fix B's own committed_lsn-vs-log-tail conflation is exactly the kind of second sibling gap that audit is meant to catch systematically).
|
||||
|
||||
- **Task 8.5 redirection (design.md's original mechanism was wrong; this is not the same as a review rejection):** the original hypothesis for bug-857's divergence — a job whose queue is removed by a config upgrade gets re-homed to `lost_and_found` on some members but not others, based on per-node restart timing — turned out to be **aspirational, never-implemented code**. Exhaustive grep across `job.c`/`queue.c`/`worker.c`/`config_repl.c` at both the pre-fix baseline commit (`e0c5e8b7`) and current HEAD confirmed: `create_lost_and_found` only ever creates the sentinel queue/host objects (and its entire body is `#ifndef XCE_TESTING`-gated); `restore_job`/`restore_job_array` never consult `qmgr`; `restore_queue_state` explicitly *skips* rather than re-homes when a queue is missing. `.wolf/buglog.json`'s bug-857 entry itself flags this: "identified via TLA+ gap analysis... NOT empirically reproduced yet." design.md's Requirements Traceability table assigned 6.2 to a live "HA soak new scenario" — also written against the same since-disproven hypothesis. Redirected (orchestrator decision, not a task-local implementer call) to a real, code-backed mechanism instead: `schedule_checkpoint`'s pre-existing, unmodified-by-this-spec WAL-record-count threshold trigger is genuinely decoupled from the config-marker `(term,lsn)` boundary, so two members' independently-timed checkpoints can capture different job/queue sets — the actual, general-purpose truth behind bug-857's claim ("no snapshot transfers an identical materialized state"). This is testable entirely within `controller_unit_test`, no live multi-process soak or old-commit worktree needed, since the threshold-trigger mechanism is still live, unmodified production code coexisting with the new marker-trigger at HEAD. Landed as `test_bug857_regression.c` (2 tests: divergence under threshold-only triggering vs. convergence under the real marker-triggered path, the latter under deliberately differing per-member processing pace to make the convergence claim non-trivial). Reviewer independently re-derived and confirmed the mechanism claim from first principles (not just re-running the tests) before approving — see `git blame`/review transcript for the full verification chain (schedule_checkpoint predates this spec via `git merge-base --is-ancestor`, zero lines of its body touched by tasks 1-7, `persist_apply_pump_run`'s strictly-sequential dispatch loop explains why the marker-triggered checkpoint's convergence isn't a timing coincidence). design.md's Requirements Traceability table (row 6.2) and Testing Strategy section were both corrected to match. **Heads-up for tasks 8.6-8.9**: they still legitimately need live multi-process HA-soak-level testing for Requirement 6.1 (chaotic multi-kill + config upgrade) — that infrastructure (a live config-upgrade CLI trigger doesn't exist anywhere in this codebase yet; confirmed via grep of `test.c`/`src/frontend`/`src/tools`) was NOT built by this task and will need to be built by whichever of 8.6-8.9 first needs to drive a live `OP_upgradeConfig` RPC against a running cluster.
|
||||
|
||||
- **Task 8.4 round-1 rejection (real gap, caught by review):** the first pass built `configcompare_paths()` (correct, well-tested against `persist.c`/`config_repl.c`'s real `workdir/name/conf/current`+`conf/cluster.bin.<hex>` resolution convention) and linked `configcompare.c` into both `xce_test_src` and `controller_unit_test_src` in meson, but never registered it as a `cmd_configcompare` subcommand in `testsuite/test.c` — unlike its siblings `walcompare`/`snapverify`/`snapdiff`, each of which has a `cmd_*` wrapper in BOTH `top_commands[]` and `client_commands[]`. Left as delivered, the tool was a tested library function with no shell entry point — unusable by task 8.6/8.9's later HA soak scenarios, which design.md explicitly says will invoke `configcompare` directly against each node's data directory. Fixed with a 25-line, `test.c`-only remediation mirroring `cmd_snapdiff`'s exact shape; verified end-to-end via `echo 'configcompare <dir1> <dir2>' | xce_test` (checked `$?` for both IDENTICAL/exit-0 and DIVERGENT/exit-1 cases, not just printed text). **Heads-up for any future task in this spec (or elsewhere) that adds a new offline testsuite tool in this family**: linking the `.c` file into the meson source lists is NOT sufficient — it must also get a `cmd_*` wrapper + `top_commands[]`/`client_commands[]` entries in `test.c`, or it silently has no way to be invoked outside the unit test binary.
|
||||
|
|
|
|||
84
.kiro/specs/raft-adapter-conformance/requirements.md
Normal file
84
.kiro/specs/raft-adapter-conformance/requirements.md
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
|
||||
Systematic conformance gap-validation of the XCE controller's Raft replication ADAPTER
|
||||
(`src/controller/ha.c`, `repl.c`, `config_repl.c` — the real wire-level send/receive/reply
|
||||
code) against `raft.tla`'s action model, action by action (AppendEntries request/response,
|
||||
RequestVote, ClientRequest, AdvanceCommitIndex, term rules). This is distinct from and
|
||||
complements the existing `raft-sim-conformance` spec, whose boundary is explicitly scoped to
|
||||
`src/lib/raft/raft_sim.c` only (an in-process, in-memory harness validating the LIBRARY's
|
||||
protocol semantics) and explicitly does not touch the controller adapter code.
|
||||
|
||||
**Motivating finding (already confirmed live — seed this spec with it as a known first
|
||||
defect rather than rediscovering it):** during a fresh live-soak re-verification of
|
||||
bug-1479's fix (`config-upgrade-snapshot` task 8.6's `--config-upgrade` scenario,
|
||||
`run-ha-soak.sh`, deploy preserved at `~/tmp/xce_ha_bug1479_verify`), the fix's new
|
||||
primary-side NACK-routing code (`ha_raft_on_repl_nack` + `repl_start_catchup` in
|
||||
`repl_ack_decode_and_dispatch`'s WAL-ACK-reply-not-success branch, `ha.c` ~6680-6720) was
|
||||
confirmed CORRECT but UNREACHABLE in production: `ha.c`'s `ha_apply_wal_catchup_entry` has
|
||||
three reject paths (~3230-3232 for the term-carrying single/batch entry via
|
||||
`ha_apply_term_entry_via_library`, ~3320-3321 for a batch sub-entry apply failure, ~3373-3374
|
||||
for the legacy single-entry path) that all explicitly send NO reply at all on a logOk reject
|
||||
/ forward-gap (not even a NACK) — and `ha_send_wal_ack_reply` (`ha.c:3084-3106`), the ONLY
|
||||
function anywhere in the controller that ever constructs a `WalAckReply`, hardcodes
|
||||
`success=1` unconditionally. There is no production code path that ever sends a failing
|
||||
`WalAckReply`. In `raft.tla`, AppendEntries always yields a response (success true or false)
|
||||
that drives nextIndex backoff; this codebase's silent-drop-on-reject deviates from that and
|
||||
reproduces bug-1479's original permanent-quorum-loss symptom via a different mechanism
|
||||
(confirmed live: `ctrl-2` standby stuck rejecting `ae_reject_logmismatch` at growing
|
||||
`prevIdx` forever, `ctrl-1` primary logs zero WAL-NACK messages the entire time,
|
||||
`committed_lsn` frozen, job submissions permanently stuck at 0/3 acked).
|
||||
|
||||
**Scope for this spec:**
|
||||
1. Fix the concrete missing-NACK-reply defect at the three reject sites so a real
|
||||
`WalAckReply{success=0}` reaches the primary and the bug-1479 NACK-routing code can
|
||||
actually engage.
|
||||
2. Beyond that one confirmed defect, perform the broader systematic pass — walk `raft.tla`'s
|
||||
action list (AppendEntries send+response, RequestVote/PreVote send+response,
|
||||
ClientRequest/log append, AdvanceCommitIndex, term/UpdateTerm handling,
|
||||
Restart/recovery) against the corresponding controller adapter code paths in
|
||||
`ha.c`/`repl.c`/`config_repl.c` and flag every other place a response, retry, or backoff
|
||||
the spec model assumes is silently missing or short-circuited — the same class of gap
|
||||
this first finding represents.
|
||||
|
||||
**Depends on / builds on:** `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`,
|
||||
`raft-sim-conformance` (consumes their library entry points and prior `validate_impl`
|
||||
`known_limitations`, several of which already flagged "documentation drift" around this
|
||||
exact send/receive boundary — cross-reference those notes rather than re-deriving them).
|
||||
|
||||
**Related bugs:** bug-1479 (this session's re-verification, `buglog.json`, occurrence 2 —
|
||||
still not fully fixed), bug-1466/1471/1472/1474/1475/1477/1478 (same HA-replication defect
|
||||
family).
|
||||
|
||||
## Status Update (2026-07-29)
|
||||
|
||||
**Scope item 1 (the concrete missing-NACK-reply defect) is FIXED**, as part of closing out
|
||||
`config-upgrade-snapshot` task 8.6 / bug-1479. `ha_apply_term_entry_via_library`'s logOk-reject
|
||||
path (the one live-reachable reject site of the three named above — the other two are
|
||||
documented dormant/unreachable with real senders, see `ha_apply_wal_catchup_entry`'s own
|
||||
comments) now sends a real `WalAckReply{success=0}` via a new `ha_send_wal_ack_reply_ex`
|
||||
helper, confirmed live: the primary now genuinely receives and routes the NACK
|
||||
(`ha_raft_on_repl_nack`, previously-correct-but-unreachable code, now actually fires).
|
||||
|
||||
Closing that gap alone was still not sufficient — it exposed a SECOND, independent adapter
|
||||
gap (also now fixed, also a `raft.tla`-deviation of exactly the kind this spec's scope item 2
|
||||
is meant to find systematically): `ha_stream_wal_to_standby`'s catch-up range was hard-bounded
|
||||
at `committed_lsn` rather than the leader's real log tail, which is a `raft.tla` deviation
|
||||
(`leaderCommit` is advisory to the follower in the real spec, never a send-cap) and caused a
|
||||
self-referential deadlock whenever the streamed-to standby's own ack was required to advance
|
||||
`committed_lsn` in the first place. Fixed via a new `ha_compute_repl_stream_ceiling_lsn` pure
|
||||
function. Full live re-verification (`run-ha-soak.sh --config-upgrade`, 3/3 `force_elect_node`
|
||||
rounds) now PASSES end-to-end. See `.wolf/buglog.json` bug-1479 and
|
||||
`config-upgrade-snapshot/tasks.md` task 8.6's Implementation Notes for the complete fix
|
||||
narrative and evidence.
|
||||
|
||||
**Scope item 2 (the broader systematic `raft.tla` action-by-action pass) remains open and
|
||||
valid** — the fact that fixing item 1 alone immediately surfaced a second, independent
|
||||
sibling gap (the committed_lsn-vs-log-tail conflation) is itself evidence this class of
|
||||
adapter/spec-model divergence is not a one-off, and a systematic pass is still worth doing
|
||||
to find any remaining siblings before they're each independently rediscovered via a live
|
||||
soak the way these two were.
|
||||
|
||||
## Requirements
|
||||
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||
25
.kiro/specs/raft-adapter-conformance/spec.json
Normal file
25
.kiro/specs/raft-adapter-conformance/spec.json
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"feature_name": "raft-adapter-conformance",
|
||||
"created_at": "2026-07-29T03:10:00Z",
|
||||
"updated_at": "2026-07-29T03:10:00Z",
|
||||
"language": "en",
|
||||
"phase": "initialized",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": [],
|
||||
"related_bugs": ["bug-1479", "bug-1466", "bug-1471", "bug-1472", "bug-1474", "bug-1475", "bug-1477", "bug-1478"],
|
||||
"related_specs": ["raft-log-replication", "raft-commit-safety", "ha-term-safety", "raft-sim-conformance", "config-upgrade-snapshot"]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue