fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node rotation) can leave a standby log-conflicted after a rejoin. Fixing this needed three compounding layers, each unit-green but still failing the identical live scenario until the next layer landed: 1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft library's nextIndex backoff and immediately repl_start_catchup's the standby at the corrected position, instead of the 16-strike silent eviction with no catch-up driver after it. Correct from the start, but live soak still failed identically: nothing ever produced the NACK this code was waiting for. 2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of silently dropping the reject. ha_send_wal_ack_reply had hardcoded success=1 unconditionally -- the only WalAckReply sender in the controller -- making (1)'s NACK-routing correct but unreachable dead code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe the NACK without a live endpoint. Live-confirmed the NACK now reaches the primary -- but the scenario still failed for a third reason. 3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Widens the STREAM_WAL ceiling (and the convergence check gating it) to the leader's own log tail, matching raft.tla's real AppendEntries semantics (leaderCommit is advisory, never a send-cap), while reusing bug-1478's exact guard so this doesn't reopen that trap (never widen onto a trailing local-only checkpoint's own unstreamable LSN). Verification: controller_unit_test 1053/1053 PASS, clean twice, including new regression tests for all three layers (test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its pre-fix behavior before the real fix was restored. Three successive live run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the final run passed end-to-end -- all 3 force_elect_node rounds succeeded (ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never previously reached), zero materialized-state divergence, zero job loss. Unblocks config-upgrade-snapshot task 8.6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
aecd8766f4
commit
34ab78d72d
14 changed files with 1256 additions and 377 deletions
|
|
@ -1,25 +1,23 @@
|
|||
{
|
||||
"feature_name": "config-upgrade-snapshot",
|
||||
"created_at": "2026-06-21T11:30:00Z",
|
||||
"updated_at": "2026-06-21T11:30:00Z",
|
||||
"updated_at": "2026-07-29T04:10:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-generated",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": [
|
||||
"bug-1418 (apply-lag promotion-gate liveness deadlock) — decided 2026-07-21 as prepare work for this spec: fix before/alongside implementation. Contaminates this spec's own Requirement 6 chaotic-multi-kill soak signal, and its graceful-degrade fix pattern is the template Requirement 3's activation gate must mirror. No spec currently owns it — scope as task 0 / a small prerequisite fix. See project memory project_bug1418_prework_decision.md."
|
||||
]
|
||||
"ready_for_implementation": true,
|
||||
"blocked_on": []
|
||||
}
|
||||
|
|
|
|||
|
|
@ -121,13 +121,13 @@
|
|||
- _Requirements: 6.2_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover
|
||||
- [x] 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover
|
||||
- Drive a configuration upgrade while repeatedly killing and restarting the primary, and assert every member's materialized state and active configuration are identical afterward
|
||||
- Assert a host's administratively-set status specifically survives the same scenario, not merely inferred from the aggregate state check
|
||||
- Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members
|
||||
- _Requirements: 4.1, 4.3, 6.1, 6.4_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Blocked: bug-1477 — pre-existing, out-of-spec-boundary HA bootstrap/election defect (`myapp.c`'s `skip_election` fast path, ~343-425, plus `ha.c`'s unconditional `phase2_done=1` latch across all 4 retry call sites — `raft_cb_become_follower` 2000-2012, `raft_cb_become_leader` 1772-1783, `ha_elect_evaluate_and_signal` 5609-5629/5699-5710; introduced commits `20e59d0b`/`8f0d2d55`, ~1 month before this spec). bug-1472 (the PRIOR blocker on this task, worker-control replication) is now fixed and verified (commit `7547a8f1`); rebuilding and re-running `--config-upgrade` on top of that fix surfaced a NEW, unrelated defect: a node that finds "leader already known" at boot (`skip_election`) makes exactly one `cluster_init_phase2()` attempt, which can lose the race against config becoming available and return `-EINVAL` — and since none of the 4 retry call sites ever fire for a node that never itself contests an election, that node's `commit_engine_init` (and therefore `apply_committed_entry`/`persist_checkpoint_marker_triggered`, config-upgrade-snapshot's own task-5 mechanism) never runs, for the rest of that process's life. The member's config still "activates" cosmetically via the independent `handle_repl_config_push` RPC path, masking the defect until a checkpoint-identity check catches it: that member permanently lacks the marker-triggered checkpoint every other member takes at the same LSN, and since `WAL_CHECKPOINT` is local-only/non-replicated by design, it can never reconstruct one — a permanent `ae_reject_logmismatch` catch-up wedge, distinct from bug-1475's (already-fixed) transient race. Completely general, unrelated to config upgrades specifically — any HA bootstrap timing that lands a node on `skip_election` before its config is available is exposed the same way. See `.wolf/buglog.json` bug-1477 for full forensic detail (log/grep/git-blame evidence, no unit-level repro built — the multi-node startup-stagger timing isn't cleanly unit-testable in isolation). Unblocks when bug-1477 is fixed by whoever owns HA bootstrap/election (add a retry path for a `skip_election` node whose first `cluster_init_phase2` attempt fails, and gate the `phase2_done=1` latch on that call's return value) — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade` to confirm clean PASS)._
|
||||
- **Unblocked 2026-07-29 — bug-1479 fully fixed and live-verified.** `run-ha-soak.sh --config-upgrade` now reports a clean PASS through all 3 `force_elect_node` rounds (see Implementation Notes bullet "Task 8.6 — bug-1479 fully fixed 2026-07-29" below for the full account). Prior blockers bug-1472/bug-1477/bug-1479 all fixed and verified in sequence; no further blocker remains.
|
||||
|
||||
- [x] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
|
||||
- Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart
|
||||
|
|
@ -164,6 +164,10 @@
|
|||
- **Still open, blocking a clean PASS — bug-1471**: a separate, much narrower 1-byte checkpoint divergence in a host-admin-status frame (not a config-activation issue), found independently by two concurrent sessions working this task and root-cause-not-yet-isolated. Also noted but deliberately **not** triggered by the current soak scenario: a deeper, genuinely separate gap where no live mechanism exists anywhere in this codebase to add a *new* queue to an already-running member's runtime `qmgr` without a restart (`config_activate_committed`/`load_primary_config` refresh `myconf->cluster_config` but nothing syncs that into the live `qmgr`) — the soak's upgrade-target config change was deliberately scoped to modifying an *existing* queue's field rather than adding one, specifically to test what Requirements 4.1-4.3/6.1/6.4 actually ask for without tripping over this separate, deeper architectural gap. Both are flagged in `.wolf/buglog.json` for whoever next picks up bug-1471 (or a dedicated follow-up for the queue-creation gap) — task 8.6's own checkbox stays unchecked until the soak reports a clean PASS.
|
||||
- **Process note**: this task was interrupted by the same monthly-spend-limit condition hit earlier in this session (task 6.1), across three separate dispatch/resume cycles. Each resumption correctly checked live `git status`/`diff` before acting rather than blindly redoing work, and the final state was independently re-verified end-to-end (build + full 1025-test suite + file-integrity spot check) before being reviewed and committed — no corruption or duplicated work resulted, but it's worth flagging that this class of interruption can leave multiple agent instances transiently active in the same worktree.
|
||||
|
||||
- **Task 8.6 re-run 2026-07-29 (bug-1477 fix reconfirmed; NEW blocker bug-1479 reached, still open):** rebuilt `xce_controller`/`xce_test`/`controller_unit_test` fresh against the working tree's bug-1479 fix attempt (`ha_raft_on_repl_nack`/`repl_get_catchup_join_lsn` in `ha.c`/`ha.h`/`repl.c`/`repl.h`, 1048/1048 `controller_unit_test` green) and re-ran `run-ha-soak.sh --config-upgrade` end to end (deploy preserved, not deleted, at `~/tmp/xce_ha_bug1479_verify`). The chaotic 3-round primary rotation, the live config-upgrade commit, and the checkpoint-identity/config-identity/host-admin-status-survival checks ALL PASSED cleanly a second time (confirms bug-1477 stays fixed). The run then failed in this task's own dedicated end-of-round `force_elect_node` loop (the freshness-fence maneuver bullet 3 of task 8.6's Requirement 6.4 verification): `force-elect ctrl-1` no-op'd correctly, but `force-elect ctrl-2` hard-failed — after killing spoiler `ctrl-3`, `submit_jobs` against `{ctrl-1,ctrl-2}` got 0/3 acked across all 5 retry attempts. Root-caused (log-correlated across both nodes, not guessed): this is **bug-1479 still not fully fixed** — the fix's new primary-side NACK-routing code is correct but unreachable, because the standby's own AppendEntries-reject path (`ha_apply_wal_catchup_entry`/`ha_apply_term_entry_via_library`, `ha.c` ~3230-3232/3320-3321/3373-3374) sends no reply at all on reject, and `ha_send_wal_ack_reply` (the only `WalAckReply` sender in the controller) hardcodes `success=1` — there is no production path that ever sends a NACK. See `.wolf/buglog.json` bug-1479 (occurrence 2, updated 2026-07-29) for the full live-log evidence trail. This is a pre-existing HA-replication defect, same family as bug-1466/1471/1472/1474/1475/1477/1478, out of THIS spec's boundary (task 8.6 only exercises the `force_elect_node` mechanism that exposes it, same relationship the task already had to bug-1477/bug-1472) — task 8.6's checkbox stays unchecked until it is fixed and a fresh `run-ha-soak.sh --config-upgrade` reports a clean PASS through all 3 `force_elect_node` rounds. New follow-up spec created to track the fix plus a broader systematic check for siblings of this same gap: `.kiro/specs/raft-adapter-conformance/` (a `raft.tla` action-by-action conformance pass over the controller's Raft adapter — `ha.c`/`repl.c`/`config_repl.c` — distinct in scope from the existing `raft-sim-conformance` spec, which validates only `src/lib/raft/raft_sim.c`, the in-process library harness, and explicitly excludes this controller-adapter wire code).
|
||||
|
||||
- **Task 8.6 — bug-1479 fully fixed 2026-07-29 (task now `[x]`, clean live PASS):** closing the gap identified in the re-run above needed TWO more compounding fixes beyond the primary-side NACK routing, each verified with its own dedicated live soak re-run because the first two looked complete after a clean unit-test pass yet still failed the identical live scenario for a NEW reason each time. **Fix A (standby-side NACK sending):** `ha_apply_term_entry_via_library`'s logOk-reject path (the one live-reachable reject site; the other two in `ha_apply_wal_catchup_entry` are documented dormant/unreachable with real senders) now calls a new `ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success)` with `success=0` instead of silently dropping the reject — `ha_send_wal_ack_reply` had hardcoded `success=1` unconditionally, the ONLY `WalAckReply` sender anywhere in the controller, which made the primary's `ha_raft_on_repl_nack` unreachable dead code no matter how correct it was. New `ha_wal_ack_send_hook` (ha.h, XCE_TESTING) lets a unit test observe the NACK without a live endpoint (`test_bug1479_ae_reject_sends_nack_not_silence`, `test_config_repl.c`). Live re-run (`~/tmp/xce_ha_bug1479_verify_v2`) confirmed NACKs now genuinely reach the primary (`ctrl-1.log`: "WAL NACK from standby ... success=0 — marking catch-up") — but the SAME force-elect ctrl-2 scenario still failed, because of gap B. **Fix B (catch-up stream ceiling):** `ha_stream_wal_to_standby`'s catch-up range was hard-bounded at `committed_lsn`, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) — a self-referential deadlock invisible from either side's code in isolation. New pure function `ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn)` (ha.c/ha.h), wired into `ha_dispatch_repl_action_for_standby`, widens the STREAM_WAL ceiling (and the `ha_evaluate_repl_action` convergence check gating it) to the leader's own log tail instead of just `committed_lsn` — matching `raft.tla`'s real AppendEntries semantics (`leaderCommit` is advisory to the follower, never a send-cap) — while re-using bug-1478's exact guard (never widen onto a trailing local-only checkpoint's own unstreamable LSN slot) so this fix did not reopen that one. 4 new pure-function tests in `test_ha_role.c` (`test_compute_repl_stream_ceiling_lsn_*`). Both fixes RED-confirmed individually (each reverted to its pre-fix behavior in turn, re-run, confirmed the predicted failure, restored) before being combined. **Final verification (`~/tmp/xce_ha_bug1479_verify_v3`, not deleted):** `controller_unit_test` 1053/1053 PASS twice clean; live `run-ha-soak.sh --config-upgrade` PASSED end-to-end — all 3 `force_elect_node` rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7, never previously reached), final verdict "PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members." `.kiro/specs/raft-adapter-conformance/`'s concrete seed defect (missing NACK reply) is now fixed here; that spec's broader `raft.tla` action-by-action conformance audit remains valid, separate future work (fix B's own committed_lsn-vs-log-tail conflation is exactly the kind of second sibling gap that audit is meant to catch systematically).
|
||||
|
||||
- **Task 8.5 redirection (design.md's original mechanism was wrong; this is not the same as a review rejection):** the original hypothesis for bug-857's divergence — a job whose queue is removed by a config upgrade gets re-homed to `lost_and_found` on some members but not others, based on per-node restart timing — turned out to be **aspirational, never-implemented code**. Exhaustive grep across `job.c`/`queue.c`/`worker.c`/`config_repl.c` at both the pre-fix baseline commit (`e0c5e8b7`) and current HEAD confirmed: `create_lost_and_found` only ever creates the sentinel queue/host objects (and its entire body is `#ifndef XCE_TESTING`-gated); `restore_job`/`restore_job_array` never consult `qmgr`; `restore_queue_state` explicitly *skips* rather than re-homes when a queue is missing. `.wolf/buglog.json`'s bug-857 entry itself flags this: "identified via TLA+ gap analysis... NOT empirically reproduced yet." design.md's Requirements Traceability table assigned 6.2 to a live "HA soak new scenario" — also written against the same since-disproven hypothesis. Redirected (orchestrator decision, not a task-local implementer call) to a real, code-backed mechanism instead: `schedule_checkpoint`'s pre-existing, unmodified-by-this-spec WAL-record-count threshold trigger is genuinely decoupled from the config-marker `(term,lsn)` boundary, so two members' independently-timed checkpoints can capture different job/queue sets — the actual, general-purpose truth behind bug-857's claim ("no snapshot transfers an identical materialized state"). This is testable entirely within `controller_unit_test`, no live multi-process soak or old-commit worktree needed, since the threshold-trigger mechanism is still live, unmodified production code coexisting with the new marker-trigger at HEAD. Landed as `test_bug857_regression.c` (2 tests: divergence under threshold-only triggering vs. convergence under the real marker-triggered path, the latter under deliberately differing per-member processing pace to make the convergence claim non-trivial). Reviewer independently re-derived and confirmed the mechanism claim from first principles (not just re-running the tests) before approving — see `git blame`/review transcript for the full verification chain (schedule_checkpoint predates this spec via `git merge-base --is-ancestor`, zero lines of its body touched by tasks 1-7, `persist_apply_pump_run`'s strictly-sequential dispatch loop explains why the marker-triggered checkpoint's convergence isn't a timing coincidence). design.md's Requirements Traceability table (row 6.2) and Testing Strategy section were both corrected to match. **Heads-up for tasks 8.6-8.9**: they still legitimately need live multi-process HA-soak-level testing for Requirement 6.1 (chaotic multi-kill + config upgrade) — that infrastructure (a live config-upgrade CLI trigger doesn't exist anywhere in this codebase yet; confirmed via grep of `test.c`/`src/frontend`/`src/tools`) was NOT built by this task and will need to be built by whichever of 8.6-8.9 first needs to drive a live `OP_upgradeConfig` RPC against a running cluster.
|
||||
|
||||
- **Task 8.4 round-1 rejection (real gap, caught by review):** the first pass built `configcompare_paths()` (correct, well-tested against `persist.c`/`config_repl.c`'s real `workdir/name/conf/current`+`conf/cluster.bin.<hex>` resolution convention) and linked `configcompare.c` into both `xce_test_src` and `controller_unit_test_src` in meson, but never registered it as a `cmd_configcompare` subcommand in `testsuite/test.c` — unlike its siblings `walcompare`/`snapverify`/`snapdiff`, each of which has a `cmd_*` wrapper in BOTH `top_commands[]` and `client_commands[]`. Left as delivered, the tool was a tested library function with no shell entry point — unusable by task 8.6/8.9's later HA soak scenarios, which design.md explicitly says will invoke `configcompare` directly against each node's data directory. Fixed with a 25-line, `test.c`-only remediation mirroring `cmd_snapdiff`'s exact shape; verified end-to-end via `echo 'configcompare <dir1> <dir2>' | xce_test` (checked `$?` for both IDENTICAL/exit-0 and DIVERGENT/exit-1 cases, not just printed text). **Heads-up for any future task in this spec (or elsewhere) that adds a new offline testsuite tool in this family**: linking the `.c` file into the meson source lists is NOT sufficient — it must also get a `cmd_*` wrapper + `top_commands[]`/`client_commands[]` entries in `test.c`, or it silently has no way to be invoked outside the unit test binary.
|
||||
|
|
|
|||
84
.kiro/specs/raft-adapter-conformance/requirements.md
Normal file
84
.kiro/specs/raft-adapter-conformance/requirements.md
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
|
||||
Systematic conformance gap-validation of the XCE controller's Raft replication ADAPTER
|
||||
(`src/controller/ha.c`, `repl.c`, `config_repl.c` — the real wire-level send/receive/reply
|
||||
code) against `raft.tla`'s action model, action by action (AppendEntries request/response,
|
||||
RequestVote, ClientRequest, AdvanceCommitIndex, term rules). This is distinct from and
|
||||
complements the existing `raft-sim-conformance` spec, whose boundary is explicitly scoped to
|
||||
`src/lib/raft/raft_sim.c` only (an in-process, in-memory harness validating the LIBRARY's
|
||||
protocol semantics) and explicitly does not touch the controller adapter code.
|
||||
|
||||
**Motivating finding (already confirmed live — seed this spec with it as a known first
|
||||
defect rather than rediscovering it):** during a fresh live-soak re-verification of
|
||||
bug-1479's fix (`config-upgrade-snapshot` task 8.6's `--config-upgrade` scenario,
|
||||
`run-ha-soak.sh`, deploy preserved at `~/tmp/xce_ha_bug1479_verify`), the fix's new
|
||||
primary-side NACK-routing code (`ha_raft_on_repl_nack` + `repl_start_catchup` in
|
||||
`repl_ack_decode_and_dispatch`'s WAL-ACK-reply-not-success branch, `ha.c` ~6680-6720) was
|
||||
confirmed CORRECT but UNREACHABLE in production: `ha.c`'s `ha_apply_wal_catchup_entry` has
|
||||
three reject paths (~3230-3232 for the term-carrying single/batch entry via
|
||||
`ha_apply_term_entry_via_library`, ~3320-3321 for a batch sub-entry apply failure, ~3373-3374
|
||||
for the legacy single-entry path) that all explicitly send NO reply at all on a logOk reject
|
||||
/ forward-gap (not even a NACK) — and `ha_send_wal_ack_reply` (`ha.c:3084-3106`), the ONLY
|
||||
function anywhere in the controller that ever constructs a `WalAckReply`, hardcodes
|
||||
`success=1` unconditionally. There is no production code path that ever sends a failing
|
||||
`WalAckReply`. In `raft.tla`, AppendEntries always yields a response (success true or false)
|
||||
that drives nextIndex backoff; this codebase's silent-drop-on-reject deviates from that and
|
||||
reproduces bug-1479's original permanent-quorum-loss symptom via a different mechanism
|
||||
(confirmed live: `ctrl-2` standby stuck rejecting `ae_reject_logmismatch` at growing
|
||||
`prevIdx` forever, `ctrl-1` primary logs zero WAL-NACK messages the entire time,
|
||||
`committed_lsn` frozen, job submissions permanently stuck at 0/3 acked).
|
||||
|
||||
**Scope for this spec:**
|
||||
1. Fix the concrete missing-NACK-reply defect at the three reject sites so a real
|
||||
`WalAckReply{success=0}` reaches the primary and the bug-1479 NACK-routing code can
|
||||
actually engage.
|
||||
2. Beyond that one confirmed defect, perform the broader systematic pass — walk `raft.tla`'s
|
||||
action list (AppendEntries send+response, RequestVote/PreVote send+response,
|
||||
ClientRequest/log append, AdvanceCommitIndex, term/UpdateTerm handling,
|
||||
Restart/recovery) against the corresponding controller adapter code paths in
|
||||
`ha.c`/`repl.c`/`config_repl.c` and flag every other place a response, retry, or backoff
|
||||
the spec model assumes is silently missing or short-circuited — the same class of gap
|
||||
this first finding represents.
|
||||
|
||||
**Depends on / builds on:** `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`,
|
||||
`raft-sim-conformance` (consumes their library entry points and prior `validate_impl`
|
||||
`known_limitations`, several of which already flagged "documentation drift" around this
|
||||
exact send/receive boundary — cross-reference those notes rather than re-deriving them).
|
||||
|
||||
**Related bugs:** bug-1479 (this session's re-verification, `buglog.json`, occurrence 2 —
|
||||
still not fully fixed), bug-1466/1471/1472/1474/1475/1477/1478 (same HA-replication defect
|
||||
family).
|
||||
|
||||
## Status Update (2026-07-29)
|
||||
|
||||
**Scope item 1 (the concrete missing-NACK-reply defect) is FIXED**, as part of closing out
|
||||
`config-upgrade-snapshot` task 8.6 / bug-1479. `ha_apply_term_entry_via_library`'s logOk-reject
|
||||
path (the one live-reachable reject site of the three named above — the other two are
|
||||
documented dormant/unreachable with real senders, see `ha_apply_wal_catchup_entry`'s own
|
||||
comments) now sends a real `WalAckReply{success=0}` via a new `ha_send_wal_ack_reply_ex`
|
||||
helper, confirmed live: the primary now genuinely receives and routes the NACK
|
||||
(`ha_raft_on_repl_nack`, previously-correct-but-unreachable code, now actually fires).
|
||||
|
||||
Closing that gap alone was still not sufficient — it exposed a SECOND, independent adapter
|
||||
gap (also now fixed, also a `raft.tla`-deviation of exactly the kind this spec's scope item 2
|
||||
is meant to find systematically): `ha_stream_wal_to_standby`'s catch-up range was hard-bounded
|
||||
at `committed_lsn` rather than the leader's real log tail, which is a `raft.tla` deviation
|
||||
(`leaderCommit` is advisory to the follower in the real spec, never a send-cap) and caused a
|
||||
self-referential deadlock whenever the streamed-to standby's own ack was required to advance
|
||||
`committed_lsn` in the first place. Fixed via a new `ha_compute_repl_stream_ceiling_lsn` pure
|
||||
function. Full live re-verification (`run-ha-soak.sh --config-upgrade`, 3/3 `force_elect_node`
|
||||
rounds) now PASSES end-to-end. See `.wolf/buglog.json` bug-1479 and
|
||||
`config-upgrade-snapshot/tasks.md` task 8.6's Implementation Notes for the complete fix
|
||||
narrative and evidence.
|
||||
|
||||
**Scope item 2 (the broader systematic `raft.tla` action-by-action pass) remains open and
|
||||
valid** — the fact that fixing item 1 alone immediately surfaced a second, independent
|
||||
sibling gap (the committed_lsn-vs-log-tail conflation) is itself evidence this class of
|
||||
adapter/spec-model divergence is not a one-off, and a systematic pass is still worth doing
|
||||
to find any remaining siblings before they're each independently rediscovered via a live
|
||||
soak the way these two were.
|
||||
|
||||
## Requirements
|
||||
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||
25
.kiro/specs/raft-adapter-conformance/spec.json
Normal file
25
.kiro/specs/raft-adapter-conformance/spec.json
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"feature_name": "raft-adapter-conformance",
|
||||
"created_at": "2026-07-29T03:10:00Z",
|
||||
"updated_at": "2026-07-29T03:10:00Z",
|
||||
"language": "en",
|
||||
"phase": "initialized",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": [],
|
||||
"related_bugs": ["bug-1479", "bug-1466", "bug-1471", "bug-1472", "bug-1474", "bug-1475", "bug-1477", "bug-1478"],
|
||||
"related_specs": ["raft-log-replication", "raft-commit-safety", "ha-term-safety", "raft-sim-conformance", "config-upgrade-snapshot"]
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -12,6 +12,8 @@
|
|||
- Keep test logs for analysis, don't skip intermediate output even if a test hangs
|
||||
|
||||
## Key Learnings
|
||||
- (2026-07-28, bug-1479 ROOT-CAUSED not fixed) A permanent quorum-loss family member unrelated to admission control: when repl.c's `repl_broadcast` (live single-entry WAL tail-follow) gets an `ae_reject_logmismatch` NACK back from a standby (a genuinely NORMAL Raft outcome after rapid leader-election churn leaves a conflicting uncommitted tail entry at the same LSN with a different term — confirmed byte-precise via `xce_test waldump`: ctrl-2's own lsn=13 was its own deposed term=3 noop, while the canonical/surviving log has a DIFFERENT term=4 noop at the same lsn), NOTHING in ha.c's `repl_ack_decode_and_dispatch` NACK branch (`!success`) does anything but check for a higher observed term and then literally `WARN("...success=0 — ignoring")` — no `repl_on_ack` call (the ONLY site that decrements `pending_ack_count`), no nextIndex backoff-and-retry, no catch-up trigger. Consequence: `pending_ack_count` climbs 1-per-broadcast-attempt with zero decrements until `repl_broadcast`'s own `MAX_PENDING_ACKS=16` defensive eviction fires and marks the standby `in_catch_up=1` — which correctly excludes it from `repl_active_standby_count`/quorum, but NOTHING then drives an actual catch-up stream/snapshot-install for it (the only other `repl_complete_catchup` call site, config_repl.c's ACK-driven `catchup_lane_fn`, can never engage a standby that never sends a genuine ACK in the first place). Net effect: a standby with ONE conflicting log entry from election churn (routine, expected) becomes PERMANENTLY excluded from quorum with no self-healing path — in a 3-node cluster this is one unlucky election sequence away from total write-quorum loss. **General lesson**: when a symptom looks like "resource exhaustion" (admission cap, connection pool, whatever), always verify the actual counter's INCREMENT and DECREMENT sites are symmetric across EVERY reply outcome (success AND failure/reject), not just the happy path — a counter that only decrements on success but increments unconditionally will silently drift to its ceiling under ANY sustained failure condition, which looks identical to a genuine leak from the outside (repeated timeouts / repeated rejects) but has a completely different fix (handle the reject/failure path, not "find the missing release call"). Confirmed the ORIGINAL admission-slot-leak hypothesis for this bug was WRONG by checking `commit_resolve()`/`wait_release_locked()` — those DO release correctly on every timeout, 1:1 (17 timeouts, 17 releases) — before chasing the real mechanism.
|
||||
- (2026-07-28, waldump REPL usage) `xce_test` (built at `src/xce.build/controller/xce_test` or `src/build.dbg/controller/xce_test`) has no `waldump` CLI flag — `--help` only shows generic argp options. It's a REPL: pipe commands via stdin, e.g. `printf 'waldump <data_dir_or_wal_seg>\nquit\n' | ASAN_OPTIONS=detect_leaks=0 timeout 60 <path-to-xce_test>` (mirrors the `xtest()` shell helper in both `clean_failover_preservation.sh` and `run-ha-soak.sh`). Output columns are `LSN | TERM | TYPE | PLEN | PCRC | INFO` — the PLEN column is payload length, NOT a second type code (a WAL_CONFIG_MARKER shows as `UNKNOWN | 40 |` because waldump's type-name table has no case for type 50 yet — 40 there is the marker's 40-byte payload length, not its wire type number). This is a fast, byte-precise way to independently confirm/refute term or index divergence between two members' WALs without instrumenting production code.
|
||||
- (2026-07-27/28, bug-1475 FIXED) A term/version index that grows its backing array geometrically and zero-fills the newly-grown region to mean "absent" has a subtle, TIME-DEPENDENT bug: zero-filling with the same sentinel value used for a legitimate "recorded as literal 0" entry makes a deliberately-never-recorded slot (here: persist.c's per-LSN term_index, where a WAL_CHECKPOINT's own LSN is BY DESIGN never passed to record_entry_term) silently and RETROACTIVELY flip from "absent" to "present, value 0" the instant ANY later index is recorded and the array-growth watermark (`term_index_count`) crosses it — even though nothing ever explicitly wrote to that slot. Two readers that check the SAME never-recorded slot at DIFFERENT times (e.g. a sender that reads it AFTER appending later entries, vs. a receiver that reads it BEFORE appending anything past it) get DIFFERENT answers for what should be a stable, position-only fact — this produced a permanent AppendEntries logOk rejection (bug-1475: sender computed prev_log_term=1 (WAL_LEGACY_TERM), receiver computed prev_log_term=0 (absent), for the identical checkpoint LSN). **General fix pattern**: reserve a DIFFERENT sentinel (here `TERM_INDEX_UNSET = UINT64_MAX`, filled via `memset(...,0xFF,...)` on array growth) distinct from every legitimate stored value, so a never-explicitly-written slot reads "absent" PERMANENTLY, independent of later growth — never let "grown but unwritten" collide with a real, meaningful value (here, literal term 0 / WAL_LEGACY_TERM) in the same sentinel. **Do-not-repeat**: when hand-tracing (not instrumenting) a bug involving an in-memory index that has a documented "grows on demand, gaps read as absent" contract, don't trust that the gap actually STAYS absent across the FULL lifetime of the process/test — verify with real instrumentation whether "absent" is a stable property of the position or merely a snapshot of "not yet grown past." This was found only after two prior, plausible-looking production fixes (config_repl.c's tracker-advance + ha.c's JIT log_len resync, both already landed) still left the regression test RED — real fprintf instrumentation at the streamer, the JIT resync, and raft.c's logOk check (all added and fully removed afterward) pinned the exact moment prev_log_term flipped from 0 to 1. See `.wolf/buglog.json` bug-1475 for the full three-part fix and `test_bug1475_catchup_survives_midstream_checkpoint` (test_config_repl.c) for the regression proof.
|
||||
|
||||
- (2026-07-11, controller_unit_test isolated-group + leak-detection runs) `controller_unit_test` supports `--run STR` (repeatable, arg key `-t`) to select one or more TEST()-registered groups by exact function name (matches `argp`-parsed `-t`/`--run`, dispatch via `check_test()`/`TEST()` macro in `controller_test.c`) — use it to isolate a single suite instead of parsing the full-suite log. Full-suite direct runs interleave two PASS-print conventions: older suites only print via Unity's own per-assertion output (no `<file>:<line>:<name>:PASS` banner), while newer TDD suites (apply/HA-prevote family) add a custom `printf("%s: PASS\n", ...)` + banner — so grepping for `../controller/testsuite/<file>.c` banners under-counts groups that don't use the newer convention; don't infer suite pass/fail or crash-truncation from banner presence alone, always check the process exit code. Confirmed via meson (`Ok:1/Fail:0`, one lumped "controller test") AND a clean direct exit(0) with zero grep hits for AddressSanitizer/heap-buffer-overflow/UAF/double-free/FAIL across a 15.7k-line full-suite log — the log's abrupt-looking tail (last group's own internal UNITY_BEGIN/END subtotal printed twice) is normal, not truncation; verify with `echo $?` instead. The historical "test_ha_prevote 18/18" figure from an earlier session was the COMBINED total of `test_ha_prevote` (12 tests, confirmed via `grep -c RUN_TEST` in the .c file) + `test_ha_failover_prevote` (6 tests) — the two are separate TEST() groups in `controller_test.c`, not one suite of 18; report them separately (12/12 + 6/6) to avoid a false "regression" reading.
|
||||
|
|
@ -574,3 +576,10 @@ Resumed `/kiro-impl config-upgrade-snapshot` with buglog.json's bug-1476 entry c
|
|||
|
||||
## Do-Not-Repeat (2026-07-28, bug-1477 live verification) — confirming ONE bug's fix live can surface a DIFFERENT bug at a LATER step the soak never previously reached
|
||||
Re-running `run-ha-soak.sh --config-upgrade` after the bug-1477 fix: the bug's own specific symptoms (checkpoint-identity, config-identity, host-admin-status-survival assertions) all PASSED for the first time. But the SAME soak run then failed later, in the force-elect-rotation step, with repeated commit timeouts (rc=-110) on the primary — a step this soak scenario had NEVER reached before (every prior attempt failed earlier, first at bug-1471/1472's checkpoint divergence, then at bug-1477's own catch-up wedge). **Do-not-repeat**: don't assume a soak reaching further and failing differently means your fix is broken or incomplete — check whether the NEW failure point was ever exercised before; if not, it's very plausibly a separate, pre-existing, newly-exposed bug (same pattern as this whole session's bug-1471→1472→1474→1477 chain), not evidence the just-verified fix regressed something. Investigate the new failure on its own terms (what's different about THIS code path) rather than assuming causality from the just-changed code.
|
||||
|
||||
## Key Learning (2026-07-29, bug-1479 FIXED — 3-layer fix, each layer only verifiable live) — a correct adapter-side fix can be unit-green and still be dead code in production; only a live re-run proves the wire is actually connected
|
||||
bug-1479 (permanent quorum loss when a standby log-conflicts during chaotic election churn) needed THREE compounding fixes, and each of the first two looked complete after a clean unit-test run yet still failed the EXACT SAME live scenario (`run-ha-soak.sh --config-upgrade`'s `force_elect_node` loop) for a NEW reason each time:
|
||||
1. **`ha_raft_on_repl_nack`** (primary-side): routes a WAL-ACK-reply NACK into the raft library's nextIndex backoff + immediate `repl_start_catchup`. Unit-correct, 1048/1048 green — but live soak v1 still failed identically (0/3 acked), because nothing ever produced the `success=0` reply this code was waiting for.
|
||||
2. **Standby-side NACK sending** (`ha_apply_term_entry_via_library`'s logOk-reject path now calls `ha_send_wal_ack_reply_ex(..., success=0)` instead of silently dropping the reject — `ha_send_wal_ack_reply` had hardcoded `success=1` unconditionally, the ONLY WalAckReply sender in the whole controller). Unit-correct, verified via a new capture-hook test — but live soak v2 confirmed the NACK NOW reaches the primary (log-grepped: `WAL NACK from standby ... success=0`) and STILL failed identically, because `ha_stream_wal_to_standby`'s catch-up range was hard-capped at `committed_lsn`, which can never advance without this exact standby's own ack — a self-referential deadlock invisible from EITHER side's code in isolation.
|
||||
3. **`ha_compute_repl_stream_ceiling_lsn`**: widens the STREAM_WAL ceiling (and the convergence check gating it) to the leader's own log tail, matching raft.tla's real AppendEntries semantics (leaderCommit is advisory, never a send-cap) — while re-using bug-1478's exact guard (never widen onto a trailing local-only checkpoint's unstreamable LSN) so fixing gap 3 didn't reopen gap-1478. Live soak v3: PASS, all 3 force-elect rounds, 1053/1053 unit tests.
|
||||
**Do-not-repeat**: when an adapter/wire-protocol fix passes its own unit tests, that only proves the function is INTERNALLY correct — it does NOT prove anything upstream/downstream actually calls it with real data. For any fix whose job is "handle X when X arrives," always ALSO verify (via a live/integration run, log-grepped, not inferred) that X actually arrives in production BEFORE declaring victory — a unit-green "correct but unreachable" fix is functionally identical to no fix at all, and the only way to catch that is to re-run the EXACT scenario that surfaced the original bug, all the way to its real end state, every time a layer is added.
|
||||
|
|
|
|||
113
.wolf/memory.md
113
.wolf/memory.md
|
|
@ -6724,3 +6724,116 @@
|
|||
| 02:51 | Edited .kiro/specs/scheduler-core/requirements.md | modified loop() | ~535 |
|
||||
| 02:52 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | removed 6 lines | ~120 |
|
||||
| 02:53 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified used() | ~615 |
|
||||
| 02:53 | Edited .kiro/specs/scheduler-core/requirements.md | modified push() | ~1417 |
|
||||
| 02:53 | Session end: 16 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 6 reads | ~164121 tok |
|
||||
| 02:54 | Session end: 16 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 8 reads | ~184449 tok |
|
||||
| 02:55 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~213 |
|
||||
| 02:55 | Session end: 17 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 8 reads | ~184677 tok |
|
||||
| 02:57 | Session end: 17 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 9 reads | ~195415 tok |
|
||||
| 02:58 | Session end: 17 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 13 reads | ~417052 tok |
|
||||
| 03:02 | Edited .kiro/specs/scheduler-core/requirements.md | 3→3 lines | ~101 |
|
||||
| 03:02 | Edited .kiro/specs/scheduler-core/requirements.md | modified while() | ~1161 |
|
||||
| 03:05 | Edited .kiro/specs/scheduler-core/requirements.md | expanded (+13 lines) | ~424 |
|
||||
| 03:06 | Edited .kiro/specs/scheduler-core/requirements.md | modified struct() | ~106 |
|
||||
| 03:06 | Session end: 21 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~451981 tok |
|
||||
| 03:14 | Edited .kiro/specs/scheduler-core/requirements.md | expanded (+15 lines) | ~453 |
|
||||
| 03:14 | Session end: 22 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~452506 tok |
|
||||
| 05:21 | Session end: 22 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~452506 tok |
|
||||
| 05:21 | Session end: 22 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~452506 tok |
|
||||
| 06:06 | Edited src/controller/ha.h | modified exactly() | ~836 |
|
||||
| 06:07 | Edited src/controller/ha.c | added 11 condition(s) | ~856 |
|
||||
| 06:07 | Edited src/controller/myapp.c | modified if() | ~173 |
|
||||
| 06:08 | Edited src/controller/ha.c | added 2 condition(s) | ~242 |
|
||||
| 06:09 | Edited src/controller/ha.c | modified if() | ~279 |
|
||||
| 06:10 | Edited src/controller/ha.c | added 2 condition(s) | ~257 |
|
||||
| 06:11 | Edited src/controller/ha.c | added 2 condition(s) | ~414 |
|
||||
| 06:11 | Edited src/controller/ha.c | added 2 condition(s) | ~319 |
|
||||
| 06:13 | Edited src/controller/testsuite/test_config_repl.c | modified is() | ~1609 |
|
||||
| 06:14 | Edited src/controller/testsuite/test_config_repl.c | 2→6 lines | ~119 |
|
||||
| 06:16 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
|
||||
| 06:18 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
|
||||
| 06:26 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
|
||||
| 06:26 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
|
||||
| 07:06 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-config/c8633ffb-8da0-40a7-9811-48f1830c37b4/scratchpad/cross-spec-gap-analysis.md | — | ~2172 |
|
||||
| 07:07 | Edited ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-config/c8633ffb-8da0-40a7-9811-48f1830c37b4/scratchpad/cross-spec-gap-analysis.md | 3→4 lines | ~143 |
|
||||
| 07:07 | Session end: 34 writes across 9 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 21 reads | ~512802 tok |
|
||||
| 07:12 | Edited .kiro/specs/admin-config-tool/spec.json | modified 25() | ~375 |
|
||||
| 07:12 | Edited .kiro/specs/admin-config-tool/spec.json | inline fix | ~136 |
|
||||
| 07:12 | Edited .kiro/specs/admin-config-tool/tasks.md | inline fix | ~182 |
|
||||
| 07:14 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 23 reads | ~521081 tok |
|
||||
| 07:29 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
|
||||
| 07:34 | Investigated force-elect job-submit FAIL in ~/tmp/xce-ha-configupgrade2 soak; refuted admission-slot-leak hypothesis, root-caused via xce_test waldump to a permanent ae_reject_logmismatch wedge (ctrl-2 lsn=13 term=3 vs canonical term=4) + NACK-ignored/pending_ack_count-never-decremented + no catch-up trigger on eviction | .wolf/buglog.json (bug-1479), .wolf/cerebrum.md | logged, not fixed (needs-owner, design decision) | ~180000 |
|
||||
| 19:30 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
|
||||
| 19:31 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
|
||||
| 19:32 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
|
||||
| 19:41 | Edited src/include/repl.h | expanded (+10 lines) | ~204 |
|
||||
| 19:41 | Edited src/controller/repl.c | modified just() | ~218 |
|
||||
| 19:41 | Edited src/controller/repl.c | added 3 condition(s) | ~214 |
|
||||
| 19:42 | Edited src/controller/ha.h | modified ha_raft_on_repl_nack() | ~434 |
|
||||
| 19:42 | Edited src/controller/ha.c | added 3 condition(s) | ~439 |
|
||||
| 19:42 | Edited src/controller/ha.c | added error handling | ~687 |
|
||||
| 19:43 | Edited src/controller/ha.c | added 2 condition(s) | ~350 |
|
||||
| 19:45 | Edited src/controller/testsuite/test_term_safety_adapter.c | added error handling | ~1320 |
|
||||
| 19:45 | Edited src/controller/testsuite/test_term_safety_adapter.c | 2→6 lines | ~106 |
|
||||
| 19:46 | Edited src/controller/testsuite/test_term_safety_adapter.c | modified test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes() | ~670 |
|
||||
| 19:47 | Edited src/controller/testsuite/test_term_safety_adapter.c | 2→3 lines | ~59 |
|
||||
|
||||
## Session: 2026-07-29 19:52
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 19:55 | Edited src/controller/testsuite/test_term_safety_adapter.c | raft_init() → transition() | ~173 |
|
||||
| 19:55 | Edited src/controller/testsuite/test_term_safety_adapter.c | 6→9 lines | ~110 |
|
||||
| 19:55 | Edited src/controller/testsuite/test_term_safety_adapter.c | 11→15 lines | ~181 |
|
||||
| 20:01 | Session end: 3 writes across 1 files (test_term_safety_adapter.c) | 4 reads | ~201524 tok |
|
||||
| 20:05 | Session end: 3 writes across 1 files (test_term_safety_adapter.c) | 4 reads | ~201524 tok |
|
||||
| 20:09 | Session end: 3 writes across 1 files (test_term_safety_adapter.c) | 4 reads | ~201524 tok |
|
||||
| 20:11 | Edited .kiro/specs/scheduler-core/requirements.md | added 1 condition(s) | ~1739 |
|
||||
| 20:11 | Session end: 4 writes across 2 files (test_term_safety_adapter.c, requirements.md) | 7 reads | ~262249 tok |
|
||||
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 7→7 lines | ~167 |
|
||||
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 3→3 lines | ~79 |
|
||||
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 12 → 13 | ~9 |
|
||||
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 11→11 lines | ~457 |
|
||||
| 20:14 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~159 |
|
||||
| 20:14 | Edited .kiro/specs/scheduler-core/requirements.md | 9→9 lines | ~344 |
|
||||
| 20:14 | Edited .kiro/specs/scheduler-core/requirements.md | 17→19 lines | ~731 |
|
||||
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | 19 → 21 | ~9 |
|
||||
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~18 |
|
||||
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | 16 → 18 | ~22 |
|
||||
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~45 |
|
||||
| 20:16 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~33 |
|
||||
| 20:16 | Edited .kiro/specs/scheduler-core/requirements.md | 19 → 21 | ~25 |
|
||||
| 20:16 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~30 |
|
||||
| 20:17 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~6 |
|
||||
| 20:20 | Session end: 19 writes across 2 files (test_term_safety_adapter.c, requirements.md) | 7 reads | ~266129 tok |
|
||||
| 20:22 | Session end: 19 writes across 2 files (test_term_safety_adapter.c, requirements.md) | 7 reads | ~266129 tok |
|
||||
| 20:29 | Edited src/controller/ha.h | modified XCE_TESTING() | ~548 |
|
||||
| 20:29 | Edited src/controller/ha.c | added 1 condition(s) | ~450 |
|
||||
| 20:29 | Edited src/controller/ha.c | expanded (+10 lines) | ~227 |
|
||||
| 20:30 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~950 |
|
||||
| 20:30 | Edited src/controller/testsuite/test_config_repl.c | 3→4 lines | ~64 |
|
||||
| 20:31 | Edited src/controller/ha.c | 3→4 lines | ~35 |
|
||||
| 20:32 | Edited src/controller/ha.c | 4→3 lines | ~21 |
|
||||
| 20:33 | Created .kiro/specs/raft-adapter-conformance/spec.json | — | ~209 |
|
||||
| 20:33 | Created .kiro/specs/raft-adapter-conformance/requirements.md | — | ~986 |
|
||||
| 20:34 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 2→2 lines | ~793 |
|
||||
| 20:35 | Session end: 29 writes across 7 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 13 reads | ~401510 tok |
|
||||
| 20:35 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | added error handling | ~858 |
|
||||
| 20:35 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 3→4 lines | ~31 |
|
||||
| 20:37 | Session end: 31 writes across 7 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 13 reads | ~402460 tok |
|
||||
| 20:51 | Edited src/controller/ha.h | modified ha_compute_repl_stream_ceiling_lsn() | ~512 |
|
||||
| 20:51 | Edited src/controller/ha.c | added 1 condition(s) | ~265 |
|
||||
| 20:52 | Edited src/controller/ha.c | added 1 condition(s) | ~291 |
|
||||
| 20:54 | Edited src/controller/ha.c | modified XCE_TESTING() | ~563 |
|
||||
| 20:54 | Edited src/controller/ha.h | expanded (+7 lines) | ~181 |
|
||||
| 20:55 | Edited src/controller/testsuite/test_ha_role.c | added error handling | ~870 |
|
||||
| 20:55 | Edited src/controller/testsuite/test_ha_role.c | modified 1479() | ~134 |
|
||||
| 20:55 | Edited src/controller/ha.c | modified ha_compute_repl_stream_ceiling_lsn() | ~110 |
|
||||
| 20:58 | Edited src/controller/ha.c | added 1 condition(s) | ~105 |
|
||||
| 21:01 | Session end: 40 writes across 8 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 17 reads | ~461541 tok |
|
||||
| 21:03 | Session end: 40 writes across 8 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 17 reads | ~461541 tok |
|
||||
| 21:06 | bug-1479 FULLY FIXED (3-layer: primary NACK routing + standby NACK sending + stream-ceiling widening) | ha.c, ha.h, repl.c, repl.h, test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c | 1053/1053 tests + live soak PASS (3/3 force-elect rounds) | ~180000 |
|
||||
| 21:08 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 7→7 lines | ~277 |
|
||||
| 21:08 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | added error handling | ~1047 |
|
||||
| 21:09 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 23→22 lines | ~135 |
|
||||
| 21:10 | Edited .kiro/specs/raft-adapter-conformance/requirements.md | added error handling | ~567 |
|
||||
|
|
|
|||
|
|
@ -3081,19 +3081,36 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to
|
||||
* the primary over ep. Skipped when ep or c is NULL (test/offline path).
|
||||
* Task 3.1 (ha-wal-replication): ACK each successfully applied WAL entry. */
|
||||
static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
|
||||
cluster_t *c, uint64_t lsn)
|
||||
#ifdef XCE_TESTING
|
||||
void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success) = NULL;
|
||||
#endif
|
||||
|
||||
/* ha_send_wal_ack_reply_ex: send WalAckReply {term, lsn, success} back to
|
||||
* the primary over ep. Skipped when ep or c is NULL (test/offline path),
|
||||
* unless ha_wal_ack_send_hook is set (XCE_TESTING), in which case the hook is
|
||||
* called with the built {term, lsn, success} INSTEAD of sending over ep --
|
||||
* lets a unit test observe a NACK (success=0) without a live endpoint
|
||||
* (bug-1479). Task 3.1 (ha-wal-replication): ACK/NACK each applied/rejected
|
||||
* WAL entry. */
|
||||
static void ha_send_wal_ack_reply_ex(ha_state_t *ha, struct endpoint_t *ep,
|
||||
cluster_t *c, uint64_t lsn, int success)
|
||||
{
|
||||
uint64_t term = (uint64_t)atomic_load_explicit(&ha->term, memory_order_acquire);
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
if (ha_wal_ack_send_hook) {
|
||||
ha_wal_ack_send_hook(term, lsn, success);
|
||||
return;
|
||||
}
|
||||
#endif
|
||||
|
||||
if (!ep || !c)
|
||||
return;
|
||||
|
||||
wal_ack_reply_t ack;
|
||||
ack.term = (uint64_t)atomic_load_explicit(&ha->term, memory_order_acquire);
|
||||
ack.term = term;
|
||||
ack.lsn = lsn;
|
||||
ack.success = 1;
|
||||
ack.success = success ? 1 : 0;
|
||||
|
||||
xce_reply_t reply;
|
||||
memset(&reply, 0, sizeof(reply));
|
||||
|
|
@ -3105,6 +3122,14 @@ static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
|
|||
xce_reply(ep, c->myhost, &reply);
|
||||
}
|
||||
|
||||
/* ha_send_wal_ack_reply: send a successful WalAckReply {term, lsn,
|
||||
* success=1} -- the common case, every existing call site. */
|
||||
static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
|
||||
cluster_t *c, uint64_t lsn)
|
||||
{
|
||||
ha_send_wal_ack_reply_ex(ha, ep, c, lsn, /*success*/1);
|
||||
}
|
||||
|
||||
/* ha_apply_term_entry_via_library: route ONE term-carrying replicated entry
|
||||
* (a live broadcast or a catch-up sub-entry) through the raft library
|
||||
* (raft_on_append_entries_request via ha_consume_append_entries) so the LIBRARY
|
||||
|
|
@ -3227,8 +3252,18 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
|
|||
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
|
||||
return 1;
|
||||
}
|
||||
/* logOk reject (e.g. forward gap before the prefix arrives): no ACK; the
|
||||
* ACK-driven catch-up continuation re-streams from current_lsn in order. */
|
||||
/* logOk reject (e.g. forward gap before the prefix arrives, or a genuine
|
||||
* (term,lsn) conflict): bug-1479 -- send a NACK (success=0) so the
|
||||
* primary's ha_raft_on_repl_nack can back off nextIndex and immediately
|
||||
* mark this standby for catch-up, matching raft.tla's AppendEntries
|
||||
* action (the receiver ALWAYS replies, success = logOk). Before this fix
|
||||
* NOTHING was sent on reject: the primary's NACK-handling branch
|
||||
* (repl_ack_decode_and_dispatch) was correct but unreachable, since no
|
||||
* production path ever produced the success=0 reply it was waiting for
|
||||
* -- the standby would loop rejecting the identical re-broadcast entry
|
||||
* forever, discovered live via the config-upgrade soak's force-elect
|
||||
* rotation. */
|
||||
ha_send_wal_ack_reply_ex(ha, ep, c, e->lsn, /*success*/0);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -3843,6 +3878,20 @@ uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
|
|||
return committed_lsn;
|
||||
}
|
||||
|
||||
/* bug-1479 (gap 3): see full doc comment in ha.h. Distinct from
|
||||
* ha_compute_repl_target_lsn above (the CONVERGENCE target) -- this is the
|
||||
* STREAM ceiling: how far a leader may actually push WAL to a standby, which
|
||||
* must extend past committed_lsn or a standby whose own ack is required to
|
||||
* advance committed_lsn can never receive the entries it needs to ack. */
|
||||
uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
|
||||
uint64_t last_written,
|
||||
uint64_t checkpoint_lsn)
|
||||
{
|
||||
if (last_written == checkpoint_lsn && last_written > committed_lsn)
|
||||
last_written = committed_lsn;
|
||||
return (last_written > committed_lsn) ? last_written : committed_lsn;
|
||||
}
|
||||
|
||||
/* ───────────────────────────────────────────────────────────────────────
|
||||
* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198).
|
||||
*
|
||||
|
|
@ -4233,8 +4282,38 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
|
|||
uint64_t standby_tag = ha->standby_reported_tag[peer_idx];
|
||||
uint64_t standby_lsn = ha->standby_reported_lsn[peer_idx];
|
||||
|
||||
/* bug-1479: a standby marked in_catch_up (via ha_raft_on_repl_nack's
|
||||
* immediate trigger on log-conflict rejection, or repl_broadcast's
|
||||
* MAX_PENDING_ACKS eviction) can never correct its OWN self-reported
|
||||
* standby_reported_lsn — it's stuck rejecting every AppendEntries, so it
|
||||
* never sends a stateReport reflecting genuine progress. Use the
|
||||
* corrected join_applied_lsn instead: this heartbeat-driven cycle (runs
|
||||
* every heartbeat_interval_ms regardless of ACK/reject outcome) becomes
|
||||
* the actual catch-up driver for this case — the ACK-driven continuation
|
||||
* (ha_walack_on_ack, config_repl.c) can only fire on a REAL ack, which a
|
||||
* log-conflicted standby can never produce until streamed from the
|
||||
* corrected position at least once. */
|
||||
if (c) {
|
||||
uint64_t join_lsn;
|
||||
if (repl_get_catchup_join_lsn(&c->persist.repl, ep, &join_lsn))
|
||||
standby_lsn = join_lsn;
|
||||
}
|
||||
|
||||
/* bug-1479 (gap 3): stream (and evaluate convergence against) the
|
||||
* leader's own log tail, not just committed_lsn -- see
|
||||
* ha_compute_repl_stream_ceiling_lsn's doc comment (ha.h) for why capping
|
||||
* at committed_lsn is a permanent deadlock whenever this exact standby's
|
||||
* own ack is required to advance committed_lsn in the first place. */
|
||||
uint64_t stream_ceiling_lsn = committed_lsn;
|
||||
if (c) {
|
||||
uint64_t last_written = (c->persist.current_lsn > 0)
|
||||
? c->persist.current_lsn - 1 : 0;
|
||||
stream_ceiling_lsn = ha_compute_repl_stream_ceiling_lsn(
|
||||
committed_lsn, last_written, c->persist.checkpoint_lsn);
|
||||
}
|
||||
|
||||
ha_repl_action_t action = ha_evaluate_repl_action(standby_tag, standby_lsn,
|
||||
committed_tag, committed_lsn,
|
||||
committed_tag, stream_ceiling_lsn,
|
||||
oldest_wal_lsn);
|
||||
|
||||
switch (action) {
|
||||
|
|
@ -4283,16 +4362,19 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
|
|||
case HA_REPL_STREAM_WAL:
|
||||
/* Task 3.4: batch WAL streaming.
|
||||
* Enqueues all batch OPCode_walCatchupEntry messages for the range
|
||||
* [standby_lsn+1, committed_lsn] via ha_stream_wal_to_standby().
|
||||
* [standby_lsn+1, stream_ceiling_lsn] via ha_stream_wal_to_standby()
|
||||
* (bug-1479 gap 3: the ceiling extends past committed_lsn to the
|
||||
* leader's own log tail — see ha_compute_repl_stream_ceiling_lsn).
|
||||
* All xce_request() calls are non-blocking (enqueue-only). */
|
||||
INFO("HA repl: standby[%d] lsn=%" PRIu64
|
||||
" < committed_lsn=%" PRIu64 " → stream WAL batches",
|
||||
peer_idx, standby_lsn, committed_lsn);
|
||||
" < stream_ceiling_lsn=%" PRIu64 " (committed_lsn=%" PRIu64
|
||||
") → stream WAL batches",
|
||||
peer_idx, standby_lsn, stream_ceiling_lsn, committed_lsn);
|
||||
#ifdef XCE_TESTING
|
||||
if (ha_repl_stream_wal_hook != NULL) {
|
||||
const char *wal_path = (c && c->persist.wal_path[0])
|
||||
? c->persist.wal_path : "";
|
||||
ha_repl_stream_wal_hook(ep, wal_path, standby_lsn, committed_lsn);
|
||||
ha_repl_stream_wal_hook(ep, wal_path, standby_lsn, stream_ceiling_lsn);
|
||||
break;
|
||||
}
|
||||
#endif
|
||||
|
|
@ -4311,7 +4393,7 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
|
|||
uint64_t leader_term =
|
||||
atomic_load_explicit(&c->ha.term, memory_order_acquire);
|
||||
ha_stream_wal_to_standby(ep, c->persist.wal_path, &c->persist,
|
||||
leader_term, standby_lsn, committed_lsn);
|
||||
leader_term, standby_lsn, stream_ceiling_lsn);
|
||||
}
|
||||
break;
|
||||
|
||||
|
|
@ -4820,6 +4902,38 @@ int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
|
|||
return 0;
|
||||
}
|
||||
|
||||
int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
|
||||
uint64_t *corrected_lsn_out)
|
||||
{
|
||||
if (!c)
|
||||
return -EINVAL;
|
||||
|
||||
/* Same defense-in-depth bounds/self check as ha_raft_on_repl_ack. */
|
||||
int self_idx = c->ha.my_priority;
|
||||
int n_peers = c->ha.raft.n_peers;
|
||||
if (abs_peer_idx < 0 || abs_peer_idx >= n_peers || abs_peer_idx == self_idx) {
|
||||
ERROR("ha_raft_on_repl_nack: rejecting out-of-range peer index %d "
|
||||
"(n_peers=%d self_idx=%d) — skipping library NACK accounting",
|
||||
abs_peer_idx, n_peers, self_idx);
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
/* A rejected AppendEntries reports a log conflict at the sent
|
||||
* prevLogIndex/prevLogTerm. match_index is unused on the failure branch
|
||||
* (raft.c: "failure: next_index[peer] = Max(next_index[peer]-1, 1). Does
|
||||
* NOT advance commit.") -- pass 0. The library applies the backoff only
|
||||
* when reply_term == s->term (stale-term drop), same as the ack path. */
|
||||
(void)raft_on_append_entries_reply(&c->ha.raft, abs_peer_idx, /*success*/0,
|
||||
/*match_index unused*/0, reply_term);
|
||||
|
||||
if (corrected_lsn_out) {
|
||||
uint64_t next = c->ha.raft.next_index[abs_peer_idx];
|
||||
*corrected_lsn_out = (next > 0) ? next - 1 : 0;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_arm_own_term_gate: append the own-term WAL_NOOP and arm the commit
|
||||
* engine's writable_lsn gate (ha-single-leader-role task 2.2, Req 1.4/2.1/2.2).
|
||||
|
|
@ -6638,8 +6752,36 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
|
|||
pthread_mutex_lock(&ha->raft_lock);
|
||||
(void)raft_on_higher_term_observed(&ha->raft, reply_term);
|
||||
pthread_mutex_unlock(&ha->raft_lock);
|
||||
|
||||
/* bug-1479: a same-or-lower-term NACK means an ordinary Raft
|
||||
* log conflict (ae_reject_logmismatch), NOT a stale-term
|
||||
* primary — the case above doesn't cover it. Before this fix
|
||||
* the reject was silently dropped: the library's own,
|
||||
* already-correct nextIndex backoff (raft_on_append_entries_
|
||||
* reply) was never invoked, so the primary kept re-
|
||||
* broadcasting NEW entries assuming agreement at an index
|
||||
* the standby had already rejected, forever — the standby
|
||||
* would eventually get silently evicted via
|
||||
* MAX_PENDING_ACKS, but nothing would ever stream it back
|
||||
* into sync (repl_start_catchup's own driver, the ACK-driven
|
||||
* continuation, needs a real ACK to fire, and a log-
|
||||
* conflicted standby can never send one while stuck).
|
||||
* Immediately (not after 16 silent failures) back off via
|
||||
* the library and mark catch-up at the corrected position —
|
||||
* ha_dispatch_repl_action_for_standby's heartbeat-driven
|
||||
* decision (below) is what actually streams to it. */
|
||||
if (c) {
|
||||
int ack_abs_idx = ha_repl_ack_abs_peer_idx(
|
||||
&c->ha, /*ep_is_vote*/ vctx->ctx != NULL, vctx->peer_idx);
|
||||
uint64_t corrected_lsn = 0;
|
||||
if (ha_raft_on_repl_nack(c, ack_abs_idx, reply_term,
|
||||
&corrected_lsn) == 0) {
|
||||
repl_start_catchup(&c->persist.repl, ep, corrected_lsn);
|
||||
}
|
||||
}
|
||||
WARN("repl_ack_decode_and_dispatch: WAL NACK from standby"
|
||||
" lsn=%lu success=0 — ignoring", (unsigned long)acked_lsn);
|
||||
" lsn=%lu success=0 — marking catch-up",
|
||||
(unsigned long)acked_lsn);
|
||||
}
|
||||
} else if (rep_s.data_which == RequestReply_data_replconfigpushreplymsg) {
|
||||
/* Config-push ACK from standby (ha-commit-barrier-rollout task
|
||||
|
|
|
|||
|
|
@ -596,6 +596,32 @@ int ha_primary_log_append(cluster_t *c, uint32_t type,
|
|||
int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
|
||||
uint64_t reply_term);
|
||||
|
||||
/*
|
||||
* ha_raft_on_repl_nack (bug-1479): drive the library on a REJECTED
|
||||
* AppendEntries (WalAckReply.success == 0, i.e. the standby's log conflicts
|
||||
* at the sent prevLogIndex/prevLogTerm). Routes the rejection through
|
||||
* raft_on_append_entries_reply(success=0), which backs off
|
||||
* next_index[abs_peer_idx] by one (raft.tla's AppendEntries failure branch,
|
||||
* floored at 1) -- mirrors ha_raft_on_repl_ack's bounds-checked peer-index
|
||||
* resolution exactly, but does NOT call raft_advance_commit (a rejection
|
||||
* can never advance commit).
|
||||
*
|
||||
* Before this fix, a NACK was only ever used for stale-term discovery
|
||||
* (raft_on_higher_term_observed) and otherwise silently dropped -- the
|
||||
* library's own, already-correct nextIndex backoff mechanism was never
|
||||
* invoked, so a standby stuck in a genuine log conflict (an ordinary
|
||||
* post-election occurrence) never had its resend position corrected: the
|
||||
* primary kept broadcasting new entries assuming agreement at an index the
|
||||
* standby had already rejected, forever.
|
||||
*
|
||||
* Returns the corrected resume LSN (next_index[abs_peer_idx] - 1, i.e. the
|
||||
* position the standby is now believed caught up through) via
|
||||
* *corrected_lsn_out on success; returns -EINVAL (leaving *corrected_lsn_out
|
||||
* untouched) on a no-cluster context or a rejected peer index, matching
|
||||
* ha_raft_on_repl_ack's defense-in-depth bounds check. */
|
||||
int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
|
||||
uint64_t *corrected_lsn_out);
|
||||
|
||||
/*
|
||||
* ha_repl_ack_abs_peer_idx (ha-term-safety task 3.3 remediation): resolve the
|
||||
* ABSOLUTE library peer index from an ACK endpoint's stored peer_idx, branching
|
||||
|
|
@ -917,15 +943,36 @@ int ha_schedule_phase2_retry(cluster_t *c);
|
|||
* On success (rc == 0): applied_lsn advances to entry->lsn+1 and a
|
||||
* WalAckReply {term, lsn, success=1} is sent back to the primary via
|
||||
* xce_reply(ep, c->myhost, &reply) (Task 3.1, Req 3.1).
|
||||
* On error (rc < 0): logs ERROR, returns error; no ACK sent (Task 3.2).
|
||||
* On a genuine apply/persist ERROR (rc < 0): logs ERROR, returns error; no ACK
|
||||
* sent at all (Task 3.2) -- there is nothing coherent to report back yet.
|
||||
* On a term-carrying entry's AppendEntries REJECT (logOk fails -- forward gap
|
||||
* or a genuine (term,lsn) conflict; rc == 0, entry not accepted): bug-1479 --
|
||||
* a WalAckReply {term, lsn, success=0} (a NACK) IS now sent, so the primary's
|
||||
* ha_raft_on_repl_nack can back off nextIndex and immediately mark this
|
||||
* standby for catch-up. Before this fix the reject was dropped with no reply
|
||||
* at all, so the primary's NACK-handling could never engage no matter how
|
||||
* correct it was -- the standby would loop rejecting the identical
|
||||
* re-broadcast entry forever.
|
||||
* applied_lsn is in-memory only — not persisted to disk.
|
||||
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK is skipped.
|
||||
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK/NACK is skipped
|
||||
* (or captured via ha_wal_ack_send_hook when set).
|
||||
* Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
|
||||
*/
|
||||
int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
||||
wal_catchup_entry_t *entry,
|
||||
struct endpoint_t *ep, cluster_t *c);
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/*
|
||||
* ha_wal_ack_send_hook: test-only seam for the outbound WalAckReply built by
|
||||
* ha_send_wal_ack_reply_ex (bug-1479). When non-NULL, called with
|
||||
* (term, lsn, success) INSTEAD OF building/sending the real reply over ep --
|
||||
* lets a unit test assert that an AppendEntries reject now sends a NACK
|
||||
* (success=0), without needing a live endpoint. NULL in production.
|
||||
*/
|
||||
extern void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success);
|
||||
#endif
|
||||
|
||||
/* ha_nop_reply_state_report — populate a heartbeat reply with stateReport.
|
||||
* Fills *sr from ha->applied_lsn (atomic read) and myconf->config_version_tag,
|
||||
* then sets reply_out->data_which = stateReportMsg and
|
||||
|
|
@ -1195,6 +1242,36 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
|
|||
uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
|
||||
uint64_t last_written);
|
||||
|
||||
/*
|
||||
* ha_compute_repl_stream_ceiling_lsn (bug-1479, gap 3): the upper bound of
|
||||
* what a leader may actually STREAM to a standby via ha_stream_wal_to_standby
|
||||
* — distinct from ha_compute_repl_target_lsn's CONVERGENCE target above.
|
||||
*
|
||||
* Bounding the stream range at committed_lsn is a permanent deadlock whenever
|
||||
* the very standby being streamed to must ACK before committed_lsn can
|
||||
* advance at all (e.g. a 3-node cluster with one member down and this
|
||||
* standby log-conflicted after a chaotic election): it can never receive the
|
||||
* entries it needs to ack to unblock the majority. raft.tla's real
|
||||
* AppendEntries replicates the leader's FULL log — leaderCommit is advisory
|
||||
* to the follower, never a cap on what gets sent. Streaming past commit is
|
||||
* always safe: a later leadership change's own (term,lsn) conflict-truncate
|
||||
* (ha_consume_append_entries) correctly unwinds anything that turns out not
|
||||
* to have been committed.
|
||||
*
|
||||
* bug-1478 guard: never widen the ceiling to exactly the trailing local-only
|
||||
* checkpoint's own consumed LSN slot — no standby's applied_lsn can ever
|
||||
* legitimately reach that value (WAL_CHECKPOINT is never streamed or applied
|
||||
* by any standby, by design), so doing so would reintroduce bug-1478's
|
||||
* permanent-non-convergence trap. Falling back to committed_lsn for just
|
||||
* that transient instant is safe — the very next real entry appended after
|
||||
* the checkpoint widens the ceiling normally again.
|
||||
*
|
||||
* Pure, side-effect-free — safe to unit test directly.
|
||||
*/
|
||||
uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
|
||||
uint64_t last_written,
|
||||
uint64_t checkpoint_lsn);
|
||||
|
||||
/* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */
|
||||
|
||||
/*
|
||||
|
|
@ -1864,8 +1941,15 @@ uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base);
|
|||
* - HA_REPL_PUSH_CONFIG: In production builds, calls push_config_to_standbys(c, …).
|
||||
* In XCE_TESTING builds, calls ha_repl_push_config_for_standby_hook(c, peer_idx).
|
||||
* - HA_REPL_STREAM_WAL: calls ha_stream_wal_to_standby(ep, wal_path, standby_lsn,
|
||||
* committed_lsn) to batch-stream WAL entries via xce_request() (non-blocking).
|
||||
* In XCE_TESTING builds, calls ha_repl_stream_wal_hook if non-NULL.
|
||||
* stream_ceiling_lsn) to batch-stream WAL entries via xce_request()
|
||||
* (non-blocking). stream_ceiling_lsn (bug-1479 gap 3) is
|
||||
* ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written,
|
||||
* checkpoint_lsn) when c is non-NULL — the leader's own log tail, NOT
|
||||
* just committed_lsn, so a standby whose own ack is required to advance
|
||||
* committed_lsn can still receive the entries it needs to ack. Falls
|
||||
* back to committed_lsn when c is NULL (test fixtures with no real
|
||||
* persist_t). In XCE_TESTING builds, calls ha_repl_stream_wal_hook if
|
||||
* non-NULL.
|
||||
* - HA_REPL_SEND_SNAPSHOT: calls ha_send_snapshot_to_standby(ep, snap_path, lsn).
|
||||
* In XCE_TESTING builds, calls ha_repl_send_snapshot_hook(peer_idx) if non-NULL.
|
||||
* - HA_REPL_CONVERGED: no-op.
|
||||
|
|
|
|||
|
|
@ -775,6 +775,12 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
|
|||
if (r->standbys[i].ep == ep) {
|
||||
atomic_store_explicit(&r->standbys[i].in_catch_up, 0,
|
||||
memory_order_release);
|
||||
/* bug-1479: reset the eviction counter on re-entry to live
|
||||
* broadcast -- otherwise a stale pending_ack_count carried over
|
||||
* from before catch-up could immediately re-trip
|
||||
* repl_broadcast's MAX_PENDING_ACKS eviction on the very next
|
||||
* send, undoing the catch-up this function just completed. */
|
||||
r->standbys[i].pending_ack_count = 0;
|
||||
changed = 1;
|
||||
break;
|
||||
}
|
||||
|
|
@ -794,6 +800,28 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
|
|||
pthread_mutex_unlock(&r->ack_lock);
|
||||
}
|
||||
|
||||
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
|
||||
uint64_t *join_applied_lsn_out)
|
||||
{
|
||||
if (r == NULL || join_applied_lsn_out == NULL)
|
||||
return 0;
|
||||
|
||||
int found = 0;
|
||||
mylock_lock(&r->lock);
|
||||
for (int i = 0; i < MAX_STANDBYS; i++) {
|
||||
if (r->standbys[i].ep == ep) {
|
||||
if (atomic_load_explicit(&r->standbys[i].in_catch_up,
|
||||
memory_order_acquire)) {
|
||||
*join_applied_lsn_out = r->standbys[i].join_applied_lsn;
|
||||
found = 1;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
mylock_unlock(&r->lock);
|
||||
return found;
|
||||
}
|
||||
|
||||
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn,
|
||||
uint64_t oldest_wal_lsn)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -3922,6 +3922,93 @@ static void test_wal_catchup_entry_error_skipped_no_crash(void)
|
|||
printf("test_wal_catchup_entry_error_skipped_no_crash: PASS\n");
|
||||
}
|
||||
|
||||
/* bug-1479: a term-carrying entry that fails logOk (forward gap / genuine
|
||||
* (term,lsn) conflict) must now send a NACK {success=0} back to the primary,
|
||||
* not silence the reject entirely -- otherwise the primary's NACK-handling
|
||||
* (ha_raft_on_repl_nack, ha.c) can never engage, no matter how correct it is,
|
||||
* because nothing ever produces the reply it is waiting for. This is exactly
|
||||
* the live gap the config-upgrade soak's force-elect rotation exposed: a
|
||||
* standby that fell behind (a genuine forward gap against its empty log)
|
||||
* looped rejecting the identical re-broadcast entry forever, with no NACK
|
||||
* ever reaching the primary to trigger backoff + catch-up.
|
||||
*
|
||||
* ha_wal_ack_send_hook (ha.h, XCE_TESTING) captures the {term, lsn, success}
|
||||
* that ha_send_wal_ack_reply_ex would have sent, without a live endpoint. */
|
||||
static uint64_t g_b1479_hook_term;
|
||||
static uint64_t g_b1479_hook_lsn;
|
||||
static int g_b1479_hook_success;
|
||||
static int g_b1479_hook_called;
|
||||
|
||||
static void b1479_wal_ack_capture(uint64_t term, uint64_t lsn, int success)
|
||||
{
|
||||
g_b1479_hook_called = 1;
|
||||
g_b1479_hook_term = term;
|
||||
g_b1479_hook_lsn = lsn;
|
||||
g_b1479_hook_success = success;
|
||||
}
|
||||
|
||||
static void test_bug1479_ae_reject_sends_nack_not_silence(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_cr_b1479_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
cluster_t *c = t42_make_cluster(tmp);
|
||||
TEST_ASSERT_NOT_NULL(c);
|
||||
|
||||
/* Bring the node up as a raft FOLLOWER at term 3 with an EMPTY log
|
||||
* (log_len=0) -- mirrors t_bug1475_make_follower further below in this
|
||||
* file (that fixture is defined later; duplicated inline here to avoid
|
||||
* an ordering dependency). */
|
||||
atomic_store(&c->ha.role, CTRL_STANDBY);
|
||||
atomic_store(&c->ha.term, 3);
|
||||
ha_init_raft(&c->ha, c, /*n_peers*/3);
|
||||
c->ha.raft.term = 3;
|
||||
c->ha.raft.role = RAFT_FOLLOWER;
|
||||
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len);
|
||||
|
||||
g_b1479_hook_called = 0;
|
||||
ha_wal_ack_send_hook = b1479_wal_ack_capture;
|
||||
|
||||
/* A term-carrying entry at lsn=5 whose prev_log_index=4 is a forward gap
|
||||
* against this node's empty log (0 < 4) -- logOk MUST reject it. */
|
||||
wal_catchup_entry_t inner;
|
||||
memset(&inner, 0, sizeof(inner));
|
||||
inner.lsn = 5;
|
||||
inner.term = 3;
|
||||
inner.prev_log_index = 4;
|
||||
inner.prev_log_term = 3;
|
||||
inner.entry_type = (uint8_t)WAL_JOB_UPDATE;
|
||||
|
||||
wal_catchup_entry_t *sub_arr[1] = { &inner };
|
||||
wal_catchup_entry_t outer;
|
||||
memset(&outer, 0, sizeof(outer));
|
||||
outer.n_entries = 1;
|
||||
outer.entries = sub_arr;
|
||||
|
||||
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &outer,
|
||||
/*ep*/NULL, c);
|
||||
|
||||
ha_wal_ack_send_hook = NULL; /* uninstall before any assertion can fail */
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, rc); /* reject, not a hard error */
|
||||
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len); /* nothing was accepted */
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(g_b1479_hook_called,
|
||||
"bug-1479: an AppendEntries reject must send a WalAckReply "
|
||||
"(a NACK), not silence it entirely");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_b1479_hook_success,
|
||||
"the reply for a rejected entry must carry success=0");
|
||||
TEST_ASSERT_EQUAL_UINT64(3, g_b1479_hook_term);
|
||||
TEST_ASSERT_EQUAL_UINT64(5, g_b1479_hook_lsn);
|
||||
|
||||
t42_teardown(c);
|
||||
char cmd[512];
|
||||
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
|
||||
system(cmd);
|
||||
printf("test_bug1479_ae_reject_sends_nack_not_silence: PASS\n");
|
||||
}
|
||||
|
||||
/* Test 2.1-C: NULL entry is handled gracefully (null guard) */
|
||||
static void test_wal_catchup_entry_null_entry_no_crash(void)
|
||||
{
|
||||
|
|
@ -6764,6 +6851,7 @@ void test_config_repl(void)
|
|||
/* Task 2.1: OPCode_walCatchupEntry handler */
|
||||
RUN_TEST(test_wal_catchup_entry_advances_applied_lsn);
|
||||
RUN_TEST(test_wal_catchup_entry_error_skipped_no_crash);
|
||||
RUN_TEST(test_bug1479_ae_reject_sends_nack_not_silence);
|
||||
RUN_TEST(test_wal_catchup_entry_null_entry_no_crash);
|
||||
/* Task 2.2: batch WAL catch-up entries */
|
||||
RUN_TEST(test_wal_catchup_batch_advances_applied_lsn);
|
||||
|
|
|
|||
|
|
@ -718,6 +718,61 @@ static void test_compute_repl_target_lsn_committed_already_ahead(void)
|
|||
printf("test_compute_repl_target_lsn_committed_already_ahead: PASS\n");
|
||||
}
|
||||
|
||||
/* ── bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn ─────────────────
|
||||
*
|
||||
* Distinct from ha_compute_repl_target_lsn above (the CONVERGENCE target,
|
||||
* bootstrap-only fallback). This is the STREAM ceiling: a live 3-node HA
|
||||
* soak (config-upgrade-snapshot task 8.6, force-elect round) caught
|
||||
* committed_lsn-bounded streaming in a permanent deadlock -- with one member
|
||||
* down and the other standby log-conflicted after a chaotic election, that
|
||||
* standby's own ack is the ONLY path to a new majority, but it can never
|
||||
* receive any entry past committed_lsn to ack in the first place, since
|
||||
* ha_stream_wal_to_standby refuses to send anything beyond it. The ceiling
|
||||
* must extend to the leader's real log tail so catch-up can make progress
|
||||
* past a stalled commit point, while still excluding a trailing local-only
|
||||
* checkpoint LSN (the exact bug-1478 trap) since no standby can ever
|
||||
* legitimately ack that exact slot either. */
|
||||
|
||||
static void test_compute_repl_stream_ceiling_lsn_widens_past_committed(void)
|
||||
{
|
||||
/* The core bug-1479 gap-3 fix: committed_lsn is stalled at 14, but the
|
||||
* leader has real, streamable entries up to 33 -- the ceiling must
|
||||
* widen to 33, not stay capped at 14. */
|
||||
TEST_ASSERT_EQUAL_UINT64(33,
|
||||
ha_compute_repl_stream_ceiling_lsn(14, 33, /*checkpoint_lsn*/0));
|
||||
printf("test_compute_repl_stream_ceiling_lsn_widens_past_committed: PASS\n");
|
||||
}
|
||||
|
||||
static void test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint(void)
|
||||
{
|
||||
/* bug-1478's exact trap, reintroduced here if unguarded: last_written
|
||||
* (15) IS the trailing local-only checkpoint's own consumed LSN slot
|
||||
* (checkpoint_lsn=15) -- no standby can ever legitimately ack that exact
|
||||
* value, so the ceiling must fall back to committed_lsn (14), not widen
|
||||
* to the unreachable checkpoint slot. */
|
||||
TEST_ASSERT_EQUAL_UINT64(14,
|
||||
ha_compute_repl_stream_ceiling_lsn(14, 15, /*checkpoint_lsn*/15));
|
||||
printf("test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint: PASS\n");
|
||||
}
|
||||
|
||||
static void test_compute_repl_stream_ceiling_lsn_committed_already_ahead(void)
|
||||
{
|
||||
/* Normal steady state: committed_lsn is already the higher value (a
|
||||
* standby that hasn't caught up to a recently-advanced commit). The
|
||||
* ceiling must simply be committed_lsn, never go backward. */
|
||||
TEST_ASSERT_EQUAL_UINT64(100,
|
||||
ha_compute_repl_stream_ceiling_lsn(100, 50, /*checkpoint_lsn*/0));
|
||||
printf("test_compute_repl_stream_ceiling_lsn_committed_already_ahead: PASS\n");
|
||||
}
|
||||
|
||||
static void test_compute_repl_stream_ceiling_lsn_zero_zero(void)
|
||||
{
|
||||
/* Genuinely fresh cluster, nothing appended yet. */
|
||||
TEST_ASSERT_EQUAL_UINT64(0,
|
||||
ha_compute_repl_stream_ceiling_lsn(0, 0, /*checkpoint_lsn*/0));
|
||||
printf("test_compute_repl_stream_ceiling_lsn_zero_zero: PASS\n");
|
||||
}
|
||||
|
||||
/* The transitional-leader timeout default test was removed: the config knob
|
||||
* and its default macro are deleted by ha-single-leader-role (the wall-clock
|
||||
* force-promotion watchdog it configured is gone — Req 1.5). */
|
||||
|
|
@ -955,6 +1010,11 @@ void test_ha_role(void)
|
|||
RUN_TEST(test_compute_repl_target_lsn_zero_zero);
|
||||
RUN_TEST(test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn);
|
||||
RUN_TEST(test_compute_repl_target_lsn_committed_already_ahead);
|
||||
/* bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn */
|
||||
RUN_TEST(test_compute_repl_stream_ceiling_lsn_widens_past_committed);
|
||||
RUN_TEST(test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint);
|
||||
RUN_TEST(test_compute_repl_stream_ceiling_lsn_committed_already_ahead);
|
||||
RUN_TEST(test_compute_repl_stream_ceiling_lsn_zero_zero);
|
||||
/* Task 3.3: ha_dispatch_repl_action_for_standby PUSH_CONFIG wiring */
|
||||
RUN_TEST(test_dispatch_repl_action_push_config_when_tag_behind);
|
||||
RUN_TEST(test_dispatch_repl_action_no_push_when_converged);
|
||||
|
|
|
|||
|
|
@ -1983,6 +1983,181 @@ static void test_repl_ack_out_of_range_peer_rejected(void)
|
|||
printf("test_repl_ack_out_of_range_peer_rejected: PASS\n");
|
||||
}
|
||||
|
||||
/* ================================================================== */
|
||||
/* bug-1479 — drive the library on a REJECTED AppendEntries (WAL NACK) */
|
||||
/* ================================================================== */
|
||||
|
||||
/* Before this fix, a WAL NACK (WalAckReply.success == 0) for an ordinary
|
||||
* Raft log conflict (ae_reject_logmismatch) was silently dropped after
|
||||
* stale-term discovery -- the library's own, already-correct nextIndex
|
||||
* backoff (raft_on_append_entries_reply's failure branch, raft.tla 401) was
|
||||
* never invoked, so a standby stuck in log conflict never had its resend
|
||||
* position corrected: the primary kept broadcasting new entries assuming
|
||||
* agreement at an index the standby had already rejected, forever, until
|
||||
* MAX_PENDING_ACKS silently evicted it into catch-up -- at which point
|
||||
* NOTHING streamed it back into sync either (the ACK-driven continuation
|
||||
* needs a real ACK to fire, which a log-conflicted standby can never send).
|
||||
*
|
||||
* ha_raft_on_repl_nack mirrors ha_raft_on_repl_ack's bounds-checked
|
||||
* peer-index resolution, but routes success=0 into the library (backing
|
||||
* off next_index[peer] by one, per raft.c's documented failure branch) and
|
||||
* returns the corrected resume LSN instead of trying to advance commit. */
|
||||
|
||||
/* (a) A rejection backs off next_index[peer] by exactly one and returns the
|
||||
* corrected LSN (next_index - 1) -- the position the standby is now
|
||||
* believed caught up through. */
|
||||
static void test_repl_nack_backs_off_next_index_and_returns_corrected_lsn(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_ts_nack_backoff_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
const uint64_t T = 6;
|
||||
|
||||
cluster_t* c = make_cluster(tmp);
|
||||
TEST_ASSERT_NOT_NULL(c);
|
||||
make_leader(c, T, /*n_peers*/3);
|
||||
leader_seed_entries(c, T, /*count*/5);
|
||||
|
||||
/* make_leader pokes role=RAFT_LEADER directly without going through the
|
||||
* library's real BecomeLeader transition (raft_init_leader_progress,
|
||||
* raft.c:97-103), so next_index[] stays 0 for every peer regardless of
|
||||
* log_len. Seed it here to what a genuine BecomeLeader would have set
|
||||
* (log_len + 1), so this test exercises a realistic starting position. */
|
||||
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
|
||||
TEST_ASSERT_EQUAL_UINT64(6, c->ha.raft.next_index[1]);
|
||||
|
||||
uint64_t corrected_lsn = 999; /* sentinel -- must be overwritten */
|
||||
int rc = ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
|
||||
&corrected_lsn);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
|
||||
/* next_index backed off by exactly one (raft.c: Max(next_index-1, 1)). */
|
||||
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.next_index[1]);
|
||||
/* Corrected resume point is next_index - 1. */
|
||||
TEST_ASSERT_EQUAL_UINT64(4, corrected_lsn);
|
||||
/* A rejection never advances commit. */
|
||||
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.commit_index);
|
||||
|
||||
teardown_cluster(c);
|
||||
printf("test_repl_nack_backs_off_next_index_and_returns_corrected_lsn: PASS\n");
|
||||
}
|
||||
|
||||
/* (b) Repeated rejections keep backing off, floored at 1 -- never underflows
|
||||
* (raft.c's Max(next_index-1, 1), matching raft.tla exactly). */
|
||||
static void test_repl_nack_repeated_rejections_floor_at_one(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_ts_nack_floor_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
const uint64_t T = 6;
|
||||
|
||||
cluster_t* c = make_cluster(tmp);
|
||||
TEST_ASSERT_NOT_NULL(c);
|
||||
make_leader(c, T, /*n_peers*/3);
|
||||
leader_seed_entries(c, T, /*count*/2);
|
||||
|
||||
/* Seed next_index[1] to what a real BecomeLeader would set (log_len+1 =
|
||||
* 3) -- make_leader itself leaves it at 0 (see the backoff test above).
|
||||
* Reject enough times to walk it down to 1 and confirm it never goes
|
||||
* below. */
|
||||
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
|
||||
uint64_t corrected_lsn = 0;
|
||||
for (int i = 0; i < 5; i++) {
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
|
||||
&corrected_lsn));
|
||||
}
|
||||
TEST_ASSERT_EQUAL_UINT64(1, c->ha.raft.next_index[1]);
|
||||
TEST_ASSERT_EQUAL_UINT64(0, corrected_lsn);
|
||||
|
||||
teardown_cluster(c);
|
||||
printf("test_repl_nack_repeated_rejections_floor_at_one: PASS\n");
|
||||
}
|
||||
|
||||
/* (c) Defense-in-depth: same out-of-range/self rejection as the ACK seam --
|
||||
* no library accounting on a bogus peer index. */
|
||||
static void test_repl_nack_out_of_range_peer_rejected(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_ts_nack_oob_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
const uint64_t T = 6;
|
||||
|
||||
cluster_t* c = make_cluster(tmp);
|
||||
TEST_ASSERT_NOT_NULL(c);
|
||||
make_leader(c, T, /*n_peers*/3);
|
||||
leader_seed_entries(c, T, /*count*/2);
|
||||
|
||||
/* Seed next_index[1] as a real BecomeLeader would (log_len+1 = 3); see
|
||||
* the backoff test above for why make_leader itself leaves it at 0. */
|
||||
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
|
||||
|
||||
uint64_t corrected_lsn = 0;
|
||||
TEST_ASSERT_EQUAL_INT(-EINVAL,
|
||||
ha_raft_on_repl_nack(c, /*abs_peer_idx*/3, /*reply_term*/T, &corrected_lsn));
|
||||
TEST_ASSERT_EQUAL_INT(-EINVAL,
|
||||
ha_raft_on_repl_nack(c, /*abs_peer_idx*/0, /*reply_term*/T, &corrected_lsn));
|
||||
|
||||
/* next_index untouched by either rejected call. */
|
||||
TEST_ASSERT_EQUAL_UINT64(3, c->ha.raft.next_index[1]);
|
||||
|
||||
teardown_cluster(c);
|
||||
printf("test_repl_nack_out_of_range_peer_rejected: PASS\n");
|
||||
}
|
||||
|
||||
/* (d) THE CORE REGRESSION: composing ha_raft_on_repl_nack + repl_start_catchup
|
||||
* -- exactly what repl_ack_decode_and_dispatch's NACK branch now does --
|
||||
* marks the standby in_catch_up at the CORRECTED position on the very FIRST
|
||||
* rejection. Before this fix, nothing marked it at all until repl_broadcast's
|
||||
* own MAX_PENDING_ACKS=16 eviction fired (silently, using the standby's last
|
||||
* ACKED lsn rather than a library-backed-off position) -- and even then,
|
||||
* nothing would ever stream to it (the ACK-driven continuation requires a
|
||||
* real ACK a log-conflicted standby can never send). This test proves the
|
||||
* immediate half of the fix: no 16-strike wait, and the resume point comes
|
||||
* from the library's own nextIndex backoff, not a guess. */
|
||||
static void test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_ts_nack_catchup_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
const uint64_t T = 6;
|
||||
|
||||
cluster_t* c = make_cluster(tmp);
|
||||
TEST_ASSERT_NOT_NULL(c);
|
||||
make_leader(c, T, /*n_peers*/3);
|
||||
leader_seed_entries(c, T, /*count*/5);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
repl_add_standby(&c->persist.repl, &g_ae_dummy_ep, "sb0", 7000));
|
||||
|
||||
uint64_t join_lsn_before;
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
|
||||
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_before),
|
||||
"a freshly-added standby must not already be in catch-up");
|
||||
|
||||
/* THE FIRST rejection -- not the 16th. */
|
||||
uint64_t corrected_lsn = 999;
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T, &corrected_lsn));
|
||||
repl_start_catchup(&c->persist.repl, &g_ae_dummy_ep, corrected_lsn);
|
||||
|
||||
uint64_t join_lsn_after;
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
|
||||
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_after),
|
||||
"a single rejection must immediately mark the standby in catch-up "
|
||||
"(bug-1479 -- no 16-strike wait)");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(corrected_lsn, join_lsn_after,
|
||||
"the catch-up resume point must be the library's backed-off "
|
||||
"next_index - 1, not a guess");
|
||||
|
||||
teardown_cluster(c);
|
||||
printf("test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes: PASS\n");
|
||||
}
|
||||
|
||||
/* ================================================================== */
|
||||
/* Task 5.1 — every-callback + marker-path end-to-end integration */
|
||||
/* ================================================================== */
|
||||
|
|
@ -2209,6 +2384,11 @@ void test_term_safety_adapter(void)
|
|||
RUN_TEST(test_repl_ack_abs_peer_idx_endpoint_type_roundtrip);
|
||||
RUN_TEST(test_repl_ack_heartbeat_standby_absolute_slot);
|
||||
RUN_TEST(test_repl_ack_out_of_range_peer_rejected);
|
||||
/* bug-1479: NACK seam drives the library's nextIndex backoff. */
|
||||
RUN_TEST(test_repl_nack_backs_off_next_index_and_returns_corrected_lsn);
|
||||
RUN_TEST(test_repl_nack_repeated_rejections_floor_at_one);
|
||||
RUN_TEST(test_repl_nack_out_of_range_peer_rejected);
|
||||
RUN_TEST(test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes);
|
||||
/* Task 5.1: every-callback + marker-path end-to-end integration. */
|
||||
RUN_TEST(test_every_callback_and_marker_path_end_to_end);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -137,6 +137,16 @@ void repl_start_catchup(replicator_t *r, repl_ep_t ep, uint64_t join_applied_lsn
|
|||
/* Mark standby catch-up complete (called on OP_replCatchUpDone receipt) */
|
||||
void repl_complete_catchup(replicator_t *r, repl_ep_t ep);
|
||||
|
||||
/* bug-1479: look up whether the given standby is currently in catch-up and,
|
||||
* if so, its corrected resume point (join_applied_lsn). Returns 1 and sets
|
||||
* *join_applied_lsn_out when in_catch_up is set; returns 0 (leaves
|
||||
* *join_applied_lsn_out untouched) otherwise. Lets a caller (the heartbeat-
|
||||
* driven replication decision) use the corrected position instead of a
|
||||
* standby's own self-reported LSN, which is unreliable for a standby stuck
|
||||
* rejecting every AppendEntries due to a log conflict. */
|
||||
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
|
||||
uint64_t *join_applied_lsn_out);
|
||||
|
||||
/* Returns 1 if standby needs a full snapshot transfer (its LSN predates oldest WAL).
|
||||
* oldest_wal_lsn == 0 means no WAL exists — no snapshot needed for LSN 0 standbys. */
|
||||
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn, uint64_t oldest_wal_lsn);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue