wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

113
.wolf/memory.md
View file

@ -6724,3 +6724,116 @@
| 02:51 | Edited .kiro/specs/scheduler-core/requirements.md | modified loop() | ~535 |
| 02:52 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | removed 6 lines | ~120 |
| 02:53 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | modified used() | ~615 |
| 02:53 | Edited .kiro/specs/scheduler-core/requirements.md | modified push() | ~1417 |
| 02:53 | Session end: 16 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 6 reads | ~164121 tok |
| 02:54 | Session end: 16 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 8 reads | ~184449 tok |
| 02:55 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~213 |
| 02:55 | Session end: 17 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 8 reads | ~184677 tok |
| 02:57 | Session end: 17 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 9 reads | ~195415 tok |
| 02:58 | Session end: 17 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 13 reads | ~417052 tok |
| 03:02 | Edited .kiro/specs/scheduler-core/requirements.md | 3→3 lines | ~101 |
| 03:02 | Edited .kiro/specs/scheduler-core/requirements.md | modified while() | ~1161 |
| 03:05 | Edited .kiro/specs/scheduler-core/requirements.md | expanded (+13 lines) | ~424 |
| 03:06 | Edited .kiro/specs/scheduler-core/requirements.md | modified struct() | ~106 |
| 03:06 | Session end: 21 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~451981 tok |
| 03:14 | Edited .kiro/specs/scheduler-core/requirements.md | expanded (+15 lines) | ~453 |
| 03:14 | Session end: 22 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~452506 tok |
| 05:21 | Session end: 22 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~452506 tok |
| 05:21 | Session end: 22 writes across 4 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md) | 15 reads | ~452506 tok |
| 06:06 | Edited src/controller/ha.h | modified exactly() | ~836 |
| 06:07 | Edited src/controller/ha.c | added 11 condition(s) | ~856 |
| 06:07 | Edited src/controller/myapp.c | modified if() | ~173 |
| 06:08 | Edited src/controller/ha.c | added 2 condition(s) | ~242 |
| 06:09 | Edited src/controller/ha.c | modified if() | ~279 |
| 06:10 | Edited src/controller/ha.c | added 2 condition(s) | ~257 |
| 06:11 | Edited src/controller/ha.c | added 2 condition(s) | ~414 |
| 06:11 | Edited src/controller/ha.c | added 2 condition(s) | ~319 |
| 06:13 | Edited src/controller/testsuite/test_config_repl.c | modified is() | ~1609 |
| 06:14 | Edited src/controller/testsuite/test_config_repl.c | 2→6 lines | ~119 |
| 06:16 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
| 06:18 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
| 06:26 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
| 06:26 | Session end: 32 writes across 8 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 17 reads | ~498442 tok |
| 07:06 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-config/c8633ffb-8da0-40a7-9811-48f1830c37b4/scratchpad/cross-spec-gap-analysis.md | — | ~2172 |
| 07:07 | Edited ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-config/c8633ffb-8da0-40a7-9811-48f1830c37b4/scratchpad/cross-spec-gap-analysis.md | 3→4 lines | ~143 |
| 07:07 | Session end: 34 writes across 9 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 21 reads | ~512802 tok |
| 07:12 | Edited .kiro/specs/admin-config-tool/spec.json | modified 25() | ~375 |
| 07:12 | Edited .kiro/specs/admin-config-tool/spec.json | inline fix | ~136 |
| 07:12 | Edited .kiro/specs/admin-config-tool/tasks.md | inline fix | ~182 |
| 07:14 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 23 reads | ~521081 tok |
| 07:29 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
| 07:34 | Investigated force-elect job-submit FAIL in ~/tmp/xce-ha-configupgrade2 soak; refuted admission-slot-leak hypothesis, root-caused via xce_test waldump to a permanent ae_reject_logmismatch wedge (ctrl-2 lsn=13 term=3 vs canonical term=4) + NACK-ignored/pending_ack_count-never-decremented + no catch-up trigger on eviction | .wolf/buglog.json (bug-1479), .wolf/cerebrum.md | logged, not fixed (needs-owner, design decision) | ~180000 |
| 19:30 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
| 19:31 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
| 19:32 | Session end: 37 writes across 10 files (config_repl.c, requirements.md, test_convergence_verification.c, tasks.md, ha.h) | 24 reads | ~529661 tok |
| 19:41 | Edited src/include/repl.h | expanded (+10 lines) | ~204 |
| 19:41 | Edited src/controller/repl.c | modified just() | ~218 |
| 19:41 | Edited src/controller/repl.c | added 3 condition(s) | ~214 |
| 19:42 | Edited src/controller/ha.h | modified ha_raft_on_repl_nack() | ~434 |
| 19:42 | Edited src/controller/ha.c | added 3 condition(s) | ~439 |
| 19:42 | Edited src/controller/ha.c | added error handling | ~687 |
| 19:43 | Edited src/controller/ha.c | added 2 condition(s) | ~350 |
| 19:45 | Edited src/controller/testsuite/test_term_safety_adapter.c | added error handling | ~1320 |
| 19:45 | Edited src/controller/testsuite/test_term_safety_adapter.c | 2→6 lines | ~106 |
| 19:46 | Edited src/controller/testsuite/test_term_safety_adapter.c | modified test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes() | ~670 |
| 19:47 | Edited src/controller/testsuite/test_term_safety_adapter.c | 2→3 lines | ~59 |
## Session: 2026-07-29 19:52
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 19:55 | Edited src/controller/testsuite/test_term_safety_adapter.c | raft_init() → transition() | ~173 |
| 19:55 | Edited src/controller/testsuite/test_term_safety_adapter.c | 6→9 lines | ~110 |
| 19:55 | Edited src/controller/testsuite/test_term_safety_adapter.c | 11→15 lines | ~181 |
| 20:01 | Session end: 3 writes across 1 files (test_term_safety_adapter.c) | 4 reads | ~201524 tok |
| 20:05 | Session end: 3 writes across 1 files (test_term_safety_adapter.c) | 4 reads | ~201524 tok |
| 20:09 | Session end: 3 writes across 1 files (test_term_safety_adapter.c) | 4 reads | ~201524 tok |
| 20:11 | Edited .kiro/specs/scheduler-core/requirements.md | added 1 condition(s) | ~1739 |
| 20:11 | Session end: 4 writes across 2 files (test_term_safety_adapter.c, requirements.md) | 7 reads | ~262249 tok |
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 7→7 lines | ~167 |
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 3→3 lines | ~79 |
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 12 → 13 | ~9 |
| 20:13 | Edited .kiro/specs/scheduler-core/requirements.md | 11→11 lines | ~457 |
| 20:14 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~159 |
| 20:14 | Edited .kiro/specs/scheduler-core/requirements.md | 9→9 lines | ~344 |
| 20:14 | Edited .kiro/specs/scheduler-core/requirements.md | 17→19 lines | ~731 |
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | 19 → 21 | ~9 |
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~18 |
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | 16 → 18 | ~22 |
| 20:15 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~45 |
| 20:16 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~33 |
| 20:16 | Edited .kiro/specs/scheduler-core/requirements.md | 19 → 21 | ~25 |
| 20:16 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~30 |
| 20:17 | Edited .kiro/specs/scheduler-core/requirements.md | inline fix | ~6 |
| 20:20 | Session end: 19 writes across 2 files (test_term_safety_adapter.c, requirements.md) | 7 reads | ~266129 tok |
| 20:22 | Session end: 19 writes across 2 files (test_term_safety_adapter.c, requirements.md) | 7 reads | ~266129 tok |
| 20:29 | Edited src/controller/ha.h | modified XCE_TESTING() | ~548 |
| 20:29 | Edited src/controller/ha.c | added 1 condition(s) | ~450 |
| 20:29 | Edited src/controller/ha.c | expanded (+10 lines) | ~227 |
| 20:30 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~950 |
| 20:30 | Edited src/controller/testsuite/test_config_repl.c | 3→4 lines | ~64 |
| 20:31 | Edited src/controller/ha.c | 3→4 lines | ~35 |
| 20:32 | Edited src/controller/ha.c | 4→3 lines | ~21 |
| 20:33 | Created .kiro/specs/raft-adapter-conformance/spec.json | — | ~209 |
| 20:33 | Created .kiro/specs/raft-adapter-conformance/requirements.md | — | ~986 |
| 20:34 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 2→2 lines | ~793 |
| 20:35 | Session end: 29 writes across 7 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 13 reads | ~401510 tok |
| 20:35 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | added error handling | ~858 |
| 20:35 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 3→4 lines | ~31 |
| 20:37 | Session end: 31 writes across 7 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 13 reads | ~402460 tok |
| 20:51 | Edited src/controller/ha.h | modified ha_compute_repl_stream_ceiling_lsn() | ~512 |
| 20:51 | Edited src/controller/ha.c | added 1 condition(s) | ~265 |
| 20:52 | Edited src/controller/ha.c | added 1 condition(s) | ~291 |
| 20:54 | Edited src/controller/ha.c | modified XCE_TESTING() | ~563 |
| 20:54 | Edited src/controller/ha.h | expanded (+7 lines) | ~181 |
| 20:55 | Edited src/controller/testsuite/test_ha_role.c | added error handling | ~870 |
| 20:55 | Edited src/controller/testsuite/test_ha_role.c | modified 1479() | ~134 |
| 20:55 | Edited src/controller/ha.c | modified ha_compute_repl_stream_ceiling_lsn() | ~110 |
| 20:58 | Edited src/controller/ha.c | added 1 condition(s) | ~105 |
| 21:01 | Session end: 40 writes across 8 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 17 reads | ~461541 tok |
| 21:03 | Session end: 40 writes across 8 files (test_term_safety_adapter.c, requirements.md, ha.h, ha.c, test_config_repl.c) | 17 reads | ~461541 tok |
| 21:06 | bug-1479 FULLY FIXED (3-layer: primary NACK routing + standby NACK sending + stream-ceiling widening) | ha.c, ha.h, repl.c, repl.h, test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c | 1053/1053 tests + live soak PASS (3/3 force-elect rounds) | ~180000 |
| 21:08 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | 7→7 lines | ~277 |
| 21:08 | Edited .kiro/specs/config-upgrade-snapshot/tasks.md | added error handling | ~1047 |
| 21:09 | Edited .kiro/specs/config-upgrade-snapshot/spec.json | 23→22 lines | ~135 |
| 21:10 | Edited .kiro/specs/raft-adapter-conformance/requirements.md | added error handling | ~567 |