docs(buglog): bug-1915 and bug-1919 fixed; log bug-1920

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-04 09:14:58 +00:00

4.4 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1920: new primary's WAL catch-up stalls after log-mismatch NACK; standby never advances, commits stuck

Description

Observed:

# new primary ctrl-2 (term 21), its only reachable standby ctrl-1 (ctrl-3 killed):
07:06:45,520 ctrl-2 HA repl: standby[0] lsn=323 < stream_ceiling_lsn=342 (committed_lsn=341) → stream WAL batches
07:06:45,522 ctrl-2 ha_stream_wal_to_standby: streamed WAL [323+1..342] to standby
07:06:45,520 ctrl-1 ha_consume_append_entries: leader_idx=-1 term=21 prevIdx=341 n=1 -> success=0 match=0   (ae_reject_logmismatch)
07:06:45,523 ctrl-2 repl_ack_decode_and_dispatch: WAL NACK from standby lsn=342 success=0 — marking catch-up
07:06:48,603 ctrl-2 HA repl: standby[0] lsn=340 < stream_ceiling_lsn=342 (committed_lsn=341) → stream WAL batches
07:06:48,603 ctrl-2 ha_stream_wal_to_standby: streamed WAL [340+1..342] to standby
07:06:48,602 ctrl-1 ha_consume_append_entries: leader_idx=-1 term=21 prevIdx=323 n=0 -> success=1 match=323   (nop heartbeat)
... same pair every 3 s for >60 s; ctrl-1 never logs another walCatchupEntry ...
07:07:47,044 ctrl-2 commit: WAIT deadline expired without commit (lsn=343 committed_lsn=341 writable_lsn=342 timeout_ms=500)
worker: xcecli failover: cluster not ready after 21 attempts (last status=-10013)
[xce-test-ha] FAIL: round 19: worker-resync job ...: worker never logged a fresh rejoin after the failover (pre=18 post=18)

After the term-21 election the standby's Raft log matched only to 323. The first catch-up stream reached ctrl-1 only as a single entry (prevIdx=341, n=1), which was rejected. The bug-1479 NACK backoff then set the standby's catch-up join position to 340 (one below the rejected index) instead of 323. From then on the primary "streams [341..342]" every heartbeat, yet ctrl-1 receives only nop heartbeats (no walCatchupEntry), so no further NACK arrives to walk the position back. ctrl-1 stays at 323, the 2-of-3 cluster cannot commit (committed_lsn stuck at 341), and every client request gets -10013 not ready.

Expected: the primary walks its catch-up position back to the standby's real match index (323) and streams [324..], so the standby catches up and commits resume, as bug-1479 intended. Impact: after a failover, the new primary is unavailable (no commits, clients get not-ready) for as long as the only other live member stays stalled. Observed for >60 s, unresolved until the soak tore the cluster down. Found by: soak (/xce-test-ha sequential 20 rounds, validating bug-1919, 2026-10-04)

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (xce.build)
Code: scheduler-framework @ acf78f9a-dirty
Deployment: local 3-node HA (/xce-test-ha sequential), one xcecd_virtual soak worker + scheduler; working tree also carries the uncommitted bug-1915 and bug-1919 changes

Reproduction

XCE_HA_ROUNDS=20 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh 20 <scratch dir>.
Needs a failover where the surviving standby's log is far behind the new primary's (here match 323 vs leader tip 342) when the new primary registers it.
Round fails "worker never logged a fresh rejoin"; ctrl-2.log repeats standby[0] lsn=340 < stream_ceiling_lsn=342 every 3 s with no walCatchupEntry arriving at ctrl-1.

Frequency: 1/4 soak runs, 2026-10-04 (round 19 of run 3)
Evidence: session scratchpad stall-ctrl-1.log, stall-ctrl-2.log; soak log soak-seq3.log (deploy dir ha-seq3)

Root cause

TODO: required for fixed/partial

Fix