docs(buglog): bug-1915 and bug-1919 fixed; log bug-1920

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/bugs/bug-1920.md

@@ -0,0 +1,67 @@
+---
+id: "bug-1920"
+status: "investigating"
+status_updated: "2026-10-04"
+status_note: "Next: why ha_stream_wal_to_standby sends no walCatchupEntry for [341..342] after the NACK, and why the NACK backoff lands at 340 rather than the standby's match 323"
+component: "controller"
+severity: "hang"
+timestamp: "2026-10-04T07:25:24Z"
+last_seen: "2026-10-04T07:25:24Z"
+occurrences: 1
+files: ["src/controller/ha.c", "src/controller/config_repl.c", "src/controller/repl.c"]
+tags: ["ha", "repl", "catch-up", "soak"]
+related: ["bug-1479"]
+---
+
+# bug-1920: new primary's WAL catch-up stalls after log-mismatch NACK; standby never advances, commits stuck
+
+## Description
+**Observed:**
+```
+# new primary ctrl-2 (term 21), its only reachable standby ctrl-1 (ctrl-3 killed):
+07:06:45,520 ctrl-2 HA repl: standby[0] lsn=323 < stream_ceiling_lsn=342 (committed_lsn=341) → stream WAL batches
+07:06:45,522 ctrl-2 ha_stream_wal_to_standby: streamed WAL [323+1..342] to standby
+07:06:45,520 ctrl-1 ha_consume_append_entries: leader_idx=-1 term=21 prevIdx=341 n=1 -> success=0 match=0   (ae_reject_logmismatch)
+07:06:45,523 ctrl-2 repl_ack_decode_and_dispatch: WAL NACK from standby lsn=342 success=0 — marking catch-up
+07:06:48,603 ctrl-2 HA repl: standby[0] lsn=340 < stream_ceiling_lsn=342 (committed_lsn=341) → stream WAL batches
+07:06:48,603 ctrl-2 ha_stream_wal_to_standby: streamed WAL [340+1..342] to standby
+07:06:48,602 ctrl-1 ha_consume_append_entries: leader_idx=-1 term=21 prevIdx=323 n=0 -> success=1 match=323   (nop heartbeat)
+... same pair every 3 s for >60 s; ctrl-1 never logs another walCatchupEntry ...
+07:07:47,044 ctrl-2 commit: WAIT deadline expired without commit (lsn=343 committed_lsn=341 writable_lsn=342 timeout_ms=500)
+worker: xcecli failover: cluster not ready after 21 attempts (last status=-10013)
+[xce-test-ha] FAIL: round 19: worker-resync job ...: worker never logged a fresh rejoin after the failover (pre=18 post=18)
+```
+After the term-21 election the standby's Raft log matched only to 323. The first catch-up stream reached ctrl-1 only as a single entry (prevIdx=341, n=1), which was rejected. The bug-1479 NACK backoff then set the standby's catch-up join position to 340 (one below the rejected index) instead of 323. From then on the primary "streams [341..342]" every heartbeat, yet ctrl-1 receives only nop heartbeats (no walCatchupEntry), so no further NACK arrives to walk the position back. ctrl-1 stays at 323, the 2-of-3 cluster cannot commit (committed_lsn stuck at 341), and every client request gets -10013 not ready.
+
+**Expected:** the primary walks its catch-up position back to the standby's real match index (323) and streams [324..], so the standby catches up and commits resume, as bug-1479 intended.
+**Impact:** after a failover, the new primary is unavailable (no commits, clients get not-ready) for as long as the only other live member stays stalled. Observed for >60 s, unresolved until the soak tore the cluster down.
+**Found by:** soak (`/xce-test-ha` sequential 20 rounds, validating bug-1919, 2026-10-04)
+
+## Environment
+- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
+- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (xce.build)
+- Code: scheduler-framework @ acf78f9a-dirty
+- Deployment: local 3-node HA (`/xce-test-ha` sequential), one xcecd_virtual soak worker + scheduler; working tree also carries the uncommitted bug-1915 and bug-1919 changes
+
+## Reproduction
+1. `XCE_HA_ROUNDS=20 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh 20 <scratch dir>`.
+2. Needs a failover where the surviving standby's log is far behind the new primary's (here match 323 vs leader tip 342) when the new primary registers it.
+3. Round fails "worker never logged a fresh rejoin"; ctrl-2.log repeats `standby[0] lsn=340 < stream_ceiling_lsn=342` every 3 s with no walCatchupEntry arriving at ctrl-1.
+- Frequency: 1/4 soak runs, 2026-10-04 (round 19 of run 3)
+- Evidence: session scratchpad stall-ctrl-1.log, stall-ctrl-2.log; soak log soak-seq3.log (deploy dir ha-seq3)
+
+## Root cause
+TODO: required for fixed/partial
+
+## Fix
+TODO: required for fixed/partial
+
+## Verification
+- Test: TODO: name + file; RED before the fix, GREEN after
+- Suites: TODO: suite results and build profile
+- Live: TODO: soak/deploy run, or n/a and why
+
+## References
+- Related: none