chore(buglog): bug-1890 fixed (8b561547); log bug-1891 (snapshot sent to the wrong standby)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/bugs/bug-1891.md

@@ -0,0 +1,58 @@
+---
+id: "bug-1891"
+status: "open"
+status_updated: "2026-09-29"
+status_note: "newly logged"
+component: "controller"
+severity: "hang"
+timestamp: "2026-09-29T14:33:30Z"
+last_seen: "2026-09-29T14:33:30Z"
+occurrences: 1
+files: ["src/controller/ha.c (ha_dispatch_repl_action_for_standby)", "src/controller/config_repl.c (ha_send_snapshot_to_standby)"]
+tags: ["ha", "snapshot", "standby", "repl", "soak"]
+related: ["bug-1871", "bug-1819", "bug-1474"]
+---
+
+# bug-1891: xce_controller primary sends a rejoining standby's snapshot to a different standby
+
+## Description
+**Observed:** after a standby (ctrl-2) is killed, wiped and rejoins empty, the primary decides a snapshot is needed from ctrl-2's state but the transfer lands on the OTHER, already caught-up standby (ctrl-3). Primary (ctrl-1), xce-1890-nd5:
+```
+[2026-09-29 14:28:20,377] ... on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
+[2026-09-29 14:28:44,429] ... ha_process_standby_state] HA: standby[1] converged: config_tag=7302089038923290268 applied_lsn=114 (committed_tag=... committed_lsn=114)
+[2026-09-29 14:28:44,429] ... ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=0 < oldest_wal_lsn=112 → requesting snapshot transfer
+[2026-09-29 14:28:44,437] ... ha_send_snapshot_to_standby: snapshot transfer start checkpoint_lsn=114 size=528825
+```
+Every such transfer (14:28:41, :44, :47, :56) was received and installed by ctrl-3 (`HA: snapshot loaded lsn=114`, 7 installs); ctrl-2 received none. The same standby[1] alternates every heartbeat between `converged ... applied_lsn=114` (ctrl-2's pre-wipe position, or ctrl-3's) and `lsn=0 → requesting snapshot`, and each request is logged twice.
+
+**Expected:** the snapshot is sent on the rejoining standby's own connection, and its status reflects only that live connection.
+**Impact:** a wiped/rejoining standby never catches up while the misrouting persists; the other standby reinstalls the same snapshot every few seconds; ctrlstatus/soak convergence is reported from stale state (the soak printed `cluster reconverged (3-of-3)` and then failed `ctrl-2's log shows NO snapshot install`).
+**Found by:** soak -- xce-test-ha --force-snapshot while verifying bug-1890, 2026-09-29. Suspected: two repl slots for peer[1] after kill+rejoin (the late-joiner slot 2 plus the pre-kill slot, cf. bug-1819/bug-1474), with state read from one slot and the send going to an endpoint that belongs to another peer.
+
+## Environment
+- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
+- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (src/xce.build.bug1871; netio TCP_NODELAY patch for bug-1890 applied, uncommitted)
+- Code: scheduler-framework @ 3061df14-dirty
+- Deployment: local 3-node HA controller cluster (xce-test-ha --force-snapshot), localhost:9991-9993, 32 CPUs
+
+## Reproduction
+1. `nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --force-snapshot <dir>`
+2. Compare the primary's `snapshot transfer start` timestamps with `ha_handle_snapshot_chunk: chunk crc32 ... chunk_index=0` in each standby log: a transfer the primary started for the rejoining target that arrives on the other standby is the bug; the soak FAILs when none reach the target within its 30 s poll.
+- Frequency: 1/9 runs failed outright (xce-1890-nd5), 2026-09-29; misrouted transfers also seen without failing in xce-1871-fs16 (13:36:18 and 13:36:48 went to ctrl-3 before the target got one at 13:36:51), i.e. pre-dates the bug-1871 and bug-1890 changes
+- Evidence: /home/claude/tmp/xce-1890-nd5 (failing run, logs/ctrl-*.log, ../xce-1890-nd5.out), /home/claude/tmp/xce-1871-fs16
+
+## Root cause
+TODO: required for fixed/partial
+
+## Fix
+TODO: required for fixed/partial
+
+## Verification
+- Test: TODO: name + file; RED before the fix, GREEN after
+- Suites: TODO: suite results and build profile
+- Live: TODO: soak/deploy run, or n/a and why
+
+## References
+- Related: none