myxce/.wolf/bugs/bug-1891.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1891	fixed	2026-09-29		controller	hang	2026-09-29T14:33:30Z	2026-09-29T14:33:30Z	1	src/controller/ha.c (repl_ack_decode_and_dispatch) src/controller/config_repl.c (ha_walack_on_ack) src/controller/ha.h (catchup_inflight)	ha snapshot standby repl soak	bug-1871 bug-1819 bug-1474	4922f6c8	337	fixed	bd7825fc6673

bug-1891: xce_controller primary sends a rejoining standby's snapshot to a different standby

Description

Observed: after a standby (ctrl-2) is killed, wiped and rejoins empty, the primary decides a snapshot is needed from ctrl-2's state but the transfer lands on the OTHER, already caught-up standby (ctrl-3). Primary (ctrl-1), xce-1890-nd5:

[2026-09-29 14:28:20,377] ... on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
[2026-09-29 14:28:44,429] ... ha_process_standby_state] HA: standby[1] converged: config_tag=7302089038923290268 applied_lsn=114 (committed_tag=... committed_lsn=114)
[2026-09-29 14:28:44,429] ... ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=0 < oldest_wal_lsn=112 → requesting snapshot transfer
[2026-09-29 14:28:44,437] ... ha_send_snapshot_to_standby: snapshot transfer start checkpoint_lsn=114 size=528825

Every such transfer (14:28:41, :44, :47, :56) was received and installed by ctrl-3 (HA: snapshot loaded lsn=114, 7 installs); ctrl-2 received none. The same standby[1] alternates every heartbeat between converged ... applied_lsn=114 (ctrl-2's pre-wipe position, or ctrl-3's) and lsn=0 → requesting snapshot, and each request is logged twice.

Expected: the snapshot is sent on the rejoining standby's own connection, and its status reflects only that live connection. Impact: a wiped/rejoining standby never catches up while the misrouting persists; the other standby reinstalls the same snapshot every few seconds; ctrlstatus/soak convergence is reported from stale state (the soak printed cluster reconverged (3-of-3) and then failed ctrl-2's log shows NO snapshot install). Found by: soak -- xce-test-ha --force-snapshot while verifying bug-1890, 2026-09-29. Suspected: two repl slots for peer[1] after kill+rejoin (the late-joiner slot 2 plus the pre-kill slot, cf. bug-1819/bug-1474), with state read from one slot and the send going to an endpoint that belongs to another peer.

UPDATE (2026-09-29): root-caused -- not duplicate repl slots as suspected, but an index-space mix-up between vote and heartbeat endpoints (see Root cause).

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (src/xce.build.bug1871; netio TCP_NODELAY patch for bug-1890 applied, uncommitted)
• Code: scheduler-framework @ 3061df14-dirty
• Deployment: local 3-node HA controller cluster (xce-test-ha --force-snapshot), localhost:9991-9993, 32 CPUs

Reproduction

1. nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --force-snapshot <dir>
2. Compare the primary's snapshot transfer start timestamps with ha_handle_snapshot_chunk: chunk crc32 ... chunk_index=0 in each standby log: a transfer the primary started for the rejoining target that arrives on the other standby is the bug; the soak FAILs when none reach the target within its 30 s poll.

• Frequency: 1/9 runs failed outright (xce-1890-nd5), 2026-09-29; misrouted transfers also seen without failing in xce-1871-fs16 (13:36:18 and 13:36:48 went to ctrl-3 before the target got one at 13:36:51), i.e. pre-dates the bug-1871 and bug-1890 changes
• Evidence: /home/claude/tmp/xce-1890-nd5 (failing run, logs/ctrl-*.log, ../xce-1890-nd5.out), /home/claude/tmp/xce-1871-fs16

Root cause

Two kinds of endpoint deliver heartbeat state reports to the primary. A promoted vote endpoint's vote_ep_ctx_t.peer_idx is RELATIVE to the votes array, which leaves out the primary itself; a late-joiner's heartbeat standby endpoint carries the ABSOLUTE peer index. repl_ack_decode_and_dispatch (ha.c) passed the raw vctx->peer_idx to ha_process_standby_state (which stores ha->standby_reported_tag/lsn[peer_idx]) and ha_dispatch_repl_action_for_standby (which reads those back and acts on the endpoint the report arrived on). The WAL-ack branch beside it already remapped with ha_repl_ack_abs_peer_idx; the state-report branch did not. With ctrl-1 primary (self 0), ctrl-3's vote endpoint (relative 1) and the rejoined ctrl-2's heartbeat endpoint (absolute 1) shared slot 1: their handlers interleaved, one read the other's position and acted on its own endpoint. ctrl-3's handler read ctrl-2's lsn=0 and sent the snapshot over ctrl-3's connection (hence the duplicated requesting snapshot transfer lines and the alternating converged ... applied_lsn=114 for standby[1]). ha_walack_on_ack also received the raw index, so catchup_inflight[] single-flight flags could collide the same way.

Fix

src/controller/ha.c: the state-report branch of repl_ack_decode_and_dispatch resolves the absolute index with ha_repl_ack_abs_peer_idx(ha, vctx->ctx != NULL, vctx->peer_idx) before recording, deciding and sending, and ha_walack_on_ack gets the already-resolved absolute index. src/controller/ha.h + config_repl.c: catchup_inflight[] is sized MAX_HA_CANDIDATES (was MAX_STANDBYS=3) and its four bounds checks updated, so every absolute index fits (a 4-candidate cluster's peer at absolute 3 would otherwise be rejected). Each standby's state now has its own slot, so the snapshot decision and the send are for the same standby. Test seam ha_repl_ack_dispatch_for_test (XCE_TESTING) drives the dispatcher.

Verification

• Test: test_vote_ep_report_lands_on_its_absolute_index in src/controller/testsuite/test_standby_state_peer_idx.c sends a real encoded state report through repl_ack_decode_and_dispatch on a promoted vote endpoint with relative index 1 (primary at 0): RED before the fix (Expected 5 Was 777. bug-1891: peer 2's state report was not recorded under peer 2), GREEN after; the heartbeat-endpoint baseline passes both ways
• Suites: controller 1258/1258, asan, no sanitizer errors
• Live: xce-test-ha --force-snapshot, 16 runs (2026-09-29, xce-1891-fx1..16): 16/16 PASS and 0 snapshots installed on a non-target node after the target rejoined -- versus misrouted installs in 25 of 48 runs before the fix (xce-1871-fs/-fix, xce-1890-nd). xce-test-ha --sequential 6 rounds (xce-1891-seq2): PASS, all 3 nodes led, zero job loss, 6 worker-resync jobs reconciled, replay parity 7 files 0 mismatches

References

• Fix commit: 4922f6c8
• Related: bug-1871, bug-1890 (found while verifying it), bug-1819
• Gitea: #337