myxce/.wolf/bugs/bug-1913.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1913	fixed	2026-10-03		controller	wrong-result	2026-10-03T14:18:58Z	2026-10-03T14:18:58Z	1	src/controller/ha.c src/controller/repl.c src/controller/config_repl.c	ha raft replication latency	bug-1485	2aefa939

bug-1913: after a late-joiner election the primary cannot commit client writes for ~2 heartbeat cycles

Description

Observed:

repl_broadcast: no standbys registered ... not replicated
...
WAL NACK from standby lsn=2 ... marking catch-up
repl_wait_majority_ack: timeout ... OP_upgradeConfig rejected (status=-110)

In some elections the vote connections are not converted into standby channels (raft_cb_become_leader), so both standbys register later through on_standby_connected (ha.c ~9643-9660). ha_arm_own_term_gate (ha.c ~5178) has already appended the own-term WAL_NOOP (lsn 1) while no standby was registered (repl.c ~306-311), and repl_add_standby does not resend it. Catch-up then runs from the 3 s heartbeat, one action per standby per cycle (config push first, WAL stream on the next cycle), so committed_lsn stays 0 for ~6 s. Any client write in that window (e.g. OP_upgradeConfig) is NACKed by both standbys and fails with -ETIMEDOUT after the 500 ms majority wait.

Expected: a newly elected primary commits its own-term no-op, and so accepts client writes, promptly after its standbys register, not two heartbeat cycles later. Impact: for ~6 s after such an election every client write is rejected with -110 (fail-loud, no data loss, no false success). Clients retry, but tests and operators see spurious failures right after failover (the root cause of bug-1485's versionTag=0 flake). A fix must respect the async-I/O rules (no blocking on the delaymgr/heartbeat path; streaming offloaded) in .kiro/steering/ha-async-io.md. Found by: test (root cause of bug-1485)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 91782e40-dirty
• Deployment: local 3-node HA (xceadmin live e2e fixture), CPU-loaded sandbox (40 busy loops on 32 cores) makes late-joiner elections frequent

Reproduction

1. Start a 3-node HA cluster (e.g. the live_ha_cluster fixture in src/xceadmin/tests/test_e2e_live_ha_push.py) under CPU load.
2. In the primary log, find an election with no "promoted" standby channels and a later "repl_add_standby"/late registration; push a config (xceadmin config push ... --controller <primary>) within ~6 s.
3. The push fails with controller returned status -110; the log shows the NACK and majority-ack timeout above.

• Frequency: 7/28 runs of the HA push test under 40-spinner CPU load, 0/6 idle (2026-10-03); all 7 failures were late-joiner elections.
• Evidence: not kept (scratchpad loop logs from the bug-1485 investigation).

Root cause

Nothing asked a late-registered standby for its state, so its catch-up waited for the heartbeat clock. ha_arm_own_term_gate (ha.c ~5180) appends the own-term no-op while no standby is registered; repl_broadcast drops it (repl.c:306-311) and repl_add_standby never resends it. on_standby_connected (ha.c ~9511-9690) sends only a ctrlhello (no reply) and sets last_push_time = now. Catch-up is driven solely by a heartbeat reply's state report → repl_ack_decode_and_dispatch (peer-control vtask) → ha_dispatch_repl_action_for_standby (ha.c:4455), one action per report (config push if the tag is behind, else WAL stream). A standby rejoining with tag 0 therefore got the config push on the next 3 s tick and the WAL stream carrying the no-op on the tick after: ~6 s with committed_lsn below the no-op and every client write NACKed. The config-before-WAL order is a pacing rule of the ha-repl-hardening design ("no further decisions until the next state report confirms"), not a correctness requirement. Why the vote connections were not promoted in the first place is tracked separately (bug-1915).

Fix

Make the state report arrive sooner; the decision loop and its one-action-per-report pacing are unchanged. New ha_send_catchup_probe() (ha.c) sends the ordinary xce_ha_heartbeat_req (same fields, read the same way as primary_heartbeat_fn), only while CTRL_PRIMARY, encode + enqueue only (same non-blocking path as the ctrlhello and the delaymgr tick); a send failure is a WARN and the periodic tick stays the fallback. Triggers: (1) once at the end of on_standby_connected (iomgr connect callback, after msg_handler is set), which also arms ha->catchup_kick_armed[peer] (ha.h); (2) the accepted config-push-ACK branch of repl_ack_decode_and_dispatch (peer-control vtask) after a 1→0 CAS on the arm, so the WAL stream follows the config push at ~RTT while keeping config-before-WAL. The arm is cleared on the standby's disconnect (standby_ep_err_handler_primary) and on every raft_cb_become_leader / raft_cb_become_follower, so it never outlives its registration or leadership; at most two probes per connect. No new executor, opcode or protocol change; heartbeat interval unchanged; WAL streaming still runs only on the vtask.

Verification

• Test: src/controller/testsuite/test_late_joiner_catchup_kick.c — test_late_joiner_registration_probes_standby_state, test_config_ack_after_registration_reprobes_once, test_rejected_config_ack_does_not_probe RED before the fix (Expected 1 Was 0), GREEN after; test_step_down_clears_catchup_arm RED with the become_follower clear removed (Expected 0 Was 1); guards for no-registration and non-primary.
• Suites: controller 5/5 meson suites, main 1285/0 (asan); xce-build.sh all --incremental green (C tests 127/0, scheduler 16/0, xceadmin 373), 2026-10-03.
• Live: /xce-test-ha sequential 20 rounds PASS (63 jobs, zero loss, all 3 led, SIMVERIFY 21/0); chaotic 6 rounds PASS (zero committed divergence/loss); sequential 8 rounds under 40 busy loops PASS (with the bug-1916 fix; earlier loaded runs aborted on bug-1916). In those soaks catch-up started 0-6 ms after each late standby registration and a restarted standby was fully caught up 46 ms after registering (was up to ~6 s). Proxy (wiped standby) before/after: WAL stream at +5.8 s → +18-60 ms. Live HA push tests 6/6 under load; promoted-vote elections commit the no-op in median 6 ms (max 16 ms). Observation, not a regression: after catch-up the decision loop re-streams already-acked ranges on later stale state reports (harmless; standby discards duplicates).

References

• Related: none