myxce/.wolf/bugs/bug-1879.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1879	fixed	2026-09-28		controller	hang	2026-09-28T07:11:39Z	2026-09-28T07:11:39Z	1	src/controller/ha.c src/controller/myapp.c	controller ha election cold-boot	bug-1876 bug-606	aa7fc65a	330	fixed	4cfc1a40516a

bug-1879: fresh HA cluster has no primary for ~10 s: first election lost, not retried until liveness timeout

Description

Observed:

[14:47:32,960] ctrl-1 HA FSM: -> CTRL_ELECTING (async, n_peers=3)
[14:47:32,962] ctrl-1 HA FSM (raft): -> CTRL_STANDBY (term=1, first_boot=1)
[14:47:32,962] ctrl-1 myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
[14:47:44,368] ctrl-1 WARN liveness_poll_fn: HA: liveness poll: no primary activity for 11s (timeout=10s) -> failover
[14:47:44,661] ctrl-1 HA FSM (raft): -> CTRL_PRIMARY (term=2, first_boot=0)

(pc13v; locally ctrl-2/ctrl-3 log ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only) Expected: a fresh cluster elects its primary as soon as a quorum of nodes is up (about a second with start-all.sh's 0.5-1 s stagger). Impact: every cold start of an HA cluster has no primary for about 10 s (the standby liveness timeout): no job submission, config push or scheduling in that window, a spurious failover WARN, and the cluster always starts at term 2. Also the cause of the 10 s of standby 'waiting for config' seen in bug-1876. Found by: user reading ctrl-1's boot log and ctrlstatus (electedTerm 2) on pc13v after the bug-1872..1877 retest

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 44173f74 (controller unchanged since; pc13v ran 4c134127)
• Deployment: local 3-node HA (localhost:9991-9993, src/tools/deploy-ha.sh, nodes started 1 s apart by start-all.sh, xcecd_virtual) on this dev host; first seen on pc13v: NixOS 26.05, kernel 6.18.33, glibc 2.42-47, debug build, 3-node HA (ctrl-1..3, localhost:9901-9903) + scheduler + apiserver + real root xcecd, /tmp/xce-ha-deploy, 2026-09-28 14:47 +0800

Reproduction

1. /xce-deploy-ha (fresh deploy; only ctrl-1 gets the cluster config)
2. scripts/start-all.sh (starts ctrl-1, ctrl-2, ctrl-3 about 1 s apart)
3. grep -E 'CTRL_|liveness poll|voter only' logs/ctrl-*.log: ctrl-1 goes CTRL_STANDBY within ~50 ms, ctrl-2/3 are voter-only, and the first primary appears ~10 s later at term 2

• Frequency: 2/2 cold starts (pc13v 14:47 and the local run 07:10, 2026-09-28)
• Evidence: logs quoted above; the local deploy's logs were in /tmp/claude-1000/xce-deploy-coldboot/logs (not kept)

Root cause

Established from both runs: ctrl-1, the only node with a local config, starts its term-1 election before its peers are running, cannot reach a quorum, and drops to CTRL_STANDBY ("waiting for primary") within ~50 ms. ctrl-2 and ctrl-3 have no local config, so ha_start_election_async() makes them voters only -- they never stand. That is by design (confirmed by the user, 2026-09-28): on a cold boot only the node holding the initial configuration (ctrl-1) may become primary, so the retry has to come from ctrl-1. Nothing starts another election until ctrl-1's standby liveness poll (10 s) treats the missing primary as a failure and runs PreVote, which it then wins at once.

Confirmed: why the lost first-boot election leaves ctrl-1 in CTRL_STANDBY with only the 10 s liveness poll as a retry, instead of retrying after a short (randomized) election timeout as Raft does for a candidate that fails to win -- to be read in ha.c's raft_cb_become_follower()/first_boot path and myapp_run().

Confirmed in the code: vote_reply_dispatch() counts a connect failure as a "no" vote (bug-1423), so with both peers down the raft library's all-replied-no-quorum step (raft.c Step 6) makes ctrl-1 a follower within ~50 ms; raft_cb_become_follower() only arms the standby liveness watchdog (heartbeat_timeout_ms, 10 s). A second defect blocked a faster retry: ha_heartbeat_is_fresh() treats last_activity -- which ha_arm_heartbeat_timeout_vtask() sets to the arm time -- as primary contact, so the failover gates believed a primary was alive.

Fix

ha_note_leaderless() (ha.c) marks a config-holding node that becomes a follower with no leader known. While marked, the liveness poll retries the election through the existing PreVote failover after HA_LEADERLESS_RETRY_MS (1 s), doubling up to heartbeat_timeout_ms, and polls no slower than the pending retry; ha_heartbeat_is_fresh() reports "not fresh" (no primary heard from yet). ha_on_heartbeat_received() and raft_cb_become_leader() clear it. Config-less nodes stay voters only, as designed.

Verification

• Test: test_leaderless_follower_retries_election_after_short_delay, test_leaderless_retry_waits_for_its_delay, test_leaderless_retry_backoff_is_capped, test_primary_contact_clears_leaderless and test_only_a_config_holder_without_leader_is_leaderless in src/controller/testsuite/test_ha_failover_prevote.c (new behaviour; the first one also caught the freshness-gate defect)
• Suites: controller suite 1256/0 (asan); test_ha_failover_prevote 11/0, test_ha_liveness 38/0, test_ha_failover 8/0, test_ctrlstatus 23/0
• Live: local cold start 2026-09-28 07:23:58: ctrl-1 lost its first election at .476, retried after 1381 ms, CTRL_PRIMARY at 07:24:00.152 (1.7 s after start; ~10 s before). HA soaks on aa7fc65a's tree: sequential 6 rounds PASS (21 jobs, zero job loss, all 3 nodes led); chaotic 6 rounds PASS (zero committed divergence, zero committed job loss); pc13v retest pending; both soaks' cold starts reached a primary in 1.7 s

References

• Related: bug-1876 (the standby 'waiting for config' warnings this caused), bug-606 (PreVote staggering)