myxce/.wolf/bugs/bug-1695.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1695	fixed	2026-09-24		controller	unknown	2026-08-25T15:05:00Z	2026-09-19T09:00:00Z	2	src/controller/myapp.c (myapp_run's bootstrap check call site only); src/controller/startup_validation.c (myapp_leader_already_known the extracted+fixed decision function -- placed here not myapp.c so it links into controller_unit_test); src/controller/testsuite/test_ha_startup.c (7 new regression tests)	controller ha raft election election-safety bootstrap myapp live-deployment found-via-soak skip_election term_at_boot live-soak-verified bug-1487-adjacent	bug-1477 bug-1684 bug-1487	true

bug-1695: Found via the mandatory live HA chaotic soak (xce-audit-invariants controller campaign, Step 4)…

Description

Found via the mandatory live HA chaotic soak (xce-audit-invariants controller campaign, Step 4) -- NOT caused by any of the 9 fixes that campaign made (bug-1684/1687/1689/1690/1691/1692/1693/1694), all of which are in src/controller/{ha,persist,job,cluster,sched,worker}.c; this bug is in myapp.c, untouched by any of them. 10-round --chaotic soak: every single round after the first shows 'NO new primary within window (possible election stall)'; ctrl-1's own log shows, on EVERY restart after the first boot, 'myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=1, raft_term=1, primary_fd=-1) -- skipping own election' -- primary_fd=-1 proves there is no live connection to any primary, yet the node skips holding its own election anyway. ctrl-2/ctrl-3 are simultaneously stuck in a permanent ~3s retry loop ('cluster_init_phase2: cluster_config not loaded yet', rc=-22, the bug-1477 self-rescheduling retry firing forever without ever succeeding). The cluster never elects a new, live, functioning primary after the very first election of the cluster's lifetime.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

myapp_run()'s post-stagger 'is a leader already known' bootstrap check (myapp.c ~361-384) has two independent fallback conditions that can each unilaterally set leader_known=1: c.ha.raft.term > 0 and c.ha.primary_ctrl_fd >= 0. The comment's stated intent (lines 345-353) is to catch a narrow race -- 'iomgr was NOT running during the nap ... but the winner's ctrlhello has already established the control channel' -- i.e. genuine NEW raft activity processed during THIS boot's startup-stagger nanosleep. But ha_restore_term() (bug-1684's neighbor, unmodified by that fix) restores ha->term from ha-term.bin BEFORE this check ever runs, as part of normal boot -- so raft.term > 0 is trivially true on EVERY restart after the cluster's first-ever election in its entire lifetime, regardless of whether any NEW raft traffic was processed this boot at all. The check cannot distinguish 'term advanced because I just received a real, fresh vote/heartbeat during this nap' (genuine liveness evidence) from 'term is nonzero because I loaded an old value off disk at startup' (tells us nothing about whether ANY leader is currently reachable). Consequence: after the cluster's first election, EVERY node, on EVERY subsequent restart, unconditionally takes the goto skip_election fast path and never calls ha_start_election_async again -- so no node ever re-asserts leadership through a real raft election, and the cluster permanently loses the ability to elect a NEW primary after any restart, however far in the past the original election was. This is a related-but-distinct gap from bug-1477 (which fixed a stranded node never RETRYING cluster_init_phase2 after one failed attempt -- that retry loop is confirmed still firing correctly here, every ~3s, exactly as bug-1477 designed -- but the retry can never succeed because the underlying cause, no election is ever held to produce a real live primary to push config from, was never addressed).

Fix

FIXED (2026-08-25). Root cause confirmed exactly as hypothesized: myapp_run()'s raft.term>0 fallback (myapp.c ~372) is trivially true on every restart after the cluster's first-ever election, since ha_restore_term() (called from cluster_init_phase1, strictly before ha_init_raft()/the startup-stagger nap) restores a persisted, monotonically-nondecreasing term from ha-term.bin, and Raft terms never reset.

Fix: extracted the inline check into a new, unit-testable function myapp_leader_already_known(ha, my_name, term_at_boot) -- placed in startup_validation.c (NOT myapp.c) purely so it links into controller_unit_test, which meson's controller_unit_test_src deliberately excludes myapp.c/main.c from (myapp_run is an untestable integration entry point requiring a live network socket) but already builds startup_validation.c for the sibling validate_* helpers -- myapp.c now just forward-declares and calls it, mirroring the pre-existing validate_startup_config() precedent exactly. myapp_run() captures term_at_boot = c.ha.raft.term immediately after ha_init_raft() seeds ha->raft.term from the just-restored ha->term -- strictly BEFORE the startup-stagger nap -- and the fallback check now requires ha->raft.term > term_at_boot (a genuine ADVANCE during THIS boot's nap, i.e. real vote-request/heartbeat traffic actually processed by iomgr via raft_on_vote_request's/raft_on_append_entries_request's UpdateTerm steps) instead of the old bare > 0. The voted_for_name/vft==cur check and the primary_ctrl_fd>=0 check are UNTOUCHED, per the task's scope boundary.

Verification of the 5 required questions:

1. Ordering confirmed via direct source read: ha_restore_term() runs inside cluster_init_phase1 (myapp_run Step 1, ~line 67), strictly before ha_init_raft() (~line 327) and the stagger nanosleep (~line 333-343) -- on every boot including restarts, confirmed via a live log line 'HA: restored term=0 from .../ha-term.bin' printed inside cluster_init_phase1, always preceding 'HA: raft initialized (...term=N...)' from ha_init_raft.
2. No other path can legitimately advance ha->raft.term between the snapshot and the check in a way the fix would wrongly treat as stale: the only writer after ha_init_raft() is raft_cb_persist, fired only from raft_on_vote_request / raft_on_append_entries_request / raft_start_election -- all driven by genuine incoming or self-initiated raft traffic processed by iomgr during the nap, exactly the signal the check intends to detect (confirmed by reading raft.c's UpdateTerm steps in full).
3. First-ever-boot preserved: ha_restore_term() leaves term at 0 with no ha-term.bin, so term_at_boot=0, identical operative comparison to the old raft.term > 0 check on a true first boot -- covered by 2 new regression tests.
4. No new safety-relevant failure mode: the 'spurious own election while a leader is genuinely live but unobserved' race this fix re-exposes on every restart is the IDENTICAL race the cluster's first-ever boot already ran through before this fix existed (term_at_boot was always 0 pre-first-election) -- not a new class of risk, and Election Safety is unaffected (rests entirely on votedFor + majority inside the untouched raft library, never on this advisory bootstrap check).
5. voted_for_name/vft==cur deliberately left untouched per scope, but flagged in the code comment as NOT obviously immune to the same class of question (a restored (voted_for_term,term) pair is exactly as internally consistent as a live one). Investigated empirically via the live soak below rather than touched speculatively: in both the baseline and fixed runs, this path never fires for ctrl-2/ctrl-3 in the --chaotic scenario at all, because the driver only kills/restarts the current primary -- ctrl-2/ctrl-3 run continuously and never re-enter myapp_run(), so their eventual takeover goes entirely through the separate, pre-existing, untouched liveness-watchdog/PreVote/RDHUP-failover subsystem (ha_on_primary_ctrl_fd_rdhup / ha_arm_heartbeat_timeout_vtask), not through this bootstrap check. No compounding defect was exercised or found; flagged as a residual, unverified theoretical concern for a scenario this soak doesn't cover (a STANDBY node itself restarting after remembering a now-permanently-dead remote leader).

TDD: RED confirmed by extracting the function with the OLD raft.term > 0 logic first (marked with a RED/GREEN comment), adding 7 new tests to test_ha_startup.c (test_bug1695_stale_restored_term_does_not_skip_election is the core regression), and rebuilding -- 2 of the 7 genuinely FAILED ('Expected 0 Was 1'), exactly as predicted (the self-vote test also failed under the old logic, since raft.term>0 trivially short-circuited it too -- an incidental but correct extra confirmation of the bug's reach). Applied the one-line semantic fix (> 0 -> > term_at_boot); rebuilt -- all 7 new tests GREEN, full controller suite at 1191 tests with exactly the 2 pre-existing bug-1561 failures (test_conf_trim_keeps_only_n_versions, test_conf_get_saved_tag_no_config -- identical Expected/Was values to bug-1561's own documented signature, 0 new failures).

MANDATORY live soak (6-round --chaotic, fresh ASAN binaries) -- run TWICE for a rigorous A/B: once with the fix, once with myapp.c/startup_validation.c/test_ha_startup.c git-stashed back to the pre-fix original (same binaries otherwise, same scratch-dir mechanics). BASELINE (unfixed): term frozen at 4 for the ENTIRE 6-round run (never advances even once after initial cold-boot settling); 'NO new primary within window' on 6/6 rounds; ctrl-2/ctrl-3's logs show ZERO CTRL_PRIMARY/PreVote-round activity anywhere in their entire lifetime -- a real, direct confirmation of the reported permanent-stall symptom. FIXED: term genuinely progresses 1->2->3->4->7 across the run; ctrl-2 (a DIFFERENT node from the original ctrl-1) achieved a real, log-verified CTRL_PRIMARY transition via a won PreVote round + real election at term=3 -- something that never happens even once anywhere in the baseline's logs. This is direct, unambiguous proof the core bug-1695 symptom (permanent inability to elect a NEW primary after any restart, once the cluster has ever held its first election) is fixed. committed_divergence (walcompare, the authoritative WAL-level check) stayed 0 in BOTH runs.

Residual, honestly-reported finding: the FIXED run's chaotic verdict was committed_job_loss=6, result=FAIL (baseline was committed_job_loss=0, result=PASS -- matching this task's own prediction that the baseline passes trivially by never actually re-electing anyone). Root-cause investigation (waldump on all 3 nodes' raw WAL files): the SAME single jobid ('0-15-1787671840-0') was reported missing on every affected round, and does NOT appear in ANY of the 3 nodes' on-disk WALs at all -- it traces to the cluster's very first cold-boot settling window (timestamp ~1787671840, well before round 1), during which ctrl-1's FIRST-ever election (with peers still config-less, config_tag=0) produced ambiguous vote-eligibility/config-bootstrap behavior unrelated to this fix (confirmed via ctrl-1's own log: its very first boot conceded to CTRL_STANDBY, never wrote that job to its own WAL). This strongly resembles a pre-existing, already-documented false-positive class (bug-1487: 'reachable primary reports job missing... post-election own-term commit-barrier lag while a peer is still catching up' -- verify_classify's own in-code comment names this exact class), newly and more frequently exposed because this fix, for the first time, lets the chaotic driver's overlapping-kill design exercise GENUINE repeated elections at all (the baseline's frozen-forever cluster never gave bug-1487's window a chance to occur more than once, if ever). NOT independently re-fixed in this session (out of this fix's myapp.c-only scope, and .claude/skills/xce-test-ha/run-ha-soak.sh is test-harness tooling, not controller code) -- flagged here as a genuine, needs-owner follow-up: either extend bug-1487's verify_classify retry margin further, or investigate whether the initial cold-boot config-bootstrap vote-eligibility path can genuinely lose a pre-config-push submission. Confidence in the CORE fix remains high (direct log proof of a new node winning a real election; walcompare clean in both runs); confidence in the chaotic verdict's exact job-loss root cause is medium (waldump-verified absence + a highly plausible, precedented explanation, but not traced to a specific line the way the core fix was).

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1477, bug-1684, bug-1487