myxce/.wolf/bugs/bug-1475.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1475	fixed	2026-09-24		controller	unknown	2026-07-26T04:36:00Z	2026-09-19T09:00:00Z	1	src/controller/ha.c (ha_dispatch_repl_action_for_standby ha_consume_append_entries ha_evaluate_repl_action — HA_REPL_STREAM_WAL decision/dispatch loop); src/controller/config_repl.c (ha_stream_wal_to_standby catch-up streamer)	ha.c config_repl.c ha_dispatch_repl_action_for_standby ha_stream_wal_to_standby ha_consume_append_entries ae_reject_logmismatch catch-up-stall write-quorum-loss standby-liveness config-upgrade-snapshot-task-8.6 force_elect_node not-bug-1472 bug-1472-fix-verified-good same-family-as-bug-1028 needs-owner root-caused-partially	bug-1472 bug-1028	true

bug-1475: Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade), run to verify…

Description

Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade), run to verify bug-1472's fix, FAILED — but NOT at bug-1472's own assertions (materialized-state checkpoint byte-identity and active-configuration identity both PASSED cleanly across all 3 members, confirming bug-1472's fix holds). Failure occurred one step later, inside the Requirement 6.4 host-admin-status-rotation check's own force_elect_node("ctrl-1") maneuver: after 3 rounds of overlapping chaotic kills + a live config upgrade all completed and the cluster reconverged 3-of-3, force_elect_node killed the spoiler (ctrl-3) and tried to submit a job batch on the surviving {ctrl-2 primary, ctrl-1 standby} pair — 5 consecutive attempts all got 0/3 acked ("waiting for write-quorum"), and the round hard-failed: [xce-test-ha] FAIL: force-elect ctrl-1: submit on {ctrl-2,ctrl-1} failed. The Req 6.4 workerlist/admin-status assertion itself was never reached.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

NOT data loss, NOT bug-1472 (worker/host admin-status replication is unaffected — unrelated subsystem, and its own checks passed). This is a standby catch-up LIVENESS stall: ctrl-1 (which had itself served as primary in term=2 before being killed in round 1 of the chaotic sequence, then restarted) got permanently stuck at applied_lsn=12 and never advanced again for the rest of the run. ctrl-1's own log (ha_consume_append_entries) shows a revealing pattern: a zero-entry PROBE AppendEntries at prevIdx=12,n=0 correctly succeeds with match=12 (confirming index 12 IS ctrl-1's true, correct log tip), but the very next entry-carrying AppendEntries from the SAME dispatch cycle uses prevIdx=13 (not 12) for n=1, which ctrl-1 correctly rejects with event=ae_reject_logmismatch (success=0 match=0) since it has no entry at index 13 yet to check a predecessor against. This 1-off prevIdx computation (using prevIdx=standby_applied_lsn+1 instead of prevIdx=standby_applied_lsn for the FIRST entry of a catch-up batch) means every retry of the SAME heartbeat-driven HA_REPL_STREAM_WAL decision (ha_evaluate_repl_action: standby_lsn(12) < committed_lsn(31) -> stream) recomputes and resends the IDENTICAL doomed range, forever, with zero backtrack/self-correction -- confirmed via ctrl-2's log: 'HA repl: standby[0] lsn=12 < committed_lsn=31 -> stream WAL batches' repeats verbatim every ~3s from 04:31:26 through 04:32:26 (60+ seconds, 20+ identical retries, zero progress) after ctrl-2 had already logged 'repl_broadcast: standby localhost:9991 evicted (pending_ack_count=17 > 16) -- triggering catch-up reconnect' and 'HA quorum lost: connected=0 required=1'. Once the soak's own force_elect_node killed the OTHER standby (ctrl-3) as its spoiler maneuver, ctrl-1 became the sole remaining standby, and its permanent inability to catch up meant the primary could never regain 2-of-3 write quorum again -- every subsequent job submission timed out (commit: async continuation deadline expired, jobsubmit_commit_continuation: lsn=N commit failed/timed out (rc=-110)), exactly matching the soak's observed '0/3 acked; waiting for write-quorum' x5 before hard-failing. Superficially similar symptom signature to already-FIXED bug-1028 (same 'ha_dispatch_repl_action_for_standby: standby[0] lsn=X < committed_lsn=Y -> stream WAL batches' log line looping indefinitely post-failover) but a DIFFERENT specific AE-reject reason -- bug-1028 was a logOk TERM-mapping mismatch (raw hdr.term vs persist_entry_term legacy mapping), fixed 2026-06-23; this is ae_reject_logmismatch, which reads like a prevLogIndex/off-by-one issue in how the catch-up streamer picks the FIRST entry's predecessor index, not a term problem -- so this is either a narrower unfixed variant of the same family, or a new regression in a related code path. NOT root-caused down to the exact defective line (this investigation was a live-soak VERIFICATION run for a DIFFERENT bug, not a fix task) -- flagging for whoever picks this up next to read ha_dispatch_repl_action_for_standby/ha_stream_wal_to_standby's prevIdx computation directly and compare against ctrl-1's actual persisted log at lsn=12/13 (waldump). Also notable: this specific soak step (force_elect_node's spoiler-kill-then-submit maneuver, previously only exercised by --sequential mode's post-loop 'leadership coverage' step against an otherwise-calm, fully-settled cluster) had never previously been run immediately after --config-upgrade's THREE rounds of aggressive overlapping chaotic kills plus a live config upgrade -- this specific combination/history may be what's needed to reproduce the stall, rather than it being config-upgrade-specific.

Fix

FIXED (2026-07-27/28, via a regression test + real-instrumentation debugging session that continued this entry's original live-soak finding). Landed as THREE coordinated changes, all required together -- the first two alone left the new regression test (test_bug1475_catchup_survives_midstream_checkpoint, test_config_repl.c) RED with applied_lsn stuck at 0 for the whole run:

1. SEND side (config_repl.c, ha_stream_wal_to_standby): when scanning the WAL for entries to stream, a WAL_CHECKPOINT entry is still never built/sent as its own catchup entry, but the prev_log_index/prev_log_term tracker now advances across it like every other scanned entry, instead of freezing one entry behind. A standby that independently applied the same committed WAL_CONFIG_MARKER writes its OWN local checkpoint at the identical LSN (deterministic per-member trigger), so the checkpoint's LSN is a real, held position on both sides, not a gap to skip past chaining-wise.

2. RECEIVE side (ha.c, ha_apply_term_entry_via_library): a JIT resync reads persist_durable_floor(p) and monotonically raises ha->raft.log_len to match before consuming each AppendEntries -- mirroring ha_sync_raft_log_from_recovery's boot-time computation, but applied live on the AE-receive path. This corrects for persist_write_checkpoint_marker (bug-1466, still not fixed at ITS root) advancing persist_t's current_lsn without going through raft_log_append, which otherwise leaves ha->raft.log_len stale and fails logOk's prev_log_index<=log_len bound forever.

3. TERM-INDEX side (persist.c, record_entry_term/entry_term_locked) -- a THIRD, previously-undiscovered gap, found only via live fprintf instrumentation of the new regression test after fixes 1-2 above still left it RED. Fixes 1-2 both depend on persist_entry_term(checkpoint_lsn) reading back 'absent' (0) FOREVER on every member (the explicit, twice-documented design intent in both fixes' own comments and in persist_write_checkpoint_marker's comment). That invariant was actually FALSE: record_entry_term's geometric-growth realloc zero-filled any never-explicitly-recorded gap slot (like a checkpoint's own LSN, which is deliberately never passed to record_entry_term), and entry_term_locked's lsn <= term_index_count presence check then read that zero-fill back as WAL_LEGACY_TERM (1), not absent (0), the INSTANT any LATER lsn was recorded and grew term_index_count past the gap -- which happens immediately on a live leader (it appends real entries right after the checkpoint) but not yet on a receiver that has not itself appended past its own checkpoint. So the checkpoint LSN's 'term' was TIME-DEPENDENT: the sender (which streams AFTER its own later entries exist) computed prev_log_term=1 for the entry right after the checkpoint, while the receiver's honest, not-yet-grown term index still (correctly, per design) read 0 for the same LSN -- logOk's equality check spuriously mismatched (0 != 1) and rejected forever, exactly reproducing this entry's original ae_reject_logmismatch signature. Confirmed with real fprintf instrumentation at the streamer, the JIT resync, and raft.c's logOk check (added and removed during this investigation, not left in the tree): prev_log_term flipped from 0 to 1 in the SAME test run, immediately after the leader appended its 2nd post-checkpoint entry. Fixed with a TERM_INDEX_UNSET sentinel (UINT64_MAX) that record_entry_term's growth path now fills gap slots with (instead of 0), and entry_term_locked treats as absent -- so a never-explicitly-written slot reads absent PERMANENTLY, independent of later array growth. Confined to persist.c's two functions plus the term_index doc comment in persist.h; no other call site reads p->term_index[] directly.

Also completed as part of this fix: (a) audited whether WAL entry types OTHER than WAL_CHECKPOINT bypass record_entry_term the same way -- confirmed NO: WAL_CHECKPOINT (value 100) is the numeric max of wal_entry_type, persist_wal_async and persist_wal_write_replicated both explicitly reject type==WAL_CHECKPOINT up front (it cannot reach record_entry_term via either live path), and replay_wal's hdr.type < WAL_CHECKPOINT recording guard already covers every other currently-defined type (WAL_JOB_UPDATE=7, WAL_QUEUE_CTRL=20, WAL_WORKER_CTRL=30, WAL_NOOP=40, WAL_CONFIG_MARKER=50); (b) confirmed no interaction with the RequestVote/PreVote freshness gates (raft.c raft_log_ok, shared by raft_on_vote_request/raft_on_pre_vote_request): that gate also reads on_get_entry_term(s->log_len), which can legitimately land on a checkpoint LSN after the JIT resync in fix 2 -- the TERM_INDEX_UNSET fix makes that read deterministic (always absent) instead of time-dependent, which is a consistency IMPROVEMENT for vote freshness too, not a behavior change in the unsafe direction; full test_ha_prevote.c/test_ha_failover_prevote.c suites pass unchanged. Full regression suite: 1034 Tests, 0 Failures (was 1033/1034 with the new test failing before this fix). Files changed: src/controller/config_repl.c (REAL logic change, landed earlier in this same bug-1475 investigation: ha_stream_wal_to_standby's WAL_CHECKPOINT branch now falls through to the unconditional prev-tracker advance via hdr.lsn > standby_lsn && hdr.type != WAL_CHECKPOINT, instead of a hard continue that froze the tracker one entry behind -- plus cross-reference comments added in this later session); src/controller/ha.c (REAL logic change, also landed earlier in this investigation: a ~30-line JIT resync block added to the top of ha_apply_term_entry_via_library that raises ha->raft.log_len from persist_durable_floor(p) before consuming each AppendEntries -- plus cross-reference comments added in this later session); src/controller/persist.c (the TERM_INDEX_UNSET sentinel fix + doc comments, the new logic landed in THIS session); src/controller/testsuite/test_config_repl.c (test_bug1475_catchup_survives_midstream_checkpoint + updated doc comments describing all three fix parts). Correction (coordinator review 2026-07-27/28, Finding 1): an earlier version of this fix summary mischaracterized config_repl.c's and ha.c's changes as 'comment-only, no logic change' -- both carry genuine logic changes; they were simply written in an earlier part of this same bug-1475 investigation than the TERM_INDEX_UNSET sentinel (session 3 of the fix), not left unmodified by it.

Finding 2 (coordinator review, 2026-07-27/28, CONFIRMED REAL and FIXED): after the TERM_INDEX_UNSET fix, the reviewer traced a plausible NEW instance of the identical permanent-wedge bug class, reachable via the SNAPSHOT-INSTALL catch-up path (HA_REPL_SEND_SNAPSHOT, dispatched whenever a standby has fallen too far behind for incremental WAL streaming -- a live, common path) instead of WAL-streaming catch-up. ha_snapshot_do_send (ha.c) computes checkpoint_term = persist_entry_term(leader, c->persist.checkpoint_lsn) and ships it on the wire; the receiver's ha_handle_snapshot_chunk calls persist_reconcile_checkpoint_term(p, checkpoint_lsn, checkpoint_term), which called record_entry_term(p, checkpoint_lsn, checkpoint_term) directly -- an EXPLICIT write, not a growth-fill, so it never got the TERM_INDEX_UNSET sentinel even when checkpoint_term==0. entry_term_locked's pre-existing legacy-0 rule then read that raw 0 back as WAL_LEGACY_TERM(1), forever -- permanently diverging from the leader's own permanently-absent(0) value for the SAME lsn whenever c->persist.checkpoint_lsn (== last_applied_lsn at general persist_checkpoint() time, NOT the WAL_CHECKPOINT entry's own physical lsn) happens to coincide with a checkpoint's own un-recorded lsn -- a real, reachable quiescent-state timing (the apply pump has walked, one lsn at a time including a no-op pass over checkpoint slots, all the way to exactly the checkpoint's own position before the next persist_checkpoint()+snapshot-send cycle runs, e.g. right after a config upgrade with no immediate follow-up activity) -- not the common case, but real and worth closing. Proven with a dedicated regression test (test_bug1475_finding2_snapshot_install_checkpoint_term_absent, test_config_repl.c) that drives the REAL ha_send_snapshot_to_standby -> ha_handle_snapshot_chunk -> persist_reconcile_checkpoint_term path (via the xce_snapshot_chunk_hook test seam) with a leader whose checkpoint_lsn is deliberately seeded (via last_applied_lsn) to coincide with its checkpoint's own physical lsn -- confirmed RED by temporarily reverting only persist_reconcile_checkpoint_term's fix (receiver read back term=1, not the expected absent=0) and GREEN after restoring it. Fixed with the SAME TERM_INDEX_UNSET sentinel: persist_reconcile_checkpoint_term now substitutes TERM_INDEX_UNSET for a wire checkpoint_term of exactly 0 before calling record_entry_term, instead of recording the raw 0 -- a wire value of 0 is unambiguous today (ha_snapshot_do_send is the ONLY sender and always computes it via persist_entry_term, whose public contract already returns a DIFFERENT value, WAL_LEGACY_TERM, for a present-but-legacy-0 entry -- so a public 0 can only mean genuinely, permanently absent). A genuinely non-zero checkpoint_term (any real historical term) is still recorded verbatim and still correctly overwrites a stale local term at that LSN, unchanged -- the pre-existing STALE-TERM reconcile test (test_ha_core_safety.c's test_3_1_snapshot_install_does_not_reconcile_boundary_term, CHECKPOINT_TERM=7, a synthetic non-zero value) is unaffected and still passes. Also fixed the stale 'zeroed-on-grow' comment at persist.c's truncate_term_index_locked (grown slots are TERM_INDEX_UNSET-filled since the original TERM_INDEX_UNSET fix, not zero-filled -- comment had not been updated). Re-verified RequestVote/PreVote freshness non-interference for this second fix too: persist_reconcile_checkpoint_term's caller (ha_handle_snapshot_chunk) never touches ha->raft.log_len or any vote-path state; the fix only changes what value entry_term_locked returns for a checkpoint-boundary LSN after a snapshot install (now consistently absent instead of inconsistently WAL_LEGACY_TERM), which is the SAME direction of improvement (deterministic absence vs time/path-dependent) the original TERM_INDEX_UNSET fix already established as safe for the freshness gate. Full suite: 1035 Tests, 0 Failures (was 1034 before this new test).

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1472, bug-1028