myxce/.wolf/bugs/bug-1919.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1919	fixed	2026-10-04		controller	wrong-result	2026-10-04T06:10:10Z	2026-10-04T06:10:10Z	1	src/controller/ha.c src/controller/persist.c src/controller/ctrlstatus.c	ha fence recovery catch-up soak		0ab12074	339	fixed	71fd97b09ba9

bug-1919: promotion fence never releases on a node that booted RECOVER_TRUNCATED, even after full catch-up

Description

Observed:

05:57:15,308 HA: promotion REFUSED (decision=1 ...) — recovery_truncated=1 durable_floor=360 cluster_hwm=360 peer_reachable=1; fencing + arming catch-up re-alert window, staying ALIVE
05:57:15,308 HA: promotion FENCED + catch-up re-alert window armed (30000 ms) — recovery_truncated=1 durable_floor=360 < cluster HWM=360
05:57:45,483 HA: promotion fence catch-up window elapsed but progress is possible (peer_reachable=0 durable_floor=375 baseline=360 HWM=360) — remaining fenced, catching up
[xce-test-ha] FAIL: leadership-coverage: cluster did not reconverge before maneuver

promotion_fence_timeout_fn (src/controller/ha.c ~1546-1640) releases the fence only if !truncated && !below_hwm && ..., where truncated = persist_recovery_truncated() = recovery_verdict == RECOVER_TRUNCATED (persist.c:7005) — the BOOT-time recovery verdict, which never changes at runtime. So for a node that booted truncated the release branch is unreachable however far it catches up (here durable_floor 375 >= HWM 360). In the "progress is possible" branch the timer frees itself and returns WITHOUT re-arming, so the fence is never re-evaluated. in_catch_up stays 1, ctrlstatus reports fenced: true (ctrlstatus.c:193) forever, and /xce-test-ha's cluster_committed_converged (a fenced member is not converged) never succeeds.

Expected: once the fenced member has caught up to a committed-consistent state (durable floor >= the cluster HWM the fence was armed with, as a standby of the live primary), the fence is released (ha-commit-durability Req 2.6) and the node is promotable and reported converged; the re-check keeps running until that happens. Impact: a controller that was killed as primary, rebooted with a truncated WAL tail, and later attempted promotion stays fenced (non-promotable, fenced: true) until restarted, even though it is fully caught up: the cluster loses one promotion candidate (in a 3-node cluster, the next failover may then have only one eligible node), and soak convergence gates fail. No data loss (fail-safe direction). Found by: soak (/xce-test-ha sequential 20 rounds, validating bug-1915, 2026-10-04)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ acf78f9a-dirty
• Deployment: local 3-node HA (/xce-test-ha sequential), worker + scheduler joined

Reproduction

1. XCE_HA_ROUNDS=20 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh 20 <scratch dir>.
2. Needs a node killed as primary that reboots with recovery_truncated=1 and later wins a PreVote; its promotion is refused (expected) and it is fenced.
3. After it catches up as a standby, promotion_fence_timeout_fn logs "progress is possible ... remaining fenced" and never runs again; ctrlstatus keeps fenced: true; the soak fails "cluster did not reconverge".

• Frequency: 1/4 soak runs, 2026-10-04 (timing-dependent: needs a truncated-boot node to attempt promotion)
• Evidence: ctrl-2 log kept in the session scratchpad as fence-ctrl-2.log; soak log s1915-seq.log

Root cause

Two defects in promotion_fence_timeout_fn (src/controller/ha.c):

1. The release branch requires !persist_recovery_truncated(), which read the boot-time recovery_verdict only. Nothing at runtime ever cleared it, so a member that booted RECOVER_TRUNCATED could never pass the release check, however far it caught up. The existing unit test test_promotion_fence_released_on_catchup hid this by writing recovery_verdict = RECOVER_CLEAN directly, which production never does.
2. The "progress is possible" branch (peer reachable or durable floor advanced) freed the timer ctx without re-arming, citing "existing catch-up machinery" that would release the fence. No such path exists (the only other releases are the election gate, which itself refuses a truncated node, and ha_evaluate_promotion_fence, which has no production caller on this path). So once that branch ran, the fence was never re-evaluated.

Fix

• persist_t.recovery_truncation_healed (src/include/persist.h): a runtime latch, reset by persist_init/persist_recover, set by the new persist_mark_recovery_truncation_healed(). persist_recovery_truncated() now returns verdict == RECOVER_TRUNCATED && !healed. persist_recovery_verdict() still returns the raw boot verdict for diagnostics.
• promotion_fence_timeout_fn: if the member is truncated AND durable_floor >= target_hwm AND durable_floor > baseline (contiguous entries replicated from the live primary on top of the truncated log; a fenced member is never primary), mark the truncation healed and fall through to the release check. Without progress nothing proves the log matches the primary's, so the fence stays (conservative).
• The "progress is possible" branch re-arms the timer with the SAME baseline and HWM, so re-evaluation continues until the release.

Verification

• Test: test_promotion_fence_truncated_boot_released_after_catchup and test_promotion_fence_rechecks_while_catching_up (src/controller/testsuite/test_commit_durability.c) were RED before the fix (fence never released) and GREEN after; test_promotion_fence_truncated_no_progress_stays_fenced pins the conservative no-progress rule (GREEN before and after).
• Suites: xce-build.sh all asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
• Live: 4x /xce-test-ha sequential 20 rounds, 2026-10-04. Run 2 hit the exact path: ctrl boot RECOVER_TRUNCATED → promotion REFUSED (durable_floor=323, HWM=323) → 30 s later "recovery-truncated log healed by catch-up — durable_floor=338 (baseline=323)" + "promotion fence RELEASED", and the soak PASSed (all 3 nodes led). Runs 1 and 4 PASS (path not hit). Run 3 FAILed on an unrelated replication catch-up stall (logged separately, no fence/truncation in that run).

UPDATE (2026-10-08): Repeatable live coverage (2026-10-08, 5cf92335): run-ha-soak.sh --truncated-fence breaks ctrl-1's WAL tail CRC (RECOVER_TRUNCATED boot), makes it win an election (promotion REFUSED), and requires 'healed by catch-up' + 'promotion fence RELEASED' and then leadership: PASS -- refused on attempt 1, fence released after 30 s (the re-check window), healed ctrl-1 led at term 9, zero job loss.

References

• Related: none