myxce/.wolf/bugs/bug-1914.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1914	fixed	2026-10-04		controller	wrong-result	2026-10-03T15:07:26Z	2026-10-03T15:07:26Z	1	src/controller/config_repl.c src/controller/config_repl.h	ha config-push timeout load	bug-1485 bug-1913	06e99b90	341	fixed	71c6272fe0e6

bug-1914: config push fails -110 under heavy CPU load: commit lands just after the 500 ms barrier

Description

Observed:

Error: controller returned status -110

xceadmin config push to a healthy, normally elected HA primary (standbys promoted from the vote connections, own-term no-op already committed) was rejected with -ETIMEDOUT: the config marker's majority commit landed ~75 ms after the repl_wait_majority_ack(..., CONF_PUSH_TIMEOUT_MS) barrier (config_repl.c ~1658; CONF_PUSH_TIMEOUT_MS = 500, config_repl.h:33) expired. Seen only with the CPU heavily oversubscribed (40 busy loops on 32 cores).

Expected: a push to a healthy primary with a reachable majority succeeds under load, or the barrier is sized/structured so normal scheduling delay cannot turn a soon-to-commit marker into a rejection. (A commit that lands after the client was told -110 also leaves the outcome ambiguous to the client: rejected per the reply, yet committed later.) Impact: spurious push failures under heavy load (fail-loud, no data loss). If the marker then commits after the -110 reply, the client believes the push failed while the cluster activated it -- a retry is idempotent per tag, but the reply is misleading. Check before changing: whether the late commit is actually applied after the -110 reply, and the async-I/O rules for the commit-wait barrier (COMMIT_WAIT_RESERVED, .kiro/steering/ha-async-io.md). Found by: test (bug-1485 investigation, 2026-10-03)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 4cd5fb7d-dirty
• Deployment: local 3-node HA (xceadmin live_ha_cluster fixture), CPU oversubscribed with 40 busy loops on 32 cores

Reproduction

1. Start 40 CPU busy loops on a 32-core host.
2. Loop src/xceadmin/tests/test_e2e_live_ha_push.py (with the bug-1485 readiness wait in place).
3. Rarely, the push fails with controller returned status -110 on a primary whose own-term no-op had already committed; the primary log shows the marker commit ~75 ms after the 500 ms barrier expired.

• Frequency: 1/70 HA push loops under heavy load, 0 idle (2026-10-03)
• Evidence: not kept (scratchpad loop logs from the bug-1485 investigation)

Root cause

config_commit_change appends the WAL_CONFIG_MARKER first and then waits for the out-of-band byte-delivery quorum (repl_wait_majority_ack(..., CONF_PUSH_TIMEOUT_MS), configAck branch). CONF_PUSH_TIMEOUT_MS was 500 ms, which is below ordinary scheduling delay on an oversubscribed host: with 40 busy loops on 32 cores a standby's ACK arrived ~575 ms in, so the push was rejected with -ETIMEDOUT although the majority was healthy. The marker, already in the log, still committed and activated, so the client was told the push failed for a change the cluster applied (the reply text even said "upgrade not committed"). The configAck wait was also a raw condition wait on a vtask outside the commit-wait admission reservation that ha-async-io.md requires for bounded blocking there.

Fix

• CONF_PUSH_TIMEOUT_MS 500 -> 2000 ms (src/controller/config_repl.h).
• The configAck wait in repl_wait_majority_ack now holds a commit-wait admission slot for its duration (new commit_blocking_admit/commit_blocking_release in commit.c, the same budget commit_submit WAIT uses); a full budget rejects fast with -XCE_ERR_NOT_READY, so the longer bound can never take workers reserved for liveness traffic.
• The timeout reply no longer claims "upgrade not committed": it says the change was not confirmed on a majority, may still activate if it was already ordered in the log, and should be re-pushed to confirm.

Verification

• Test: test_configack_late_ack_under_load_meets_barrier (src/controller/testsuite/test_commit_barrier_rollout.c): a config ACK arriving 700 ms in must meet the production bound, and the wait must hold one admission slot (released afterwards). RED before the fix (-110 at 500 ms), GREEN after.
• Suites: xce-build.sh all asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
• Live: test_live_config_push_against_primary_reaches_success_path looped 80 times under 40 busy loops on 32 cores, 2026-10-04: 80/80 passed, no -110 (before the fix: 1/70 failed with -110 under the same load).

References

• Related: none