docs(buglog): log bug-1914 config push -110 under heavy CPU load

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-03 15:08:01 +00:00

3.1 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1914: config push fails -110 under heavy CPU load: commit lands just after the 500 ms barrier

Description

Observed:

Error: controller returned status -110

xceadmin config push to a healthy, normally elected HA primary (standbys promoted from the vote connections, own-term no-op already committed) was rejected with -ETIMEDOUT: the config marker's majority commit landed ~75 ms after the repl_wait_majority_ack(..., CONF_PUSH_TIMEOUT_MS) barrier (config_repl.c ~1658; CONF_PUSH_TIMEOUT_MS = 500, config_repl.h:33) expired. Seen only with the CPU heavily oversubscribed (40 busy loops on 32 cores).

Expected: a push to a healthy primary with a reachable majority succeeds under load, or the barrier is sized/structured so normal scheduling delay cannot turn a soon-to-commit marker into a rejection. (A commit that lands after the client was told -110 also leaves the outcome ambiguous to the client: rejected per the reply, yet committed later.) Impact: spurious push failures under heavy load (fail-loud, no data loss). If the marker then commits after the -110 reply, the client believes the push failed while the cluster activated it -- a retry is idempotent per tag, but the reply is misleading. Check before changing: whether the late commit is actually applied after the -110 reply, and the async-I/O rules for the commit-wait barrier (COMMIT_WAIT_RESERVED, .kiro/steering/ha-async-io.md). Found by: test (bug-1485 investigation, 2026-10-03)

Environment

OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: asan (xce.build)
Code: scheduler-framework @ 4cd5fb7d-dirty
Deployment: local 3-node HA (xceadmin live_ha_cluster fixture), CPU oversubscribed with 40 busy loops on 32 cores

Reproduction

Start 40 CPU busy loops on a 32-core host.
Loop src/xceadmin/tests/test_e2e_live_ha_push.py (with the bug-1485 readiness wait in place).
Rarely, the push fails with controller returned status -110 on a primary whose own-term no-op had already committed; the primary log shows the marker commit ~75 ms after the 500 ms barrier expired.

Frequency: 1/70 HA push loops under heavy load, 0 idle (2026-10-03)
Evidence: not kept (scratchpad loop logs from the bug-1485 investigation)

Root cause

TODO: required for fixed/partial

Fix