lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(buglog): bug-1914, bug-1917, bug-1918 fixed; refresh STATUS.md

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-04 23:59:29 +00:00
parent 06e99b9081
commit 8e66d463ea
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 67 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

19
.wolf/bugs/bug-1914.md
View file

@ -1,8 +1,8 @@
---
id: "bug-1914"
status: "open"
status_updated: "2026-10-03"
status_note: "newly logged"
status: "fixed"
status_updated: "2026-10-04"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-03T15:07:26Z"
@ -11,6 +11,7 @@ occurrences: 1
files: ["src/controller/config_repl.c", "src/controller/config_repl.h"]
tags: ["ha", "config-push", "timeout", "load"]
related: ["bug-1485", "bug-1913"]
fix_commits: ["06e99b90"]
---
# bug-1914: config push fails -110 under heavy CPU load: commit lands just after the 500 ms barrier
@ -42,15 +43,17 @@ Error: controller returned status -110
- Evidence: not kept (scratchpad loop logs from the bug-1485 investigation)
## Root cause
TODO: required for fixed/partial
`config_commit_change` appends the `WAL_CONFIG_MARKER` first and then waits for the out-of-band byte-delivery quorum (`repl_wait_majority_ack(..., CONF_PUSH_TIMEOUT_MS)`, configAck branch). `CONF_PUSH_TIMEOUT_MS` was 500 ms, which is below ordinary scheduling delay on an oversubscribed host: with 40 busy loops on 32 cores a standby's ACK arrived ~575 ms in, so the push was rejected with -ETIMEDOUT although the majority was healthy. The marker, already in the log, still committed and activated, so the client was told the push failed for a change the cluster applied (the reply text even said "upgrade not committed"). The configAck wait was also a raw condition wait on a vtask outside the commit-wait admission reservation that ha-async-io.md requires for bounded blocking there.
## Fix
TODO: required for fixed/partial
- `CONF_PUSH_TIMEOUT_MS` 500 -> 2000 ms (src/controller/config_repl.h).
- The configAck wait in `repl_wait_majority_ack` now holds a commit-wait admission slot for its duration (new `commit_blocking_admit`/`commit_blocking_release` in commit.c, the same budget `commit_submit` WAIT uses); a full budget rejects fast with -XCE_ERR_NOT_READY, so the longer bound can never take workers reserved for liveness traffic.
- The timeout reply no longer claims "upgrade not committed": it says the change was not confirmed on a majority, may still activate if it was already ordered in the log, and should be re-pushed to confirm.
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
- Test: `test_configack_late_ack_under_load_meets_barrier` (src/controller/testsuite/test_commit_barrier_rollout.c): a config ACK arriving 700 ms in must meet the production bound, and the wait must hold one admission slot (released afterwards). RED before the fix (-110 at 500 ms), GREEN after.
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
- Live: `test_live_config_push_against_primary_reaches_success_path` looped 80 times under 40 busy loops on 32 cores, 2026-10-04: 80/80 passed, no -110 (before the fix: 1/70 failed with -110 under the same load).
## References
- Related: none