docs(buglog): bug-1683, bug-1485, bug-1725 fixed with corrected root causes; log bug-1913
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
362098b12a
commit
4cd5fb7dd8
5 changed files with 133 additions and 37 deletions
59
.wolf/bugs/bug-1913.md
Normal file
59
.wolf/bugs/bug-1913.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
id: "bug-1913"
|
||||
status: "open"
|
||||
status_updated: "2026-10-03"
|
||||
status_note: "newly logged"
|
||||
component: "controller"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-03T14:18:58Z"
|
||||
last_seen: "2026-10-03T14:18:58Z"
|
||||
occurrences: 1
|
||||
files: ["src/controller/ha.c", "src/controller/repl.c", "src/controller/config_repl.c"]
|
||||
tags: ["ha", "raft", "replication", "latency"]
|
||||
related: ["bug-1485"]
|
||||
---
|
||||
|
||||
# bug-1913: after a late-joiner election the primary cannot commit client writes for ~2 heartbeat cycles
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
```
|
||||
repl_broadcast: no standbys registered ... not replicated
|
||||
...
|
||||
WAL NACK from standby lsn=2 ... marking catch-up
|
||||
repl_wait_majority_ack: timeout ... OP_upgradeConfig rejected (status=-110)
|
||||
```
|
||||
In some elections the vote connections are not converted into standby channels (`raft_cb_become_leader`), so both standbys register later through `on_standby_connected` (ha.c ~9643-9660). `ha_arm_own_term_gate` (ha.c ~5178) has already appended the own-term WAL_NOOP (lsn 1) while no standby was registered (repl.c ~306-311), and `repl_add_standby` does not resend it. Catch-up then runs from the 3 s heartbeat, one action per standby per cycle (config push first, WAL stream on the next cycle), so `committed_lsn` stays 0 for ~6 s. Any client write in that window (e.g. `OP_upgradeConfig`) is NACKed by both standbys and fails with -ETIMEDOUT after the 500 ms majority wait.
|
||||
|
||||
**Expected:** a newly elected primary commits its own-term no-op, and so accepts client writes, promptly after its standbys register, not two heartbeat cycles later.
|
||||
**Impact:** for ~6 s after such an election every client write is rejected with -110 (fail-loud, no data loss, no false success). Clients retry, but tests and operators see spurious failures right after failover (the root cause of bug-1485's versionTag=0 flake). A fix must respect the async-I/O rules (no blocking on the delaymgr/heartbeat path; streaming offloaded) in `.kiro/steering/ha-async-io.md`.
|
||||
**Found by:** test (root cause of bug-1485)
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ 91782e40-dirty
|
||||
- Deployment: local 3-node HA (xceadmin live e2e fixture), CPU-loaded sandbox (40 busy loops on 32 cores) makes late-joiner elections frequent
|
||||
|
||||
## Reproduction
|
||||
1. Start a 3-node HA cluster (e.g. the `live_ha_cluster` fixture in `src/xceadmin/tests/test_e2e_live_ha_push.py`) under CPU load.
|
||||
2. In the primary log, find an election with no "promoted" standby channels and a later "repl_add_standby"/late registration; push a config (`xceadmin config push ... --controller <primary>`) within ~6 s.
|
||||
3. The push fails with `controller returned status -110`; the log shows the NACK and majority-ack timeout above.
|
||||
- Frequency: 7/28 runs of the HA push test under 40-spinner CPU load, 0/6 idle (2026-10-03); all 7 failures were late-joiner elections.
|
||||
- Evidence: not kept (scratchpad loop logs from the bug-1485 investigation).
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue