docs(buglog): bug-1914, bug-1917, bug-1918 fixed; refresh STATUS.md
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
06e99b9081
commit
8e66d463ea
GPG key ID:
80533536B0EB651C
5 changed files with 67 additions and 50 deletions
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
id: "bug-1914"
|
||||
status: "open"
|
||||
status_updated: "2026-10-03"
|
||||
status_note: "newly logged"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-04"
|
||||
status_note: ""
|
||||
component: "controller"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-03T15:07:26Z"
|
||||
|
|
@ -11,6 +11,7 @@ occurrences: 1
|
|||
files: ["src/controller/config_repl.c", "src/controller/config_repl.h"]
|
||||
tags: ["ha", "config-push", "timeout", "load"]
|
||||
related: ["bug-1485", "bug-1913"]
|
||||
fix_commits: ["06e99b90"]
|
||||
---
|
||||
|
||||
# bug-1914: config push fails -110 under heavy CPU load: commit lands just after the 500 ms barrier
|
||||
|
|
@ -42,15 +43,17 @@ Error: controller returned status -110
|
|||
- Evidence: not kept (scratchpad loop logs from the bug-1485 investigation)
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
`config_commit_change` appends the `WAL_CONFIG_MARKER` first and then waits for the out-of-band byte-delivery quorum (`repl_wait_majority_ack(..., CONF_PUSH_TIMEOUT_MS)`, configAck branch). `CONF_PUSH_TIMEOUT_MS` was 500 ms, which is below ordinary scheduling delay on an oversubscribed host: with 40 busy loops on 32 cores a standby's ACK arrived ~575 ms in, so the push was rejected with -ETIMEDOUT although the majority was healthy. The marker, already in the log, still committed and activated, so the client was told the push failed for a change the cluster applied (the reply text even said "upgrade not committed"). The configAck wait was also a raw condition wait on a vtask outside the commit-wait admission reservation that ha-async-io.md requires for bounded blocking there.
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
- `CONF_PUSH_TIMEOUT_MS` 500 -> 2000 ms (src/controller/config_repl.h).
|
||||
- The configAck wait in `repl_wait_majority_ack` now holds a commit-wait admission slot for its duration (new `commit_blocking_admit`/`commit_blocking_release` in commit.c, the same budget `commit_submit` WAIT uses); a full budget rejects fast with -XCE_ERR_NOT_READY, so the longer bound can never take workers reserved for liveness traffic.
|
||||
- The timeout reply no longer claims "upgrade not committed": it says the change was not confirmed on a majority, may still activate if it was already ordered in the log, and should be re-pushed to confirm.
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
- Test: `test_configack_late_ack_under_load_meets_barrier` (src/controller/testsuite/test_commit_barrier_rollout.c): a config ACK arriving 700 ms in must meet the production bound, and the wait must hold one admission slot (released afterwards). RED before the fix (-110 at 500 ms), GREEN after.
|
||||
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
|
||||
- Live: `test_live_config_push_against_primary_reaches_success_path` looped 80 times under 40 busy loops on 32 cores, 2026-10-04: 80/80 passed, no -110 (before the fix: 1/70 failed with -110 under the same load).
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
id: "bug-1917"
|
||||
status: "investigating"
|
||||
status_updated: "2026-10-03"
|
||||
status_note: "code-review finding from bug-1916; watch loaded soaks for reconnect stacks without free_packet"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-04"
|
||||
status_note: ""
|
||||
component: "lib"
|
||||
severity: "crash"
|
||||
timestamp: "2026-10-03T22:55:30Z"
|
||||
|
|
@ -11,6 +11,7 @@ occurrences: 1
|
|||
files: ["src/lib/xcectx.c", "src/lib/netio.c"]
|
||||
tags: ["race", "failover", "client", "ssl"]
|
||||
related: ["bug-1916"]
|
||||
fix_commits: ["431a6d70"]
|
||||
---
|
||||
|
||||
# bug-1917: xce_ctx_reconnect re-inits the endpoint on the caller thread while the io_worker may still use it
|
||||
|
|
@ -39,15 +40,15 @@ UPDATE (2026-10-03): Review of the bug-1916 fix (2026-10-03) adds a second hazar
|
|||
- Evidence: code reading of src/lib/xcectx.c xce_ctx_reconnect and src/lib/netio.c endpoint_fini / handle_input / handle_output
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
`xce_ctx_reconnect` (src/lib/xcectx.c) ran `endpoint_fini`/`endpoint_init` on `ctx->ep` straight away, while work dispatched for the old connection could still be running: the io_worker inside its epoll batch (it holds an endpoint reference for the whole batch: `SSL_read`/`handle_output` on the fd and SSL being freed), and VTASK_IO reply tasks still queued on the context's pool (each holds a reference and reads `ep->owner` without a lock). A queued reply task of the old connection that ran after the reconnect found `ep->owner` re-armed to the context and was delivered to the new session.
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
`xce_ctx_reconnect` first calls `xce_ctx_quiesce_ep`: unmonitor the old fd (no further io_worker events), clear `ep->owner` under `ep->lock` (stale reply tasks are dropped; `vtask_entry` frees their packet), then wait until the endpoint's reference count is back at its base value (0 for the embedded client endpoint, 1 for a heap one), polling every 1 ms. The wait is bounded at 2000 ms: if the caller is itself a task of this endpoint its own reference never drains, and the reconnect then proceeds as before with a WARN ("reconnecting anyway") rather than deadlocking. `vtask_entry` now reads `ep->owner` under `ep->lock`.
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
- Test: `test_reconnect_drains_old_connection_tasks` (new src/lib/testsuite/test_ctx_reconnect_quiesce.c): reconnect to a local listener, queue an old-connection packet via INVOKE_MSG_HANDLER -> handle_msg with the pool stopped, reconnect again while a helper starts the pool 100 ms in. RED before the fix (the stale task reached the context's request handler: expected 0 calls, was 1), GREEN and LSan-clean after.
|
||||
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
|
||||
- Live: loaded `/xce-test-ha` sequential 30 rounds (40 busy loops on 32 cores), 2026-10-04: PASS, zero job loss, all 3 nodes led, no ASAN report in any controller/worker/scheduler log; client failover reconnects in worker/scheduler/xce_test throughout, and the quiesce never needed its timeout (no "reconnecting anyway" WARN).
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
id: "bug-1918"
|
||||
status: "open"
|
||||
status_updated: "2026-10-03"
|
||||
status_note: "newly logged"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-04"
|
||||
status_note: ""
|
||||
component: "lib"
|
||||
severity: "leak"
|
||||
timestamp: "2026-10-03T23:04:08Z"
|
||||
|
|
@ -11,6 +11,7 @@ occurrences: 1
|
|||
files: ["src/lib/xcectx.c", "src/lib/vtaskmgr.c", "src/controller/cluster.c"]
|
||||
tags: ["leak", "vtask", "netio"]
|
||||
related: ["bug-1916"]
|
||||
fix_commits: ["a74b684c"]
|
||||
---
|
||||
|
||||
# bug-1918: handle_msg / vtask_handle_msg early -EINVAL/-ENOMEM exits leak the received packet
|
||||
|
|
@ -37,15 +38,15 @@ Not yet observed at runtime -- found reviewing the bug-1916 fix. When `handle_ms
|
|||
- Evidence: code reading of the cited lines
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
`INVOKE_MSG_HANDLER` (src/include/netio.h) stored NULL into `ep->fsm_in.pkt` after every `msg_handler` call, on the old assumption that the handler always takes the packet. Since bug-1916 and raft-drive-async 3.1, a handler that takes the packet clears the slot itself (`vtask_handle_msg`, `peer_ctrl_msg_handler`). A handler that refuses it (`handle_msg` with no owner -> -EINVAL; `vtask_handle_msg` on an incomplete fsm -> -EINVAL or a failed task allocation -> -ENOMEM) leaves the packet in the slot, by the contract `test_handoff_rejects_incomplete_fsm` already pins. The macro's NULL store then dropped the only reference: the packet leaked.
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
Drop the post-handler `ep->fsm_in.pkt = NULL` from `INVOKE_MSG_HANDLER`. A refused packet stays endpoint-owned: `handle_input` reuses it for the next message and `endpoint_fini` frees it. All three production msg handlers already clear the slot when they take the packet, so no path frees it twice. The ref/unref bracket (bug-1298) is kept.
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
- Test: `test_refused_packet_stays_endpoint_owned` (src/lib/testsuite/test_vtask_pkt_handoff.c) drives the macro with a refusing handler, then with a taking handler, then endpoint_fini. RED before the fix (slot NULL after a refusal; LeakSanitizer: 88 bytes leaked from encode_nop), GREEN and LSan-clean after.
|
||||
- Suites: `xce-build.sh all` asan, 127/127 meson suites OK, Go tests ok (2026-10-04).
|
||||
- Live: loaded `/xce-test-ha` sequential 30 rounds (40 busy loops on 32 cores), 2026-10-04: PASS, zero job loss, all 3 nodes led, no ASAN report in any controller/worker/scheduler log.
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue