feat(raft-drive-async): offload delaymgr peer reconnect to slow lane (task 2.1)
Heartbeat timer detects a disconnected peer, CAS-acquires a per-peer single-flight flag, slowlane_submits the bounded connect, and returns — the delaymgr never blocks on connect. Lane worker (reconnect_lane_fn) does the bounded endpoint_connect + lock-guarded iomgr_monitor; flag cleared on all terminal paths (incl. -EEXIST race fix). Default-ON kill-switch retains the inline path for TDD-RED/fallback. Never self-terminates on failure. 2.2 adds the inbound/outbound race + failure hardening (documented seam). 196/0 green.
This commit is contained in:
parent
a36234a103
commit
e07667ff9b
4 changed files with 569 additions and 17 deletions
|
|
@ -55,7 +55,7 @@ Tasks mirror the design's 4-phase migration. The slow lane is foundational (ever
|
|||
- _Depends: 1.6_
|
||||
|
||||
- [ ] 2. Phase 2 — Reconnect offload (sender-side liveness)
|
||||
- [ ] 2.1 Offload the delaymgr peer reconnect to the slow lane
|
||||
- [x] 2.1 Offload the delaymgr peer reconnect to the slow lane
|
||||
- The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
|
||||
- Refcount the endpoint across the in-flight window.
|
||||
- Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
|
||||
|
|
@ -137,4 +137,5 @@ Tasks mirror the design's 4-phase migration. The slow lane is foundational (ever
|
|||
- **Build/test env**: builds MUST run in the nix dev shell (`nix develop --command meson compile -C src/xce.build <target>`). The IDE clang reports false errors (missing openssl/unity headers, unknown `-fvar-tracking`) because it isn't in the nix shell — ignore those. Controller tests: `rm -rf /tmp/xce_*` first (bug-924 overlayfs), run `./src/xce.build/controller/controller_unit_test`, set `ASAN_OPTIONS=detect_leaks=0`.
|
||||
- **Pre-existing leak (not ours)**: the default-options full controller suite exits non-zero due to a LeakSanitizer leak in `test_commit_barrier_reply.c` / `test_worker_join.c` / `test_egress_filter.c` (~174KB). Unrelated to this spec. Run with `detect_leaks=0` (or a `-t` filter) so it doesn't mask raft-drive-async regressions.
|
||||
- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.
|
||||
- **2.1**: reconnect offload is gated by a default-ON kill-switch `g_drive_async_reconnect_offload` (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via `ha_state_t.reconnect_inflight[]` CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = `reconnect_lane_fn` → `ha_connect_finish_peer` (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). **FOLLOW-UP (test hygiene, LOW)**: the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient.
|
||||
- **1.6**: pool reservation lives in vtaskmgr (`vtask_class_t` + `blocking_inflight`/`liveness_reserved` + `vtaskmgr_blocking_acquire/release`); commit folds in via opt-in `commit_set_pool` (kept `commit_engine_init(n_vcpus)` signature — the `n_vcpus` local cap is the genuine pool==NULL fallback for unit fixtures). **Production wiring is in `cluster.c` phase2** (once-guarded, right after `commit_engine_init`): `vtaskmgr_set_liveness_reserved(R)` + `commit_set_pool` — DO NOT remove; without it the reservation is dead. Lock order `p->lock → ce->lock → blk_lock` (cycle-free). New-cap = `n_vcpus−R` (tighter than old `n_vcpus−2` for 5/7-node — intended). LOW deferred: `vtaskmgr_blocking_cap()` reads `liveness_reserved` without `blk_lock` (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in `run_task` — ensure no double-count vs the commit path.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue