feat(raft-drive-async): add drive-async config constants + liveness-reservation helper (task 1.1)
drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved reservation logged at init; under-provisioned (n_vcpus<n_peers) clamp WARNs loudly, never self-terminates. 8 unit cases, mutation-verified.
This commit is contained in:
parent
295769d8ae
commit
729055a37f
8 changed files with 329 additions and 0 deletions
139
.kiro/specs/raft-drive-async/tasks.md
Normal file
139
.kiro/specs/raft-drive-async/tasks.md
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
# Implementation Plan
|
||||
|
||||
Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask.
|
||||
|
||||
- [ ] 1. Phase 1 — SlowLane executor and bounded connect (foundation)
|
||||
- [x] 1.1 Add drive-async configuration constants
|
||||
- Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count).
|
||||
- Define the liveness reservation as a peer-scaled derivation, not a flat literal: `VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)` (floor 1 = deadlock-freedom; ceiling `n_vcpus − 1` keeps ≥1 commit slot); WARN at init when clamped by `n_vcpus`.
|
||||
- Wire the constants into the controller config so they are overridable.
|
||||
- Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init.
|
||||
- _Requirements: 2.3, 3.1_
|
||||
|
||||
- [ ] 1.2 Implement the SlowLane executor
|
||||
- Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately.
|
||||
- Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle.
|
||||
- Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_
|
||||
- _Boundary: SlowLane_
|
||||
|
||||
- [ ] 1.3 Bound the peer connect path
|
||||
- Reorder `socket_connect` to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error.
|
||||
- Update connect callers to pass the configured timeout.
|
||||
- Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds.
|
||||
- _Requirements: 2.3_
|
||||
- _Boundary: BoundedConnect_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 1.4 (P) SlowLane unit tests
|
||||
- Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once.
|
||||
- Observable: tests pass and fail if the cap or dedup is removed.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_
|
||||
- _Boundary: SlowLane_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [ ] 1.5 (P) Bounded-connect unit test
|
||||
- Assert `socket_connect` returns within the timeout against an unreachable host and succeeds against a reachable one.
|
||||
- Observable: test passes; reverting the reorder makes it hang/fail.
|
||||
- _Requirements: 2.3_
|
||||
- _Boundary: BoundedConnect_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr
|
||||
- Add `vtask_class_t` (`VTASK_LIVENESS` / `VTASK_LIVE_BLOCK` / `VTASK_REQ_BLOCK`) + a classified submit; the dispatcher reserves the `VTASK_LIVENESS_RESERVED` value **resolved in 1.1** (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a `VTASK_REQ_BLOCK` with `-XCE_ERR_NOT_READY` when admitting it would breach the reservation (never parks a worker, never silently queues).
|
||||
- Fold `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` into this — `commit_submit` submits a `VTASK_REQ_BLOCK`; the pool owns the single reservation.
|
||||
- Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). **Hard prerequisite of 3.1** — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters).
|
||||
- Observable: a `VTASK_LIVENESS` task always finds a worker even when `*_BLOCK` tasks saturate `n_vcpus − VTASK_LIVENESS_RESERVED`; commit-wait still fast-rejects `-XCE_ERR_NOT_READY` at the cap (backpressure unchanged).
|
||||
- _Requirements: 3.1, 3.2, 3.3, 4.5_
|
||||
- _Boundary: LivenessAdmission_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 1.7 (P) LivenessAdmission unit test
|
||||
- Assert: a `VTASK_LIVENESS` task is dispatched while `*_BLOCK` tasks hold `n_vcpus − VTASK_LIVENESS_RESERVED` vcpus; `VTASK_REQ_BLOCK` over the cap returns `-XCE_ERR_NOT_READY` without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 4.5_
|
||||
- _Boundary: LivenessAdmission_
|
||||
- _Depends: 1.6_
|
||||
|
||||
- [ ] 2. Phase 2 — Reconnect offload (sender-side liveness)
|
||||
- [ ] 2.1 Offload the delaymgr peer reconnect to the slow lane
|
||||
- The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
|
||||
- Refcount the endpoint across the in-flight window.
|
||||
- Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
|
||||
- _Requirements: 2.1, 2.2, 2.5_
|
||||
- _Boundary: ReconnectOffload, SlowLane_
|
||||
- _Depends: 1.2, 1.3_
|
||||
|
||||
- [ ] 2.2 Complete the reconnect on the lane worker
|
||||
- The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
|
||||
- Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
|
||||
- _Requirements: 2.4, 2.6, 4.4_
|
||||
- _Boundary: ReconnectOffload_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 2.3 Reconnect-offload integration test
|
||||
- With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
|
||||
- Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
|
||||
- _Requirements: 2.1, 2.2, 2.5_
|
||||
- _Boundary: ReconnectOffload_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side)
|
||||
- [ ] 3.1 Dispatch peer-control message handling to the vtask pool
|
||||
- Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline.
|
||||
- Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues.
|
||||
- Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order.
|
||||
- **Deadlock-freedom gate (bug-invariant reversal):** moving ack decode to the pool co-locates the commit-barrier *wakers* (acks) and *waiters* (`commit_submit(COMMIT_MODE_WAIT)`) on one pool — today acks run on iomgr precisely to avoid this (`commit.c:16`). The old commit-only `COMMIT_WAIT_RESERVED` is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on **LivenessAdmission (task 1.6)**: ack/vote-receipt dispatched as `VTASK_LIVENESS` (never gated, never offloaded), `commit_submit` as `VTASK_REQ_BLOCK` under the pool reservation. **Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.**
|
||||
- _Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2_
|
||||
- _Boundary: PeerCtrlDispatch_
|
||||
- _Depends: 1.2, 1.6_
|
||||
|
||||
- [ ] 3.2 Offload snapshot send to the slow lane
|
||||
- Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
|
||||
- Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
|
||||
- **Add the trim cursor-bound guard (bug-1147, stopgap):** `trim_wal_segments` (`persist.c:1374`) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
|
||||
- Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby `acked_lsn` or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap).
|
||||
- _Requirements: 1.1, 1.3, 1.5, 4.3_
|
||||
- _Boundary: PeerCtrlDispatch, SlowLane_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 3.3 (P) Offload the transport-only config-push to the slow lane
|
||||
- Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged.
|
||||
- Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged.
|
||||
- _Requirements: 1.2_
|
||||
- _Boundary: PeerCtrlDispatch, SlowLane_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 3.4 Receiver-side integration tests
|
||||
- Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another.
|
||||
- Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers.
|
||||
- _Requirements: 1.1, 1.3, 1.4, 4.3, 4.5_
|
||||
- _Boundary: PeerCtrlDispatch_
|
||||
- _Depends: 3.2, 3.3_
|
||||
|
||||
- [ ] 4. Phase 4 — Verification, partition test, and regression
|
||||
- [ ] 4.1 Network-partition (unreachable-peer) soak scenario
|
||||
- Add a partition primitive that renders one node unreachable (distinct from process kill). **Primary = in-process socket-drop hook in `xce_test` (`XCE_TESTING`)** — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of `ha-term-safety` 5.2/5.4). **Secondary (optional) = firewall `iptables`/`nft` DROP** for real-network fidelity, capability-gated with a loud skip when unprivileged.
|
||||
- Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost.
|
||||
- Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election).
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4_
|
||||
- _Boundary: PartitionTest_
|
||||
- _Depends: 2.3, 3.4_
|
||||
|
||||
- [ ] 4.2 Existing-feature regression sweep
|
||||
- Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak.
|
||||
- Observable: all listed suites pass with no change to their asserted behavior.
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7_
|
||||
- _Depends: 3.4_
|
||||
|
||||
- [ ] 4.3 Thread-role compliance and heartbeat performance check
|
||||
- Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane.
|
||||
- Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path.
|
||||
- Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline.
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2_
|
||||
- _Depends: 2.3, 3.4_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- **Build/test env**: builds MUST run in the nix dev shell (`nix develop --command meson compile -C src/xce.build <target>`). The IDE clang reports false errors (missing openssl/unity headers, unknown `-fvar-tracking`) because it isn't in the nix shell — ignore those. Controller tests: `rm -rf /tmp/xce_*` first (bug-924 overlayfs), run `./src/xce.build/controller/controller_unit_test`, set `ASAN_OPTIONS=detect_leaks=0`.
|
||||
- **Pre-existing leak (not ours)**: the default-options full controller suite exits non-zero due to a LeakSanitizer leak in `test_commit_barrier_reply.c` / `test_worker_join.c` / `test_egress_filter.c` (~174KB). Unrelated to this spec. Run with `detect_leaks=0` (or a `-t` filter) so it doesn't mask raft-drive-async regressions.
|
||||
- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.
|
||||
|
|
@ -1342,6 +1342,29 @@ int cluster_init_phase2(cluster_t* c)
|
|||
c->ha.repl = &c->persist.repl;
|
||||
INFO("commit engine wired (n_peers=%d, n_vcpus=%d)",
|
||||
c->ha.n_peers, c->vtaskmgr.n_vcpus);
|
||||
|
||||
/* raft-drive-async task 1.1: resolve and log the peer-scaled liveness
|
||||
* reservation at init — both inputs (n_peers, n_vcpus) are final here.
|
||||
* Single source of truth: task 1.6 (LivenessAdmission) consumes this
|
||||
* value, it does NOT recompute the clamp/WARN. */
|
||||
int liveness_reserved =
|
||||
drive_async_liveness_reserved(c->ha.n_peers, c->vtaskmgr.n_vcpus);
|
||||
INFO("drive-async: VTASK_LIVENESS_RESERVED=%d (n_peers=%d, n_vcpus=%d; "
|
||||
"commit/REQ_BLOCK cap=%d)",
|
||||
liveness_reserved, c->ha.n_peers, c->vtaskmgr.n_vcpus,
|
||||
c->vtaskmgr.n_vcpus - liveness_reserved);
|
||||
if (c->vtaskmgr.n_vcpus < c->ha.n_peers) {
|
||||
/* Under-provisioned: the n_vcpus ceiling clamped the reservation
|
||||
* below the ideal n_peers-1. Still correct (deadlock-free via fast
|
||||
* worker cycling), but operator-visible per CLAUDE.md fail-loud.
|
||||
* A controller NEVER self-terminates on this condition. */
|
||||
WARN("drive-async: under-provisioned controller — n_vcpus=%d < "
|
||||
"n_peers=%d; liveness reservation clamped to %d (ideal %d). "
|
||||
"Per-peer liveness latency may rise under load; consider "
|
||||
"raising --vcpus. Cluster remains correct and stays up.",
|
||||
c->vtaskmgr.n_vcpus, c->ha.n_peers, liveness_reserved,
|
||||
c->ha.n_peers - 1);
|
||||
}
|
||||
}
|
||||
|
||||
/* Role set by ha_elect_evaluate_and_signal() inside cluster_run() */
|
||||
|
|
|
|||
|
|
@ -580,6 +580,24 @@ uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip)
|
|||
return safe_lsn > durable_tip ? safe_lsn : durable_tip;
|
||||
}
|
||||
|
||||
/* ── raft-drive-async task 1.1: peer-scaled liveness reservation ─────────────
|
||||
*
|
||||
* Pure: reserved = max(1, min(n_peers - 1, n_vcpus - 1)). The floor of 1 is
|
||||
* applied LAST so deadlock-freedom (>=1 guaranteed-free liveness worker) holds
|
||||
* even when the n_vcpus ceiling would otherwise drive the value to 0 (e.g.
|
||||
* n_vcpus == 1). See ha.h for the full rationale; the caller WARNs on the
|
||||
* under-provisioned (n_vcpus < n_peers) clamp — this helper never logs. */
|
||||
int drive_async_liveness_reserved(int n_peers, int n_vcpus)
|
||||
{
|
||||
int desired = n_peers - 1; /* worst-case concurrent liveness work */
|
||||
int ceiling = n_vcpus - 1; /* keep >=1 commit (REQ_BLOCK) slot */
|
||||
|
||||
int reserved = desired < ceiling ? desired : ceiling; /* min(desired, ceiling) */
|
||||
if (reserved < 1) /* floor applied LAST — always wins */
|
||||
reserved = 1;
|
||||
return reserved;
|
||||
}
|
||||
|
||||
/* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ───────────
|
||||
*
|
||||
* ha_promotion_gate_decision: pure decision (Req 2.2, 2.5, 2.6). Build the Boot
|
||||
|
|
|
|||
|
|
@ -618,6 +618,34 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
|
|||
* producing divergent WALs and a permanent prevote deadlock. */
|
||||
uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip);
|
||||
|
||||
/* ── raft-drive-async task 1.1: peer-scaled liveness reservation ─────────────
|
||||
*
|
||||
* drive_async_liveness_reserved: PURE derivation of VTASK_LIVENESS_RESERVED —
|
||||
* the number of vtask-pool workers kept free for liveness-critical handling
|
||||
* (heartbeat/vote/ack) so they are never starved by blocking offloaded tasks.
|
||||
*
|
||||
* reserved = clamp(n_peers - 1, 1, n_vcpus - 1)
|
||||
* = max(1, min(n_peers - 1, n_vcpus - 1))
|
||||
*
|
||||
* - n_peers is the TOTAL cluster size including self (matches ha->n_peers).
|
||||
* Per-endpoint serialization (endpoint_block_tasks) caps in-flight work at
|
||||
* one per peer connection, so worst-case concurrent liveness work is the
|
||||
* n_peers-1 OTHER peers (one ack/vote each) — reserving n_peers-1 means no
|
||||
* peer's liveness message queues behind a busy reserved worker (a LATENCY
|
||||
* optimization).
|
||||
* - Floor 1 is the hard CORRECTNESS invariant (deadlock-freedom): one
|
||||
* guaranteed-free worker + liveness-never-gated + sub-ms cycling is
|
||||
* deadlock-free regardless of peer count. Applied LAST so it always wins.
|
||||
* - Ceiling n_vcpus-1 keeps >=1 REQ_BLOCK (commit) slot, so the commit cap
|
||||
* (n_vcpus - reserved) is never 0.
|
||||
*
|
||||
* On an under-provisioned box (n_vcpus < n_peers) the ceiling clamps the
|
||||
* result below n_peers-1; the result is still correct via fast cycling.
|
||||
* The CALLER (controller init) is responsible for emitting a loud WARN in
|
||||
* that case so the under-provisioning is operator-visible — this helper is
|
||||
* pure and never logs. Controllers NEVER self-terminate on this condition. */
|
||||
int drive_async_liveness_reserved(int n_peers, int n_vcpus);
|
||||
|
||||
/* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ──────────
|
||||
*
|
||||
* ha_promotion_gate_decision: PURE pre-promotion eligibility decision (Req 2.2,
|
||||
|
|
|
|||
|
|
@ -139,6 +139,7 @@ controller_unit_test_src = [
|
|||
'testsuite/test_ha_heartbeat_committed.c',
|
||||
'testsuite/test_ha_vote_floor_heartbeat.c',
|
||||
'testsuite/test_term_safety_adapter.c',
|
||||
'testsuite/test_drive_async_config.c',
|
||||
'testsuite/test_waldump.c',
|
||||
'testsuite/waldump.c',
|
||||
'testsuite/test_walcompare.c',
|
||||
|
|
|
|||
|
|
@ -66,6 +66,7 @@ extern void test_ha_vote_floor_heartbeat(void);
|
|||
extern void test_term_safety_adapter(void);
|
||||
extern void test_waldump(void);
|
||||
extern void test_walcompare(void);
|
||||
extern void test_drive_async_config(void);
|
||||
|
||||
void setUp(void) { return; }
|
||||
|
||||
|
|
@ -201,6 +202,7 @@ int main(int argc, char* argv[])
|
|||
TEST(test_term_safety_adapter);
|
||||
TEST(test_waldump);
|
||||
TEST(test_walcompare);
|
||||
TEST(test_drive_async_config);
|
||||
|
||||
return UNITY_END();
|
||||
}
|
||||
|
|
|
|||
97
src/controller/testsuite/test_drive_async_config.c
Normal file
97
src/controller/testsuite/test_drive_async_config.c
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
#define _GNU_SOURCE
|
||||
#include <stdio.h>
|
||||
|
||||
#include <unity/unity.h>
|
||||
|
||||
#include "config.h"
|
||||
#include "ha.h"
|
||||
|
||||
/* ====================================================================
|
||||
* raft-drive-async task 1.1 — drive-async configuration constants and
|
||||
* the peer-scaled liveness-reservation clamp helper.
|
||||
*
|
||||
* Helper under test:
|
||||
* int drive_async_liveness_reserved(int n_peers, int n_vcpus);
|
||||
*
|
||||
* Formula (design "LivenessAdmission → Sizing (peer-scaled)"):
|
||||
* VTASK_LIVENESS_RESERVED = clamp(n_peers - 1, 1, n_vcpus - 1)
|
||||
* = max(1, min(n_peers - 1, n_vcpus - 1))
|
||||
* floor 1 -> deadlock-freedom (always >=1 guaranteed-free worker)
|
||||
* ceiling n_vcpus - 1 -> keep >=1 REQ_BLOCK (commit) slot, cap never 0
|
||||
* The floor is applied LAST so it always wins (the hard correctness invariant).
|
||||
* ==================================================================== */
|
||||
|
||||
/* ── Regime 1: normal n_peers-1 (the un-clamped middle) ───────────── */
|
||||
|
||||
static void test_liveness_reserved_normal_three_peers(void)
|
||||
{
|
||||
/* 3-node cluster, 8 vcpus: n_peers-1 = 2, well under ceiling 7. */
|
||||
TEST_ASSERT_EQUAL_INT(2, drive_async_liveness_reserved(3, 8));
|
||||
}
|
||||
|
||||
static void test_liveness_reserved_normal_five_peers(void)
|
||||
{
|
||||
/* 5-node cluster, 8 vcpus: n_peers-1 = 4, under ceiling 7. */
|
||||
TEST_ASSERT_EQUAL_INT(4, drive_async_liveness_reserved(5, 8));
|
||||
}
|
||||
|
||||
/* ── Regime 2: floor hit (n_peers-1 would be < 1) ─────────────────── */
|
||||
|
||||
static void test_liveness_reserved_floor_single_node(void)
|
||||
{
|
||||
/* n_peers=1 -> n_peers-1 = 0 -> floor 1 must win. */
|
||||
TEST_ASSERT_EQUAL_INT(1, drive_async_liveness_reserved(1, 8));
|
||||
}
|
||||
|
||||
static void test_liveness_reserved_floor_zero_peers(void)
|
||||
{
|
||||
/* Degenerate n_peers=0 -> n_peers-1 = -1 -> floor 1 must win. */
|
||||
TEST_ASSERT_EQUAL_INT(1, drive_async_liveness_reserved(0, 8));
|
||||
}
|
||||
|
||||
/* ── Regime 3: ceiling / clamp hit (under-provisioned: n_vcpus < n_peers) ── */
|
||||
|
||||
static void test_liveness_reserved_ceiling_clamps_below_npeers(void)
|
||||
{
|
||||
/* n_peers=5, n_vcpus=3: n_peers-1 = 4 clamped to ceiling n_vcpus-1 = 2. */
|
||||
TEST_ASSERT_EQUAL_INT(2, drive_async_liveness_reserved(5, 3));
|
||||
}
|
||||
|
||||
static void test_liveness_reserved_ceiling_floor_wins_one_vcpu(void)
|
||||
{
|
||||
/* n_vcpus=1 -> ceiling n_vcpus-1 = 0, but the floor of 1 must win:
|
||||
* max(1, min(n_peers-1, 0)) = max(1, 0) = 1. Never returns 0. */
|
||||
TEST_ASSERT_EQUAL_INT(1, drive_async_liveness_reserved(5, 1));
|
||||
}
|
||||
|
||||
static void test_liveness_reserved_exactly_npeers_vcpus(void)
|
||||
{
|
||||
/* n_vcpus == n_peers: ceiling = n_vcpus-1 = n_peers-1 -> no clamp.
|
||||
* 3 peers, 3 vcpus -> min(2, 2) = 2. */
|
||||
TEST_ASSERT_EQUAL_INT(2, drive_async_liveness_reserved(3, 3));
|
||||
}
|
||||
|
||||
static void test_liveness_reserved_never_exceeds_vcpus_minus_one(void)
|
||||
{
|
||||
/* Invariant: result <= n_vcpus-1 keeps >=1 commit slot (cap never 0). */
|
||||
for (int p = 0; p <= 8; p++) {
|
||||
for (int v = 1; v <= 8; v++) {
|
||||
int r = drive_async_liveness_reserved(p, v);
|
||||
TEST_ASSERT_TRUE_MESSAGE(r >= 1, "reservation floor 1 violated");
|
||||
TEST_ASSERT_TRUE_MESSAGE(r <= (v - 1 > 0 ? v - 1 : 1),
|
||||
"reservation must keep >=1 commit slot");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void test_drive_async_config(void)
|
||||
{
|
||||
RUN_TEST(test_liveness_reserved_normal_three_peers);
|
||||
RUN_TEST(test_liveness_reserved_normal_five_peers);
|
||||
RUN_TEST(test_liveness_reserved_floor_single_node);
|
||||
RUN_TEST(test_liveness_reserved_floor_zero_peers);
|
||||
RUN_TEST(test_liveness_reserved_ceiling_clamps_below_npeers);
|
||||
RUN_TEST(test_liveness_reserved_ceiling_floor_wins_one_vcpu);
|
||||
RUN_TEST(test_liveness_reserved_exactly_npeers_vcpus);
|
||||
RUN_TEST(test_liveness_reserved_never_exceeds_vcpus_minus_one);
|
||||
}
|
||||
|
|
@ -94,4 +94,25 @@ typedef uint64_t jobid_t;
|
|||
#define DEFAULT_HB_INTERVAL_MS 3000
|
||||
#define DEFAULT_MAX_WAL_RECORDS 10000
|
||||
|
||||
/* ======================================================
|
||||
* raft-drive-async constants (task 1.1)
|
||||
* Non-blocking Raft drive: bounded peer connect + slow-lane executor +
|
||||
* pool-level liveness reservation. All overridable via myconf_t (0 = use
|
||||
* the default below); see drive_async_liveness_reserved() in ha.h for the
|
||||
* peer-scaled reservation derivation.
|
||||
* ====================================================== */
|
||||
|
||||
/* R2.3: bound the peer connect attempt so an unreachable peer cannot stall
|
||||
* the delaymgr heartbeat thread for the OS connect timeout. BoundedConnect
|
||||
* polls up to this many ms after a non-blocking connect() before failing. */
|
||||
#define DEFAULT_CONNECT_TIMEOUT_MS 1000
|
||||
|
||||
/* SlowLane executor sizing (task 1.2 consumes these). Dedicated worker
|
||||
* threads isolated from the vtask request pool; sized to peer count so a
|
||||
* snapshot transfer and a reconnect to distinct peers do not starve each
|
||||
* other. MAX_HA_CANDIDATES (4) bounds a supported cluster, so a small
|
||||
* constant covers reconnect + transfer per peer without over-allocating. */
|
||||
#define DEFAULT_SLOWLANE_WORKERS 4
|
||||
#define DEFAULT_SLOWLANE_MAX_INFLIGHT 8
|
||||
|
||||
#endif
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue