From 729055a37f9edd7d6b2f070810daaf6106fbe07c Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Fri, 26 Jun 2026 04:58:07 +0000 Subject: [PATCH] feat(raft-drive-async): add drive-async config constants + liveness-reservation helper (task 1.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved reservation logged at init; under-provisioned (n_vcpuslock` and refuse to delete at/above `floor`. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec). + - Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby `acked_lsn` or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap). + - _Requirements: 1.1, 1.3, 1.5, 4.3_ + - _Boundary: PeerCtrlDispatch, SlowLane_ + - _Depends: 3.1_ + +- [ ] 3.3 (P) Offload the transport-only config-push to the slow lane + - Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged. + - Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged. + - _Requirements: 1.2_ + - _Boundary: PeerCtrlDispatch, SlowLane_ + - _Depends: 3.1_ + +- [ ] 3.4 Receiver-side integration tests + - Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another. + - Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers. + - _Requirements: 1.1, 1.3, 1.4, 4.3, 4.5_ + - _Boundary: PeerCtrlDispatch_ + - _Depends: 3.2, 3.3_ + +- [ ] 4. Phase 4 — Verification, partition test, and regression +- [ ] 4.1 Network-partition (unreachable-peer) soak scenario + - Add a partition primitive that renders one node unreachable (distinct from process kill). **Primary = in-process socket-drop hook in `xce_test` (`XCE_TESTING`)** — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of `ha-term-safety` 5.2/5.4). **Secondary (optional) = firewall `iptables`/`nft` DROP** for real-network fidelity, capability-gated with a loud skip when unprivileged. + - Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost. + - Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election). + - _Requirements: 7.1, 7.2, 7.3, 7.4_ + - _Boundary: PartitionTest_ + - _Depends: 2.3, 3.4_ + +- [ ] 4.2 Existing-feature regression sweep + - Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak. + - Observable: all listed suites pass with no change to their asserted behavior. + - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7_ + - _Depends: 3.4_ + +- [ ] 4.3 Thread-role compliance and heartbeat performance check + - Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane. + - Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path. + - Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline. + - _Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2_ + - _Depends: 2.3, 3.4_ + +## Implementation Notes + +- **Build/test env**: builds MUST run in the nix dev shell (`nix develop --command meson compile -C src/xce.build `). The IDE clang reports false errors (missing openssl/unity headers, unknown `-fvar-tracking`) because it isn't in the nix shell — ignore those. Controller tests: `rm -rf /tmp/xce_*` first (bug-924 overlayfs), run `./src/xce.build/controller/controller_unit_test`, set `ASAN_OPTIONS=detect_leaks=0`. +- **Pre-existing leak (not ours)**: the default-options full controller suite exits non-zero due to a LeakSanitizer leak in `test_commit_barrier_reply.c` / `test_worker_join.c` / `test_egress_filter.c` (~174KB). Unrelated to this spec. Run with `detect_leaks=0` (or a `-t` filter) so it doesn't mask raft-drive-async regressions. +- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader. diff --git a/src/controller/cluster.c b/src/controller/cluster.c index a96433cb..6cd12e2c 100644 --- a/src/controller/cluster.c +++ b/src/controller/cluster.c @@ -1342,6 +1342,29 @@ int cluster_init_phase2(cluster_t* c) c->ha.repl = &c->persist.repl; INFO("commit engine wired (n_peers=%d, n_vcpus=%d)", c->ha.n_peers, c->vtaskmgr.n_vcpus); + + /* raft-drive-async task 1.1: resolve and log the peer-scaled liveness + * reservation at init — both inputs (n_peers, n_vcpus) are final here. + * Single source of truth: task 1.6 (LivenessAdmission) consumes this + * value, it does NOT recompute the clamp/WARN. */ + int liveness_reserved = + drive_async_liveness_reserved(c->ha.n_peers, c->vtaskmgr.n_vcpus); + INFO("drive-async: VTASK_LIVENESS_RESERVED=%d (n_peers=%d, n_vcpus=%d; " + "commit/REQ_BLOCK cap=%d)", + liveness_reserved, c->ha.n_peers, c->vtaskmgr.n_vcpus, + c->vtaskmgr.n_vcpus - liveness_reserved); + if (c->vtaskmgr.n_vcpus < c->ha.n_peers) { + /* Under-provisioned: the n_vcpus ceiling clamped the reservation + * below the ideal n_peers-1. Still correct (deadlock-free via fast + * worker cycling), but operator-visible per CLAUDE.md fail-loud. + * A controller NEVER self-terminates on this condition. */ + WARN("drive-async: under-provisioned controller — n_vcpus=%d < " + "n_peers=%d; liveness reservation clamped to %d (ideal %d). " + "Per-peer liveness latency may rise under load; consider " + "raising --vcpus. Cluster remains correct and stays up.", + c->vtaskmgr.n_vcpus, c->ha.n_peers, liveness_reserved, + c->ha.n_peers - 1); + } } /* Role set by ha_elect_evaluate_and_signal() inside cluster_run() */ diff --git a/src/controller/ha.c b/src/controller/ha.c index 2cc4bdbb..ec5b24c0 100644 --- a/src/controller/ha.c +++ b/src/controller/ha.c @@ -580,6 +580,24 @@ uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip) return safe_lsn > durable_tip ? safe_lsn : durable_tip; } +/* ── raft-drive-async task 1.1: peer-scaled liveness reservation ───────────── + * + * Pure: reserved = max(1, min(n_peers - 1, n_vcpus - 1)). The floor of 1 is + * applied LAST so deadlock-freedom (>=1 guaranteed-free liveness worker) holds + * even when the n_vcpus ceiling would otherwise drive the value to 0 (e.g. + * n_vcpus == 1). See ha.h for the full rationale; the caller WARNs on the + * under-provisioned (n_vcpus < n_peers) clamp — this helper never logs. */ +int drive_async_liveness_reserved(int n_peers, int n_vcpus) +{ + int desired = n_peers - 1; /* worst-case concurrent liveness work */ + int ceiling = n_vcpus - 1; /* keep >=1 commit (REQ_BLOCK) slot */ + + int reserved = desired < ceiling ? desired : ceiling; /* min(desired, ceiling) */ + if (reserved < 1) /* floor applied LAST — always wins */ + reserved = 1; + return reserved; +} + /* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ─────────── * * ha_promotion_gate_decision: pure decision (Req 2.2, 2.5, 2.6). Build the Boot diff --git a/src/controller/ha.h b/src/controller/ha.h index 6970fcd4..8b02bbd8 100644 --- a/src/controller/ha.h +++ b/src/controller/ha.h @@ -618,6 +618,34 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx, * producing divergent WALs and a permanent prevote deadlock. */ uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip); +/* ── raft-drive-async task 1.1: peer-scaled liveness reservation ───────────── + * + * drive_async_liveness_reserved: PURE derivation of VTASK_LIVENESS_RESERVED — + * the number of vtask-pool workers kept free for liveness-critical handling + * (heartbeat/vote/ack) so they are never starved by blocking offloaded tasks. + * + * reserved = clamp(n_peers - 1, 1, n_vcpus - 1) + * = max(1, min(n_peers - 1, n_vcpus - 1)) + * + * - n_peers is the TOTAL cluster size including self (matches ha->n_peers). + * Per-endpoint serialization (endpoint_block_tasks) caps in-flight work at + * one per peer connection, so worst-case concurrent liveness work is the + * n_peers-1 OTHER peers (one ack/vote each) — reserving n_peers-1 means no + * peer's liveness message queues behind a busy reserved worker (a LATENCY + * optimization). + * - Floor 1 is the hard CORRECTNESS invariant (deadlock-freedom): one + * guaranteed-free worker + liveness-never-gated + sub-ms cycling is + * deadlock-free regardless of peer count. Applied LAST so it always wins. + * - Ceiling n_vcpus-1 keeps >=1 REQ_BLOCK (commit) slot, so the commit cap + * (n_vcpus - reserved) is never 0. + * + * On an under-provisioned box (n_vcpus < n_peers) the ceiling clamps the + * result below n_peers-1; the result is still correct via fast cycling. + * The CALLER (controller init) is responsible for emitting a loud WARN in + * that case so the under-provisioning is operator-visible — this helper is + * pure and never logs. Controllers NEVER self-terminate on this condition. */ +int drive_async_liveness_reserved(int n_peers, int n_vcpus); + /* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ────────── * * ha_promotion_gate_decision: PURE pre-promotion eligibility decision (Req 2.2, diff --git a/src/controller/meson.build b/src/controller/meson.build index 2f1409f2..97722284 100644 --- a/src/controller/meson.build +++ b/src/controller/meson.build @@ -139,6 +139,7 @@ controller_unit_test_src = [ 'testsuite/test_ha_heartbeat_committed.c', 'testsuite/test_ha_vote_floor_heartbeat.c', 'testsuite/test_term_safety_adapter.c', + 'testsuite/test_drive_async_config.c', 'testsuite/test_waldump.c', 'testsuite/waldump.c', 'testsuite/test_walcompare.c', diff --git a/src/controller/testsuite/controller_test.c b/src/controller/testsuite/controller_test.c index c3e2cb65..3cc1e134 100644 --- a/src/controller/testsuite/controller_test.c +++ b/src/controller/testsuite/controller_test.c @@ -66,6 +66,7 @@ extern void test_ha_vote_floor_heartbeat(void); extern void test_term_safety_adapter(void); extern void test_waldump(void); extern void test_walcompare(void); +extern void test_drive_async_config(void); void setUp(void) { return; } @@ -201,6 +202,7 @@ int main(int argc, char* argv[]) TEST(test_term_safety_adapter); TEST(test_waldump); TEST(test_walcompare); + TEST(test_drive_async_config); return UNITY_END(); } diff --git a/src/controller/testsuite/test_drive_async_config.c b/src/controller/testsuite/test_drive_async_config.c new file mode 100644 index 00000000..243ef8bd --- /dev/null +++ b/src/controller/testsuite/test_drive_async_config.c @@ -0,0 +1,97 @@ +#define _GNU_SOURCE +#include + +#include + +#include "config.h" +#include "ha.h" + +/* ==================================================================== + * raft-drive-async task 1.1 — drive-async configuration constants and + * the peer-scaled liveness-reservation clamp helper. + * + * Helper under test: + * int drive_async_liveness_reserved(int n_peers, int n_vcpus); + * + * Formula (design "LivenessAdmission → Sizing (peer-scaled)"): + * VTASK_LIVENESS_RESERVED = clamp(n_peers - 1, 1, n_vcpus - 1) + * = max(1, min(n_peers - 1, n_vcpus - 1)) + * floor 1 -> deadlock-freedom (always >=1 guaranteed-free worker) + * ceiling n_vcpus - 1 -> keep >=1 REQ_BLOCK (commit) slot, cap never 0 + * The floor is applied LAST so it always wins (the hard correctness invariant). + * ==================================================================== */ + +/* ── Regime 1: normal n_peers-1 (the un-clamped middle) ───────────── */ + +static void test_liveness_reserved_normal_three_peers(void) +{ + /* 3-node cluster, 8 vcpus: n_peers-1 = 2, well under ceiling 7. */ + TEST_ASSERT_EQUAL_INT(2, drive_async_liveness_reserved(3, 8)); +} + +static void test_liveness_reserved_normal_five_peers(void) +{ + /* 5-node cluster, 8 vcpus: n_peers-1 = 4, under ceiling 7. */ + TEST_ASSERT_EQUAL_INT(4, drive_async_liveness_reserved(5, 8)); +} + +/* ── Regime 2: floor hit (n_peers-1 would be < 1) ─────────────────── */ + +static void test_liveness_reserved_floor_single_node(void) +{ + /* n_peers=1 -> n_peers-1 = 0 -> floor 1 must win. */ + TEST_ASSERT_EQUAL_INT(1, drive_async_liveness_reserved(1, 8)); +} + +static void test_liveness_reserved_floor_zero_peers(void) +{ + /* Degenerate n_peers=0 -> n_peers-1 = -1 -> floor 1 must win. */ + TEST_ASSERT_EQUAL_INT(1, drive_async_liveness_reserved(0, 8)); +} + +/* ── Regime 3: ceiling / clamp hit (under-provisioned: n_vcpus < n_peers) ── */ + +static void test_liveness_reserved_ceiling_clamps_below_npeers(void) +{ + /* n_peers=5, n_vcpus=3: n_peers-1 = 4 clamped to ceiling n_vcpus-1 = 2. */ + TEST_ASSERT_EQUAL_INT(2, drive_async_liveness_reserved(5, 3)); +} + +static void test_liveness_reserved_ceiling_floor_wins_one_vcpu(void) +{ + /* n_vcpus=1 -> ceiling n_vcpus-1 = 0, but the floor of 1 must win: + * max(1, min(n_peers-1, 0)) = max(1, 0) = 1. Never returns 0. */ + TEST_ASSERT_EQUAL_INT(1, drive_async_liveness_reserved(5, 1)); +} + +static void test_liveness_reserved_exactly_npeers_vcpus(void) +{ + /* n_vcpus == n_peers: ceiling = n_vcpus-1 = n_peers-1 -> no clamp. + * 3 peers, 3 vcpus -> min(2, 2) = 2. */ + TEST_ASSERT_EQUAL_INT(2, drive_async_liveness_reserved(3, 3)); +} + +static void test_liveness_reserved_never_exceeds_vcpus_minus_one(void) +{ + /* Invariant: result <= n_vcpus-1 keeps >=1 commit slot (cap never 0). */ + for (int p = 0; p <= 8; p++) { + for (int v = 1; v <= 8; v++) { + int r = drive_async_liveness_reserved(p, v); + TEST_ASSERT_TRUE_MESSAGE(r >= 1, "reservation floor 1 violated"); + TEST_ASSERT_TRUE_MESSAGE(r <= (v - 1 > 0 ? v - 1 : 1), + "reservation must keep >=1 commit slot"); + } + } +} + +void test_drive_async_config(void) +{ + RUN_TEST(test_liveness_reserved_normal_three_peers); + RUN_TEST(test_liveness_reserved_normal_five_peers); + RUN_TEST(test_liveness_reserved_floor_single_node); + RUN_TEST(test_liveness_reserved_floor_zero_peers); + RUN_TEST(test_liveness_reserved_ceiling_clamps_below_npeers); + RUN_TEST(test_liveness_reserved_ceiling_floor_wins_one_vcpu); + RUN_TEST(test_liveness_reserved_exactly_npeers_vcpus); + RUN_TEST(test_liveness_reserved_never_exceeds_vcpus_minus_one); +} diff --git a/src/include/config.h b/src/include/config.h index babe0b9d..d3c6cc6b 100644 --- a/src/include/config.h +++ b/src/include/config.h @@ -94,4 +94,25 @@ typedef uint64_t jobid_t; #define DEFAULT_HB_INTERVAL_MS 3000 #define DEFAULT_MAX_WAL_RECORDS 10000 +/* ====================================================== + * raft-drive-async constants (task 1.1) + * Non-blocking Raft drive: bounded peer connect + slow-lane executor + + * pool-level liveness reservation. All overridable via myconf_t (0 = use + * the default below); see drive_async_liveness_reserved() in ha.h for the + * peer-scaled reservation derivation. + * ====================================================== */ + +/* R2.3: bound the peer connect attempt so an unreachable peer cannot stall + * the delaymgr heartbeat thread for the OS connect timeout. BoundedConnect + * polls up to this many ms after a non-blocking connect() before failing. */ +#define DEFAULT_CONNECT_TIMEOUT_MS 1000 + +/* SlowLane executor sizing (task 1.2 consumes these). Dedicated worker + * threads isolated from the vtask request pool; sized to peer count so a + * snapshot transfer and a reconnect to distinct peers do not starve each + * other. MAX_HA_CANDIDATES (4) bounds a supported cluster, so a small + * constant covers reconnect + transfer per peer without over-allocating. */ +#define DEFAULT_SLOWLANE_WORKERS 4 +#define DEFAULT_SLOWLANE_MAX_INFLIGHT 8 + #endif