diff --git a/.kiro/specs/ha-commit-durability/tasks.md b/.kiro/specs/ha-commit-durability/tasks.md index 628f560..76a5e23 100644 --- a/.kiro/specs/ha-commit-durability/tasks.md +++ b/.kiro/specs/ha-commit-durability/tasks.md @@ -134,3 +134,4 @@ - 4.1: ANALYSIS-ONLY (no .c/.h). Pinned freshness-floor formula in design.md §"Freshness-Floor Formula (task 4.1)" — THE CONTRACT 4.2 implements. CANDIDATE advertises BARE `cand_lsn = persist_durable_floor(p)` (durable WAL tip) — sites ha.c:3660-3666 (real-vote) + ha.c:3735-3741 (PreVote) must call `persist_durable_floor` DIRECTLY, BYPASSING `ha_vote_self_freshness` on the LSN axis (that fn unconditionally folds in `last_known_committed_lsn` → `max(durable_floor, learned)` = the truncated-candidate-wins bug). VOTER floor = `max(durable_floor, last_known_committed_lsn)` — sites cluster.c:388-395/471-475 + ha.c:2243 keep `ha_vote_self_freshness` with `local_lsn=durable_floor` (the learned_committed backstop stays on the VOTER axis only). Grant iff `cand_tag > floor_tag OR (==tag AND cand_lsn >= floor_lsn)` (term-then-index, matches raft.tla `LastTerm`/`mlastLogIndex |-> Len(log)`). `cand_tag`/`floor_tag` = `max(local_tag, learned_tag)`. SAFETY (Leader Completeness): quorum intersection Q_L∩Q_v ⇒ truncated candidate (durable tip < committed L) denied. LIVENESS (no bug-339 livelock): rests on D* (max durable tip in reachable majority); `durable_floor(v) ≤ D*` and `learned_committed(v) ≤ L ≤ D*`, so the most-durable candidate is always grantable. `safe_committed_lsn` STAYS voter-reply-sourced (unchanged). CAVEAT: raft.tla validates election/freshness LOGIC only, NOT durability (no TLC run — no toolchain; conceptual alignment) — durability verified by 2.x/3.4 tests. (1 remediation: candidate-vs-voter contradiction + §5b example fixed.) - 4.2: Implements the 4.1 formula. New `ha_candidate_advertise_lsn(persist*)` = bare `persist_durable_floor`. CANDIDATE sites advertise bare durable_floor for the LSN (tag still via `ha_vote_self_freshness(...,&tag,NULL)`): ha.c:~3690 (real election), ha.c:~3771 (PreVote), AND myapp.c:~234 (boot election — a THIRD candidate site the brief didn't name; fixed too, else a boot-time truncated candidate over-advertises). VOTER floor: cluster.c:~383 (real) + cluster.c:~471 (PreVote) feed `local_lsn = persist_durable_floor(&c->persist)` into `ha_vote_self_freshness` → floor = `max(durable_floor, learned_committed)`. Comparison unchanged (term-then-index). `ha.c:2243` legacy `handle_request_vote` is XCE_TESTING-only/param-driven — left unchanged. `safe_committed_lsn` UNTOUCHED (voter-reply-sourced). 3 tests in test_ha_vote_gate.c prove: candidate advertises bare durable_floor (not stale learned); voter durable-floor denies truncated; longest-durable grantable / no livelock. bug-339 regression test fixture primed `durable_lsn=10` (model-preserving). INFO: truncated-deny proven at the freshness-comparison-expression level shared by both vote+PreVote paths (no dedicated live PreVote-handler truncation driver — adequate; nice-to-have for 5.x). GOTCHA for future election tests: the freshness floor now reads `persist_durable_floor`, so any vote/election fixture that sets `applied_lsn`/`last_known_committed_lsn` but not durable state will see floor/advertise=0 — prime `durable_lsn` to the modeled tip. - 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE. +- 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit). diff --git a/src/controller/persist.c b/src/controller/persist.c index 7f71746..f0454c6 100644 --- a/src/controller/persist.c +++ b/src/controller/persist.c @@ -302,6 +302,16 @@ static uint64_t flusher_flush_once(persist_t* p) if (tip > dur) atomic_store_explicit(&p->durable_lsn, tip, memory_order_release); +#ifdef XCE_TESTING + /* Record which thread ran this successful fdatasync (task 5.1 / Req 1.6): + * a liveness test asserts this is the dedicated flusher thread, never the + * caller/iomgr/heartbeat thread. Written under p->lock; read after the + * flusher has caught up, so no extra synchronization is needed. */ + p->test_last_fdatasync_thread = pthread_self(); + atomic_store_explicit(&p->test_last_fdatasync_thread_set, 1, + memory_order_release); +#endif + pthread_mutex_unlock(&p->lock); return tip; } @@ -841,6 +851,18 @@ unsigned persist_test_replicated_fdatasync_count(persist_t* p) memory_order_relaxed); } +pthread_t persist_test_last_fdatasync_thread(persist_t* p) +{ + /* If no flusher fdatasync has advanced durable_lsn yet, return the + * caller's own id so an unprimed read never spuriously looks off-thread. + * After a flush, return the recorded flusher thread id (Req 1.6). */ + if (!p || + atomic_load_explicit(&p->test_last_fdatasync_thread_set, + memory_order_acquire) == 0) + return pthread_self(); + return p->test_last_fdatasync_thread; +} + /* ------------------------------------------------------------------ * * Durability-verification test seam (ha-commit-durability Req 3.2/3.3)* * * diff --git a/src/controller/testsuite/test_commit_durability.c b/src/controller/testsuite/test_commit_durability.c index 159e06f..dd9dc30 100644 --- a/src/controller/testsuite/test_commit_durability.c +++ b/src/controller/testsuite/test_commit_durability.c @@ -14,6 +14,8 @@ #include "config.h" #include "ha.h" /* task 4.3: ha_promotion_gate_decision / fence */ #include "persist.h" +#include "repl.h" /* task 5.1: repl_add_standby + standby ACK feed */ +#include "typemap.h" /* task 5.1: wal_catchup_entry_t (send hook) */ #include "wal_types.h" /* ------------------------------------------------------------------ * @@ -779,6 +781,365 @@ static void test_commit_never_exceeds_durable_invariant(void) printf("test_commit_never_exceeds_durable_invariant: PASS\n"); } +/* ================================================================== * + * task 5.1 — wire durable-majority commit end to end * + * * + * These three tests verify the full durable-majority commit path the * + * 2.x/4.x wiring assembled, end to end over a REAL persist_t (real WAL * + * + real group-commit flusher thread) and a REAL commit_engine_t with * + * registered standbys: * + * * + * 1. reply-ok ⇒ durable-majority (Req 1.1, 1.4): committed_lsn * + * advances and a WAIT caller releases ONLY after the entry is * + * durable on a fault-tolerant majority — self durable (durable_lsn * + * covers it, via the real flusher) AND a standby ACK. With * + * durability held off, even a standby ACK does NOT release. * + * 2. sole-durable-contributor loss survives (Req 1.5, 2.5): an op * + * committed on self + a standby is durable on >= floor(N/2)+1 * + * members, so losing one durable contributor leaves it present on * + * a surviving member (modeled at the integration level; the live * + * failover proof is task 5.2). * + * 3. liveness — durability work off the iomgr (Req 1.6): fdatasync * + * runs on the dedicated flusher thread, not the caller/iomgr * + * thread, so a heartbeat-style callback is not stalled while a * + * durable flush is in flight. * + * ------------------------------------------------------------------ */ + +/* Dummy standby endpoints — the replicator only compares pointers, and + * commit_on_ack reads acked_lsn from the registry slot. */ +static int g_dur_eps[MAX_STANDBYS]; + +/* Test-injectable send hook (defined in repl.c under XCE_TESTING) — keeps + * repl_broadcast from dereferencing the dummy endpoints as real + * endpoint_t when persist_wal_async fans out to the registered standbys. + * Same pattern as test_commit_engine.c. */ +extern int (*repl_broadcast_send_hook)(void* ep, wal_catchup_entry_t** entries, + int n_entries); + +static int dur_drop_send_hook(void* ep, wal_catchup_entry_t** entries, + int n_entries) +{ + (void)ep; + (void)entries; + (void)n_entries; + return 0; /* pretend success; drop the data (no real wire in a unit test) */ +} + +/* Feed a standby ACK exactly as the iomgr read path does: store the + * watermark into the registry slot, then notify the commit engine. */ +static void durable_standby_ack(persist_t* p, commit_engine_t* ce, int idx, + uint64_t acked_lsn) +{ + atomic_store(&p->repl.standbys[idx].acked_lsn, acked_lsn); + commit_on_ack(ce, idx, acked_lsn); +} + +/* A WAIT submit that runs on its own thread, so the test can observe that + * it does NOT return while the entry is not yet durable-majority, then DOES + * return once it becomes durable-majority (mirrors the off-iomgr AFTER_EVENT + * caller blocking on the commit barrier). */ +typedef struct { + commit_engine_t* ce; + uint64_t lsn; + uint32_t timeout_ms; + _Atomic int returned; + int rc; +} wait_submit_arg_t; + +static void* wait_submit_thread(void* a) +{ + wait_submit_arg_t* w = (wait_submit_arg_t*)a; + w->rc = commit_submit(w->ce, w->lsn, COMMIT_MODE_WAIT, w->timeout_ms); + atomic_store(&w->returned, 1); + return NULL; +} + +/* ------------------------------------------------------------------ * + * Test 5.1a (MUST-HAVE, Req 1.1 / 1.4): reply-ok only after a durable * + * majority. 3-member cluster (self + 2 standbys), so the majority for * + * an entry is self + at least 1 standby. We: * + * - force the local fdatasync to FAIL, so the flusher cannot advance * + * durable_lsn (self is NOT durable); * + * - append an entry and deliver a standby ACK covering it; * + * - assert committed_lsn does NOT advance and a WAIT submit does NOT * + * release — a majority of ACKs is not enough without local * + * durability (this is the durable precondition gate; pre-2.2 this * + * would have committed on the ACK alone — RED); * + * - clear the fault so the real flusher fdatasyncs and advances * + * durable_lsn, firing the durable-advance wake; * + * - assert committed_lsn now advances and the WAIT submit releases * + * promptly (durable self + durable standby ACK = durable majority). * + * ------------------------------------------------------------------ */ +static void test_durable_majority_commit_end_to_end(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_durmaj_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + persist_t* p = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p); + + /* Intercept the replication fan-out so persist_wal_async does not try to + * send over the dummy endpoints (they are not real endpoint_t). */ + repl_broadcast_send_hook = dur_drop_send_hook; + + /* Register two standbys against the REAL persist's replicator. n_peers=3 + * (self + 2 standbys); majority for entry N = self durable + >= 1 standby + * with acked_lsn >= N. n_vcpus high enough to admit a WAIT. */ + TEST_ASSERT_EQUAL_INT(0, repl_add_standby(&p->repl, &g_dur_eps[0], "sb0", + 7000)); + TEST_ASSERT_EQUAL_INT(1, repl_add_standby(&p->repl, &g_dur_eps[1], "sb1", + 7001)); + + commit_engine_t ce; + TEST_ASSERT_EQUAL_INT(0, commit_engine_init(&ce, p, /*n_peers=*/3, + /*n_vcpus=*/8)); + + TEST_ASSERT_EQUAL_UINT64(0, atomic_load(&p->committed_lsn)); + + /* Hold durability off: every flusher fdatasync is forced to fail, so + * durable_lsn cannot advance. (Fail-loud: the entry is never counted + * durable, so the commit cannot advance regardless of ACKs.) */ + persist_test_set_force_fsync_fail(p, 1); + + /* Append one entry (LSN 1). It is in page cache only; the forced-fail + * flusher will NOT make it durable. */ + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1)); + uint64_t lsn = p->current_lsn - 1; /* 1 */ + TEST_ASSERT_EQUAL_UINT64(1, lsn); + + /* Register a WAIT for LSN 1 on its own thread (the off-iomgr commit-barrier + * caller). It must block: the entry is not durable-majority. */ + wait_submit_arg_t w = {.ce = &ce, .lsn = lsn, .timeout_ms = 2000}; + pthread_t th; + TEST_ASSERT_EQUAL_INT(0, pthread_create(&th, NULL, wait_submit_thread, &w)); + + /* Deliver a covering standby ACK from BOTH standbys (a clear majority of + * ACKs). Pre-durability this alone advanced committed_lsn (the bug); + * post-2.2 the self-cap holds it at the non-durable self (durable_lsn=0). */ + durable_standby_ack(p, &ce, 0, lsn); + durable_standby_ack(p, &ce, 1, lsn); + + /* Give the (forced-failing) flusher ample time to run; durable_lsn must + * stay 0, committed must stay 0, and the WAIT must NOT have returned. */ + for (int i = 0; i < 300; i++) { /* up to ~300 ms */ + if (persist_durable_lsn(p) != 0 || + atomic_load(&p->committed_lsn) != 0 || + atomic_load(&w.returned)) + break; + usleep(1000); + } + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 0, persist_durable_lsn(p), + "durable_lsn must not advance while fdatasync is failing"); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 0, atomic_load(&p->committed_lsn), + "committed_lsn must NOT advance on a standby-ACK majority while the " + "local entry is not durable (durable precondition gate)"); + TEST_ASSERT_FALSE_MESSAGE( + atomic_load(&w.returned), + "the WAIT (client reply) must NOT release before durable-majority"); + + /* Now allow durability: clear the fault and force a flush. The REAL + * flusher fdatasyncs, advances durable_lsn over LSN 1, and fires the + * durable-advance wake — self is now durable AND both standbys already + * ACKed, so the entry is durable on a majority. committed_lsn advances + * and the WAIT releases promptly. */ + persist_test_set_force_fsync_fail(p, 0); + TEST_ASSERT_EQUAL_INT(0, persist_flush(p)); + TEST_ASSERT_TRUE(persist_durable_lsn(p) >= lsn); + + /* The WAIT must now return success (committed) within its timeout. */ + TEST_ASSERT_EQUAL_INT(0, pthread_join(th, NULL)); + TEST_ASSERT_TRUE_MESSAGE( + atomic_load(&w.returned), + "the WAIT must release once the entry is durable on a majority"); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, w.rc, + "commit_submit(WAIT) must return 0 (committed) after durable-majority"); + + TEST_ASSERT_TRUE_MESSAGE( + atomic_load(&p->committed_lsn) >= lsn, + "committed_lsn must advance once self is durable AND a standby ACK " + "covers the entry (durable majority)"); + /* The invariant the whole spec rests on: committed never exceeds durable. */ + TEST_ASSERT_TRUE(atomic_load(&p->committed_lsn) <= persist_durable_lsn(p)); + + commit_engine_fini(&ce); + repl_remove_standby(&p->repl, &g_dur_eps[0]); + repl_remove_standby(&p->repl, &g_dur_eps[1]); + teardown_durability_persist(p); + repl_broadcast_send_hook = NULL; + printf("test_durable_majority_commit_end_to_end: PASS\n"); +} + +/* ------------------------------------------------------------------ * + * Test 5.1b (Req 1.5 / 2.5): losing the sole durable contributor * + * leaves the acknowledged op present on a surviving member. * + * * + * The durable-majority property says: when an op is acknowledged, its * + * entry is fdatasync'd on a fault-tolerant majority (self + >= * + * floor(N/2) others). For N=3 that is 2 durable copies. So losing ONE * + * durable contributor still leaves >= 1 surviving durable copy that * + * holds the op — it is recoverable from the survivor. * + * * + * We model the two durable copies as two independent persist_t over * + * separate WAL dirs (a primary's copy and a standby's copy), both made * + * genuinely durable (fdatasync'd). We then DROP the "sole durable * + * contributor" (the first copy, simulating its loss/restart-with-loss) * + * and prove the op is still durably present and recoverable on the * + * surviving copy via a fresh-process recovery (no in-memory state). * + * * + * This is the integration-level assertion of the property that makes * + * survival true (committed ⇒ >= floor(N/2)+1 durable copies); the live * + * multi-host SIGKILL failover proof is task 5.2. * + * ------------------------------------------------------------------ */ +static void test_sole_durable_contributor_loss_survives(void) +{ + char tmp_a[256]; + char tmp_b[256]; + strcpy(tmp_a, "/tmp/xce_durlossA_XXXXXX"); + strcpy(tmp_b, "/tmp/xce_durlossB_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp_a)); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp_b)); + + const char* payload = "committed-op"; + uint32_t plen = (uint32_t)strlen(payload); + + /* Durable copy A (the contributor we will lose) and durable copy B (the + * survivor): write the SAME committed op to both and fdatasync each, so + * the op is durable on a 2-of-3 majority. */ + { + persist_t* a = make_durability_persist(tmp_a); + TEST_ASSERT_NOT_NULL(a); + TEST_ASSERT_EQUAL_INT(0, + persist_wal_async(a, WAL_JOB_UPDATE, payload, plen)); + TEST_ASSERT_EQUAL_INT(0, persist_flush(a)); + TEST_ASSERT_TRUE(persist_durable_lsn(a) >= 1); + teardown_durability_persist(a); + } + { + persist_t* b = make_durability_persist(tmp_b); + TEST_ASSERT_NOT_NULL(b); + /* The standby applies the SAME replicated entry durably before ACK + * (persist_wal_write_replicated fdatasyncs inline — Req 1.3). */ + TEST_ASSERT_EQUAL_INT(0, + persist_wal_write_replicated(b, /*lsn=*/1, WAL_JOB_UPDATE, + payload, plen)); + TEST_ASSERT_TRUE(persist_test_durable_lsn_on_disk(b) >= 1); + teardown_durability_persist(b); + } + + /* "Lose" the sole durable contributor A: destroy its on-disk WAL entirely + * (simulates a member that is gone / came back with a truncated log). */ + { + char rm[600]; + snprintf(rm, sizeof(rm), "rm -rf %s", tmp_a); + TEST_ASSERT_EQUAL_INT(0, system(rm)); + } + + /* The acknowledged op MUST still be present and recoverable on the + * surviving member B — read straight from disk by a fresh process with + * NO in-memory state (the durable-majority property in action). */ + { + persist_t* b2 = make_durability_persist(tmp_b); + TEST_ASSERT_NOT_NULL(b2); + + /* On-disk durable floor (CRC-validated re-read) still covers the op. */ + TEST_ASSERT_TRUE_MESSAGE( + persist_test_durable_lsn_on_disk(b2) >= 1, + "the acknowledged op must survive on the durable majority after " + "the sole-durable-contributor copy is lost"); + + /* And a real recovery reconstructs it cleanly (not truncated). */ + TEST_ASSERT_EQUAL_INT(0, persist_recover(b2)); + TEST_ASSERT_EQUAL_INT_MESSAGE( + RECOVER_CLEAN, persist_recovery_verdict(b2), + "the survivor recovers the committed op as a clean intact log"); + TEST_ASSERT_TRUE_MESSAGE( + persist_durable_floor(b2) >= 1, + "the survivor's recovered durable floor covers the committed op"); + + teardown_durability_persist(b2); + } + + { + char rm[600]; + snprintf(rm, sizeof(rm), "rm -rf %s", tmp_b); + (void)system(rm); + } + printf("test_sole_durable_contributor_loss_survives: PASS " + "(durable-majority property; live failover proof is task 5.2)\n"); +} + +/* ------------------------------------------------------------------ * + * Test 5.1c (Req 1.6): durability work does not stall heartbeat / * + * election handling — fdatasync runs on the dedicated flusher thread, * + * NOT on the caller (iomgr/heartbeat) thread. * + * * + * Structural + timing assertion: while a durable flush is in flight on * + * the flusher thread, a heartbeat-style callback executed on the * + * caller/iomgr thread completes within a tight tolerance. We prove the * + * flush is OFF the caller thread by capturing the flusher's thread id at * + * the fdatasync site (via the XCE_TESTING seam) and asserting it is NOT * + * the caller thread. (persist_wal_async only appends + signals; it never * + * fdatasyncs inline, so the append path itself never blocks on disk.) * + * ------------------------------------------------------------------ */ +static void test_durability_work_off_iomgr_preserves_liveness(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_durliveness_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + persist_t* p = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p); + + pthread_t caller = pthread_self(); + + /* Drive a steady stream of appends on the CALLER (iomgr-equivalent) + * thread; the real flusher thread does the fdatasync work concurrently. + * After each append, a heartbeat-style callback must complete promptly — + * the append path signals the flusher and returns without blocking on + * fdatasync, so heartbeat handling on this thread is never stalled. */ + const long tolerance_us = 50 * 1000; /* 50 ms — generous heartbeat budget */ + for (int i = 0; i < 64; i++) { + TEST_ASSERT_EQUAL_INT(0, + persist_wal_async(p, WAL_JOB_UPDATE, "h", 1)); + + struct timespec t0, t1; + clock_gettime(CLOCK_MONOTONIC, &t0); + /* Heartbeat-style work on the caller thread (a cheap atomic read of + * commit/durable state, exactly what the heartbeat path inspects). */ + volatile uint64_t hb = atomic_load(&p->committed_lsn) + + persist_durable_lsn(p); + (void)hb; + clock_gettime(CLOCK_MONOTONIC, &t1); + + long elapsed_us = (t1.tv_sec - t0.tv_sec) * 1000000L + + (t1.tv_nsec - t0.tv_nsec) / 1000L; + TEST_ASSERT_TRUE_MESSAGE( + elapsed_us < tolerance_us, + "heartbeat-style work on the caller thread must not be stalled " + "by durability (fdatasync) work in flight on the flusher thread"); + } + + /* Let the flusher make everything durable, then confirm the fdatasync + * provably ran on a DIFFERENT thread than this caller — durability work + * is off the iomgr/heartbeat thread (Req 1.6). The seam records the + * thread id of the last successful flusher fdatasync. */ + uint64_t tip = p->current_lsn - 1; + TEST_ASSERT_TRUE(wait_durable_reaches(p, tip, 5000) >= tip); + + pthread_t flusher_tid = persist_test_last_fdatasync_thread(p); + TEST_ASSERT_FALSE_MESSAGE( + pthread_equal(flusher_tid, caller), + "fdatasync must run on the dedicated flusher thread, never on the " + "caller/iomgr/heartbeat thread"); + + teardown_durability_persist(p); + printf("test_durability_work_off_iomgr_preserves_liveness: PASS\n"); +} + /* ================================================================== * * task 3.1 — recovery integrity verdict (Req 2.1) * * * @@ -1662,6 +2023,10 @@ void test_commit_durability(void) RUN_TEST(test_legacy_sync_append_holds_committed_at_durable); RUN_TEST(test_legacy_sync_append_lsn0_edge_valid); RUN_TEST(test_commit_never_exceeds_durable_invariant); + /* task 5.1 — durable-majority commit wired end to end */ + RUN_TEST(test_durable_majority_commit_end_to_end); + RUN_TEST(test_sole_durable_contributor_loss_survives); + RUN_TEST(test_durability_work_off_iomgr_preserves_liveness); RUN_TEST(test_recover_verdict_clean_intact_log); RUN_TEST(test_recover_verdict_empty_first_boot_clean); RUN_TEST(test_recover_verdict_crc_break_first_entry_truncated); diff --git a/src/include/persist.h b/src/include/persist.h index 5d6a4b7..44cbda2 100644 --- a/src/include/persist.h +++ b/src/include/persist.h @@ -205,6 +205,13 @@ struct persist_s { * cannot distinguish fsync'd from cache-only, so a direct sync signal is * required. Compiled out of production. */ atomic_uint test_replicated_fdatasync_count; + /* Liveness observability (ha-commit-durability task 5.1, Req 1.6): + * the thread id of the last flusher fdatasync that advanced durable_lsn. + * A test reads this to prove fdatasync runs on the dedicated flusher + * thread, NOT on the caller/iomgr/heartbeat thread. Compiled out of + * production. */ + pthread_t test_last_fdatasync_thread; + atomic_int test_last_fdatasync_thread_set; #endif pthread_t fsync_thread; pthread_mutex_t fsync_mutex; /* guards the fsync timer wait */ @@ -450,6 +457,15 @@ unsigned persist_test_replicated_fdatasync_count(persist_t* p); * standing up the replication wire. Compiled out of production. */ void persist_test_sync_append_legacy_commit_store(persist_t* p, uint64_t written_lsn); + +/* Return the thread id of the last flusher fdatasync that advanced + * durable_lsn (task 5.1 / Req 1.6 liveness). A test asserts this is NOT the + * caller/iomgr thread — proving durability (fdatasync) work runs on the + * dedicated flusher thread and never stalls heartbeat/election handling. + * Returns pthread_self() if no flusher sync has advanced durable_lsn yet + * (so an unprimed read can never falsely look "off-thread"). Compiled out of + * production. */ +pthread_t persist_test_last_fdatasync_thread(persist_t* p); #endif /* XCE_TESTING */ /* Checkpoint */