diff --git a/.kiro/specs/ha-commit-durability/tasks.md b/.kiro/specs/ha-commit-durability/tasks.md index e87b9027..e8abe6a3 100644 --- a/.kiro/specs/ha-commit-durability/tasks.md +++ b/.kiro/specs/ha-commit-durability/tasks.md @@ -70,7 +70,7 @@ - _Depends: 3.1, 3.2_ - _Boundary: RecoveryIntegrity_ -- [ ] 3.4 Re-derive the durable floor from the recovered WAL tip +- [x] 3.4 Re-derive the durable floor from the recovered WAL tip - On recovery, publish the member's durable floor as the highest intact replayed LSN so later advertisement is backed by durable state rather than a volatile in-memory position. - Precision: this floor is the durable **last-log index** (WAL tip), used by the Raft freshness gate — not the commit position; the post-election safe-commit point continues to come from voter replies, not from this floor. - Observable: a unit test shows that after a truncating restart the published durable floor equals the highest intact LSN (the WAL tip), not a stale higher in-memory value or the commit position. @@ -130,3 +130,4 @@ - 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs. - 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code. - 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits). +- 3.4: On recovery, `persist_recover` publishes the durable floor = highest intact replayed LSN (recovered WAL tip = `current_lsn-1` after `replay_wal`) into `durable_lsn` via `atomic_store` (ASSIGN, NOT max — so a truncating restart lands at the LOWER intact tip, never over-advertising; this is the crux of Req 2.3). Added `persist_durable_floor()` (NULL-safe thin accessor) as the seam task 4.2 consumes. Floor is the durable LAST-LOG INDEX (WAL tip), NOT the commit position (`committed_lsn`) and NOT a volatile in-memory value. Invariant-safe (recovery/init-time; recovered entries are CRC-validated on-disk = genuinely durable; preserves `durable_lsn <= current_lsn-1`). 4.2 (ha.c/cluster.c) CONSUMES this floor in `ha_vote_self_freshness` — 3.4 only PUBLISHES (persistence-only). Phase 3 (recovery integrity) COMPLETE. diff --git a/src/controller/persist.c b/src/controller/persist.c index ef0de997..7f717465 100644 --- a/src/controller/persist.c +++ b/src/controller/persist.c @@ -804,6 +804,16 @@ uint64_t persist_durable_lsn(persist_t* p) return atomic_load_explicit(&p->durable_lsn, memory_order_acquire); } +uint64_t persist_durable_floor(persist_t* p) +{ + /* The durable freshness floor (durable last-log index) IS the durable + * watermark: persist_recover re-derives it from the recovered WAL tip + * (ha-commit-durability task 3.4), and the flusher advances it on every + * successful fdatasync — so a thin wrapper avoids a redundant field. NULL + * guard keeps the accessor safe for the election consumers (task 4.2). */ + return p ? atomic_load_explicit(&p->durable_lsn, memory_order_acquire) : 0; +} + void persist_set_durable_cb(persist_t* p, void (*fn)(void* ctx), void* ctx) { if (!p) @@ -1626,6 +1636,31 @@ int persist_recover(persist_t* p) "corrupt (CRC break / mismatch / short read); member marked " "recovery-truncated (non-promotable until caught up)"); + /* --- Step 3b: re-derive the durable floor from the recovered WAL tip + * (ha-commit-durability task 3.4, Req 2.3) --- + * + * Publish the member's durable floor (the durable last-log index consumed + * by the Raft freshness gate, task 4.2) as the highest INTACT replayed LSN. + * replay_wal advanced current_lsn to one past the last intact entry — incl. + * stopping at a corruption boundary on TRUNCATED — so current_lsn - 1 is + * exactly that intact WAL tip, and a snapshot-sentinel load leaves it at the + * sentinel/replayed tip. We store it into durable_lsn because the recovered + * on-disk entries are GENUINELY durable (each was CRC-validated and read + * from stable storage by replay_wal), so the freshness advertisement is now + * backed by durable state rather than a volatile in-memory position. + * + * Crux (Req 2.3 — a truncated member cannot over-advertise): this is an + * ASSIGN to the recovered tip, NOT a max() with whatever durable_lsn held. + * A fresh recover starts from durable_lsn=0 (persist_init), so on a + * truncating restart the floor lands at the LOWER intact tip — never a + * stale higher in-memory value and never the commit position + * (committed_lsn). The recovered tip is, by construction, <= current_lsn-1, + * preserving the durable_lsn <= current_lsn-1 invariant. The post-election + * safe-commit point continues to come from voter replies, not this floor + * (CLAUDE.md). */ + uint64_t recovered_tip = (p->current_lsn > 0) ? p->current_lsn - 1 : 0; + atomic_store_explicit(&p->durable_lsn, recovered_tip, memory_order_release); + /* --- Step 4: retention pass (TTL check) — no-op in P0 (no entities) --- */ return 0; diff --git a/src/controller/testsuite/test_commit_durability.c b/src/controller/testsuite/test_commit_durability.c index 103ef59a..01585830 100644 --- a/src/controller/testsuite/test_commit_durability.c +++ b/src/controller/testsuite/test_commit_durability.c @@ -990,6 +990,212 @@ static void test_recover_verdict_snapshot_sentinel_clean(void) printf("test_recover_verdict_snapshot_sentinel_clean: PASS\n"); } +/* ================================================================== * + * task 3.4 — re-derive the durable floor from the recovered WAL tip * + * (Req 2.3) * + * * + * On recovery the member publishes its durable floor as the highest * + * INTACT replayed LSN (the WAL tip / durable last-log index), so the * + * freshness advertisement (task 4.2) is backed by durable on-disk * + * state rather than a volatile in-memory position. The floor is the * + * durable last-log index, NOT the commit position, and NOT a stale * + * higher in-memory value. persist_durable_floor() exposes it (a thin * + * wrapper over the durable_lsn watermark, which recovery sets to the * + * recovered tip). * + * ------------------------------------------------------------------ */ + +/* Test 3.4a (CRUX, Req 2.3): after a TRUNCATING restart the published + * durable floor equals the highest INTACT LSN (the WAL tip), NOT a stale + * higher in-memory value and NOT the commit position. + * + * Writer lays down 4 durable entries (LSN 1..4). We then corrupt entry 3's + * CRC on disk so the intact prefix is LSN 1..2 — replay stops at the chain + * break and the recovered tip is LSN 2. To make the "not a stale higher + * in-memory value" assertion meaningful we deliberately seed the fresh + * instance's in-memory durable_lsn / committed_lsn to an over-advertised + * value (4) BEFORE recovery; recovery must publish the durable FLOOR at the + * intact tip 2, never the seeded 4. */ +static void test_durable_floor_truncating_restart_is_intact_tip(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_floortrunc_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + /* Writer: four durable entries (LSN 1..4), then die. */ + { + persist_t* p = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "c", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "d", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_flush(p)); + /* All four are durable on disk before the truncation. */ + TEST_ASSERT_EQUAL_UINT64(4, persist_test_durable_lsn_on_disk(p)); + teardown_durability_persist(p); + } + + /* Corrupt entry 3 (LSN 3): walk the segment to its offset and flip its + * crc. Each entry is sizeof(hdr) + payload_len (=1) bytes; entries 1 and 2 + * precede it. This breaks the chain at LSN 3, so the intact tip is LSN 2. */ + { + int fd = open_first_seg_rw(tmp); + TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "first WAL segment must exist"); + + uint32_t off = 0; + wal_entry_hdr_t hdr; + /* Skip entries 1 and 2 (LSN 1, LSN 2). */ + for (int i = 0; i < 2; i++) { + TEST_ASSERT_EQUAL_INT((int)sizeof(hdr), + (int)pread(fd, &hdr, sizeof(hdr), (off_t)off)); + off += (uint32_t)(sizeof(hdr) + hdr.payload_len); + } + /* Entry 3 (LSN 3) is at `off`: corrupt its CRC. */ + TEST_ASSERT_EQUAL_INT((int)sizeof(hdr), + (int)pread(fd, &hdr, sizeof(hdr), (off_t)off)); + TEST_ASSERT_EQUAL_UINT64(3, hdr.lsn); + hdr.crc ^= 0xDEADBEEFu; + TEST_ASSERT_EQUAL_INT((int)sizeof(hdr), + (int)pwrite(fd, &hdr, sizeof(hdr), (off_t)off)); + fsync(fd); + close(fd); + } + + /* Recover into a fresh instance that we deliberately poison with a stale, + * over-advertised in-memory position (4) BEFORE recovery. */ + { + persist_t* p2 = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p2); + + /* Stale higher in-memory values: simulate a volatile position that + * over-states what is durable. The published floor must NOT come from + * these — it must come from the recovered intact WAL tip. */ + atomic_store(&p2->durable_lsn, 4); + atomic_store(&p2->committed_lsn, 4); + + int rc = persist_recover(p2); + TEST_ASSERT_EQUAL_INT(0, rc); + + /* Verdict is TRUNCATED (chain break at LSN 3). */ + TEST_ASSERT_EQUAL_INT_MESSAGE( + RECOVER_TRUNCATED, persist_recovery_verdict(p2), + "a CRC break at LSN 3 must yield RECOVER_TRUNCATED"); + + /* The published durable floor == the highest INTACT LSN (2): the WAL + * tip, equal to current_lsn - 1 after replay (replay advanced + * current_lsn only to one past the last intact entry). */ + uint64_t floor = persist_durable_floor(p2); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 2, floor, + "durable floor after a truncating restart must equal the highest " + "intact LSN (the WAL tip), NOT the stale in-memory value"); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + p2->current_lsn - 1, floor, + "durable floor must equal the recovered WAL tip (current_lsn - 1)"); + /* Independently confirmed against the on-disk re-read seam. */ + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 2, persist_test_durable_lsn_on_disk(p2), + "on-disk re-read agrees: only LSN 1..2 are intact"); + + teardown_durability_persist(p2); + } + printf("test_durable_floor_truncating_restart_is_intact_tip: PASS\n"); +} + +/* Test 3.4b: an INTACT-log recovery publishes the FULL tip (no truncation). + * Three durable entries (LSN 1..3) recover cleanly; the floor is the full + * recovered tip 3 == current_lsn - 1. */ +static void test_durable_floor_intact_recovery_full_tip(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_floorintact_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + { + persist_t* p = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "c", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_flush(p)); + teardown_durability_persist(p); + } + { + persist_t* p2 = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p2); + int rc = persist_recover(p2); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(RECOVER_CLEAN, persist_recovery_verdict(p2)); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 3, persist_durable_floor(p2), + "intact recovery publishes the full recovered WAL tip"); + TEST_ASSERT_EQUAL_UINT64(p2->current_lsn - 1, persist_durable_floor(p2)); + teardown_durability_persist(p2); + } + printf("test_durable_floor_intact_recovery_full_tip: PASS\n"); +} + +/* Test 3.4c: an EMPTY first-boot WAL publishes floor 0 (nothing durable). */ +static void test_durable_floor_empty_recovery_zero(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_floorempty_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + persist_t* p = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p); + int rc = persist_recover(p); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 0, persist_durable_floor(p), + "an empty first-boot WAL has nothing durable — floor is 0"); + teardown_durability_persist(p); + printf("test_durable_floor_empty_recovery_zero: PASS\n"); +} + +/* Test 3.4d (snapshot-sentinel edge): a member recovered from a 16-byte + * LSN-sentinel snapshot plus a clean replay above it publishes the recovered + * tip (the highest replayed LSN). The floor is the recovered tip, consistent + * with 3.1's CLEAN classification — "behind-and-must-catch-up", not + * over-advertising. */ +static void test_durable_floor_snapshot_sentinel_tip(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_floorsnap_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + { + persist_t* p = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_flush(p)); + /* Sentinel snapshot at the checkpoint position; then one entry above. */ + TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(p)); + TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "c", 1)); + TEST_ASSERT_EQUAL_INT(0, persist_flush(p)); + teardown_durability_persist(p); + } + { + persist_t* p2 = make_durability_persist(tmp); + TEST_ASSERT_NOT_NULL(p2); + int rc = persist_recover(p2); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(RECOVER_CLEAN, persist_recovery_verdict(p2)); + /* The floor is the recovered tip = current_lsn - 1 (whatever the + * highest replayed LSN above the sentinel is). It is the recovered + * tip, not a stale higher value. */ + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + p2->current_lsn - 1, persist_durable_floor(p2), + "snapshot-sentinel recovery publishes the recovered WAL tip"); + TEST_ASSERT_TRUE_MESSAGE( + persist_durable_floor(p2) > 0, + "a sentinel + clean replay-above has a non-zero recovered tip"); + teardown_durability_persist(p2); + } + printf("test_durable_floor_snapshot_sentinel_tip: PASS\n"); +} + /* ================================================================== * * task 3.2 — Boot Policy Decision Table (DECISION ONLY) * * * @@ -1274,6 +1480,11 @@ void test_commit_durability(void) RUN_TEST(test_recover_verdict_empty_first_boot_clean); RUN_TEST(test_recover_verdict_crc_break_first_entry_truncated); RUN_TEST(test_recover_verdict_snapshot_sentinel_clean); + /* task 3.4 — re-derive the durable floor from the recovered WAL tip. */ + RUN_TEST(test_durable_floor_truncating_restart_is_intact_tip); + RUN_TEST(test_durable_floor_intact_recovery_full_tip); + RUN_TEST(test_durable_floor_empty_recovery_zero); + RUN_TEST(test_durable_floor_snapshot_sentinel_tip); /* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending * 3.3/4.3). Registered so each row is reported as IGNORED, not failed; * the suite stays GREEN. See design.md "Boot Policy Decision Table". */ diff --git a/src/include/persist.h b/src/include/persist.h index 7244e257..5d6a4b7f 100644 --- a/src/include/persist.h +++ b/src/include/persist.h @@ -307,6 +307,29 @@ int persist_flush(persist_t* p); * any thread. Returns 0 when nothing is yet durable. */ uint64_t persist_durable_lsn(persist_t* p); +/* Durable freshness floor (ha-commit-durability task 3.4, Req 2.3). + * + * The member's durably-backed last-log index (WAL tip): the highest LSN that + * is on stable storage locally. persist_recover re-derives it from the + * recovered WAL tip (the highest INTACT replayed LSN) so the post-recovery + * value is backed by durable on-disk state, NOT a volatile in-memory + * applied_lsn / last_known_committed_lsn. The Raft freshness gate (task 4.2, + * ha.c / cluster.c) consumes this so a truncated member cannot over-advertise + * a tip it does not durably hold. + * + * Precision: this is the durable LAST-LOG INDEX, NOT the commit position + * (committed_lsn). The post-election safe-commit point still comes from voter + * replies, never from this floor (CLAUDE.md). After a truncating restart the + * floor equals the lower intact tip, never a stale higher in-memory value and + * never the commit position. + * + * Thin wrapper over persist_durable_lsn: at and after recovery the durable + * watermark IS the recovered WAL tip (recovered on-disk entries are genuinely + * durable), so a separate field would be redundant. Lock-free atomic read; + * safe from any thread. Returns 0 when nothing is durable (empty first-boot + * WAL). */ +uint64_t persist_durable_floor(persist_t* p); + /* Recovery-integrity verdict + flag (ha-commit-durability task 3.1, Req 2.1). * persist_recovery_verdict returns the RECOVER_CLEAN / RECOVER_TRUNCATED * verdict computed by the last persist_recover (RECOVER_CLEAN on a fresh