lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(ha-commit-durability): publish durable floor from recovered WAL tip (task 3.4)

Browse source
This commit is contained in:
Rongsong Shen 2026-06-18 08:24:46 +00:00
parent e3fc42c11e
commit c1c35ab1b3
4 changed files with 271 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.kiro/specs/ha-commit-durability/tasks.md
View file

@ -70,7 +70,7 @@
- _Depends: 3.1, 3.2_
- _Boundary: RecoveryIntegrity_
- [ ] 3.4 Re-derive the durable floor from the recovered WAL tip
- [x] 3.4 Re-derive the durable floor from the recovered WAL tip
- On recovery, publish the member's durable floor as the highest intact replayed LSN so later advertisement is backed by durable state rather than a volatile in-memory position.
- Precision: this floor is the durable **last-log index** (WAL tip), used by the Raft freshness gate — not the commit position; the post-election safe-commit point continues to come from voter replies, not from this floor.
- Observable: a unit test shows that after a truncating restart the published durable floor equals the highest intact LSN (the WAL tip), not a stale higher in-memory value or the commit position.
@ -130,3 +130,4 @@
- 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs.
- 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code.
- 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits).
- 3.4: On recovery, `persist_recover` publishes the durable floor = highest intact replayed LSN (recovered WAL tip = `current_lsn-1` after `replay_wal`) into `durable_lsn` via `atomic_store` (ASSIGN, NOT max — so a truncating restart lands at the LOWER intact tip, never over-advertising; this is the crux of Req 2.3). Added `persist_durable_floor()` (NULL-safe thin accessor) as the seam task 4.2 consumes. Floor is the durable LAST-LOG INDEX (WAL tip), NOT the commit position (`committed_lsn`) and NOT a volatile in-memory value. Invariant-safe (recovery/init-time; recovered entries are CRC-validated on-disk = genuinely durable; preserves `durable_lsn <= current_lsn-1`). 4.2 (ha.c/cluster.c) CONSUMES this floor in `ha_vote_self_freshness` — 3.4 only PUBLISHES (persistence-only). Phase 3 (recovery integrity) COMPLETE.

35
src/controller/persist.c
View file

@ -804,6 +804,16 @@ uint64_t persist_durable_lsn(persist_t* p)
return atomic_load_explicit(&p->durable_lsn, memory_order_acquire);
}
uint64_t persist_durable_floor(persist_t* p)
{
/* The durable freshness floor (durable last-log index) IS the durable
* watermark: persist_recover re-derives it from the recovered WAL tip
* (ha-commit-durability task 3.4), and the flusher advances it on every
* successful fdatasync — so a thin wrapper avoids a redundant field. NULL
* guard keeps the accessor safe for the election consumers (task 4.2). */
return p ? atomic_load_explicit(&p->durable_lsn, memory_order_acquire) : 0;
}
void persist_set_durable_cb(persist_t* p, void (*fn)(void* ctx), void* ctx)
{
if (!p)
@ -1626,6 +1636,31 @@ int persist_recover(persist_t* p)
"corrupt (CRC break / mismatch / short read); member marked "
"recovery-truncated (non-promotable until caught up)");
/* --- Step 3b: re-derive the durable floor from the recovered WAL tip
* (ha-commit-durability task 3.4, Req 2.3) ---
*
* Publish the member's durable floor (the durable last-log index consumed
* by the Raft freshness gate, task 4.2) as the highest INTACT replayed LSN.
* replay_wal advanced current_lsn to one past the last intact entry — incl.
* stopping at a corruption boundary on TRUNCATED — so current_lsn - 1 is
* exactly that intact WAL tip, and a snapshot-sentinel load leaves it at the
* sentinel/replayed tip. We store it into durable_lsn because the recovered
* on-disk entries are GENUINELY durable (each was CRC-validated and read
* from stable storage by replay_wal), so the freshness advertisement is now
* backed by durable state rather than a volatile in-memory position.
*
* Crux (Req 2.3 — a truncated member cannot over-advertise): this is an
* ASSIGN to the recovered tip, NOT a max() with whatever durable_lsn held.
* A fresh recover starts from durable_lsn=0 (persist_init), so on a
* truncating restart the floor lands at the LOWER intact tip — never a
* stale higher in-memory value and never the commit position
* (committed_lsn). The recovered tip is, by construction, <= current_lsn-1,
* preserving the durable_lsn <= current_lsn-1 invariant. The post-election
* safe-commit point continues to come from voter replies, not this floor
* (CLAUDE.md). */
uint64_t recovered_tip = (p->current_lsn > 0) ? p->current_lsn - 1 : 0;
atomic_store_explicit(&p->durable_lsn, recovered_tip, memory_order_release);
/* --- Step 4: retention pass (TTL check) — no-op in P0 (no entities) --- */
return 0;

211
src/controller/testsuite/test_commit_durability.c
View file

@ -990,6 +990,212 @@ static void test_recover_verdict_snapshot_sentinel_clean(void)
printf("test_recover_verdict_snapshot_sentinel_clean: PASS\n");
}
/* ================================================================== *
* task 3.4 — re-derive the durable floor from the recovered WAL tip *
* (Req 2.3) *
* *
* On recovery the member publishes its durable floor as the highest *
* INTACT replayed LSN (the WAL tip / durable last-log index), so the *
* freshness advertisement (task 4.2) is backed by durable on-disk *
* state rather than a volatile in-memory position. The floor is the *
* durable last-log index, NOT the commit position, and NOT a stale *
* higher in-memory value. persist_durable_floor() exposes it (a thin *
* wrapper over the durable_lsn watermark, which recovery sets to the *
* recovered tip). *
* ------------------------------------------------------------------ */
/* Test 3.4a (CRUX, Req 2.3): after a TRUNCATING restart the published
* durable floor equals the highest INTACT LSN (the WAL tip), NOT a stale
* higher in-memory value and NOT the commit position.
*
* Writer lays down 4 durable entries (LSN 1..4). We then corrupt entry 3's
* CRC on disk so the intact prefix is LSN 1..2 — replay stops at the chain
* break and the recovered tip is LSN 2. To make the "not a stale higher
* in-memory value" assertion meaningful we deliberately seed the fresh
* instance's in-memory durable_lsn / committed_lsn to an over-advertised
* value (4) BEFORE recovery; recovery must publish the durable FLOOR at the
* intact tip 2, never the seeded 4. */
static void test_durable_floor_truncating_restart_is_intact_tip(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_floortrunc_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Writer: four durable entries (LSN 1..4), then die. */
{
persist_t* p = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "c", 1));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "d", 1));
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
/* All four are durable on disk before the truncation. */
TEST_ASSERT_EQUAL_UINT64(4, persist_test_durable_lsn_on_disk(p));
teardown_durability_persist(p);
}
/* Corrupt entry 3 (LSN 3): walk the segment to its offset and flip its
* crc. Each entry is sizeof(hdr) + payload_len (=1) bytes; entries 1 and 2
* precede it. This breaks the chain at LSN 3, so the intact tip is LSN 2. */
{
int fd = open_first_seg_rw(tmp);
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "first WAL segment must exist");
uint32_t off = 0;
wal_entry_hdr_t hdr;
/* Skip entries 1 and 2 (LSN 1, LSN 2). */
for (int i = 0; i < 2; i++) {
TEST_ASSERT_EQUAL_INT((int)sizeof(hdr),
(int)pread(fd, &hdr, sizeof(hdr), (off_t)off));
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
}
/* Entry 3 (LSN 3) is at `off`: corrupt its CRC. */
TEST_ASSERT_EQUAL_INT((int)sizeof(hdr),
(int)pread(fd, &hdr, sizeof(hdr), (off_t)off));
TEST_ASSERT_EQUAL_UINT64(3, hdr.lsn);
hdr.crc ^= 0xDEADBEEFu;
TEST_ASSERT_EQUAL_INT((int)sizeof(hdr),
(int)pwrite(fd, &hdr, sizeof(hdr), (off_t)off));
fsync(fd);
close(fd);
}
/* Recover into a fresh instance that we deliberately poison with a stale,
* over-advertised in-memory position (4) BEFORE recovery. */
{
persist_t* p2 = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
/* Stale higher in-memory values: simulate a volatile position that
* over-states what is durable. The published floor must NOT come from
* these — it must come from the recovered intact WAL tip. */
atomic_store(&p2->durable_lsn, 4);
atomic_store(&p2->committed_lsn, 4);
int rc = persist_recover(p2);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Verdict is TRUNCATED (chain break at LSN 3). */
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_TRUNCATED, persist_recovery_verdict(p2),
"a CRC break at LSN 3 must yield RECOVER_TRUNCATED");
/* The published durable floor == the highest INTACT LSN (2): the WAL
* tip, equal to current_lsn - 1 after replay (replay advanced
* current_lsn only to one past the last intact entry). */
uint64_t floor = persist_durable_floor(p2);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
2, floor,
"durable floor after a truncating restart must equal the highest "
"intact LSN (the WAL tip), NOT the stale in-memory value");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
p2->current_lsn - 1, floor,
"durable floor must equal the recovered WAL tip (current_lsn - 1)");
/* Independently confirmed against the on-disk re-read seam. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
2, persist_test_durable_lsn_on_disk(p2),
"on-disk re-read agrees: only LSN 1..2 are intact");
teardown_durability_persist(p2);
}
printf("test_durable_floor_truncating_restart_is_intact_tip: PASS\n");
}
/* Test 3.4b: an INTACT-log recovery publishes the FULL tip (no truncation).
* Three durable entries (LSN 1..3) recover cleanly; the floor is the full
* recovered tip 3 == current_lsn - 1. */
static void test_durable_floor_intact_recovery_full_tip(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_floorintact_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
{
persist_t* p = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "c", 1));
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
teardown_durability_persist(p);
}
{
persist_t* p2 = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
int rc = persist_recover(p2);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(RECOVER_CLEAN, persist_recovery_verdict(p2));
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
3, persist_durable_floor(p2),
"intact recovery publishes the full recovered WAL tip");
TEST_ASSERT_EQUAL_UINT64(p2->current_lsn - 1, persist_durable_floor(p2));
teardown_durability_persist(p2);
}
printf("test_durable_floor_intact_recovery_full_tip: PASS\n");
}
/* Test 3.4c: an EMPTY first-boot WAL publishes floor 0 (nothing durable). */
static void test_durable_floor_empty_recovery_zero(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_floorempty_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
persist_t* p = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
int rc = persist_recover(p);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
0, persist_durable_floor(p),
"an empty first-boot WAL has nothing durable — floor is 0");
teardown_durability_persist(p);
printf("test_durable_floor_empty_recovery_zero: PASS\n");
}
/* Test 3.4d (snapshot-sentinel edge): a member recovered from a 16-byte
* LSN-sentinel snapshot plus a clean replay above it publishes the recovered
* tip (the highest replayed LSN). The floor is the recovered tip, consistent
* with 3.1's CLEAN classification — "behind-and-must-catch-up", not
* over-advertising. */
static void test_durable_floor_snapshot_sentinel_tip(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_floorsnap_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
{
persist_t* p = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1));
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
/* Sentinel snapshot at the checkpoint position; then one entry above. */
TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(p));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "c", 1));
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
teardown_durability_persist(p);
}
{
persist_t* p2 = make_durability_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
int rc = persist_recover(p2);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(RECOVER_CLEAN, persist_recovery_verdict(p2));
/* The floor is the recovered tip = current_lsn - 1 (whatever the
* highest replayed LSN above the sentinel is). It is the recovered
* tip, not a stale higher value. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
p2->current_lsn - 1, persist_durable_floor(p2),
"snapshot-sentinel recovery publishes the recovered WAL tip");
TEST_ASSERT_TRUE_MESSAGE(
persist_durable_floor(p2) > 0,
"a sentinel + clean replay-above has a non-zero recovered tip");
teardown_durability_persist(p2);
}
printf("test_durable_floor_snapshot_sentinel_tip: PASS\n");
}
/* ================================================================== *
* task 3.2 — Boot Policy Decision Table (DECISION ONLY) *
* *
@ -1274,6 +1480,11 @@ void test_commit_durability(void)
RUN_TEST(test_recover_verdict_empty_first_boot_clean);
RUN_TEST(test_recover_verdict_crc_break_first_entry_truncated);
RUN_TEST(test_recover_verdict_snapshot_sentinel_clean);
/* task 3.4 — re-derive the durable floor from the recovered WAL tip. */
RUN_TEST(test_durable_floor_truncating_restart_is_intact_tip);
RUN_TEST(test_durable_floor_intact_recovery_full_tip);
RUN_TEST(test_durable_floor_empty_recovery_zero);
RUN_TEST(test_durable_floor_snapshot_sentinel_tip);
/* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
* 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
* the suite stays GREEN. See design.md "Boot Policy Decision Table". */

23
src/include/persist.h
View file

@ -307,6 +307,29 @@ int persist_flush(persist_t* p);
* any thread. Returns 0 when nothing is yet durable. */
uint64_t persist_durable_lsn(persist_t* p);
/* Durable freshness floor (ha-commit-durability task 3.4, Req 2.3).
*
* The member's durably-backed last-log index (WAL tip): the highest LSN that
* is on stable storage locally. persist_recover re-derives it from the
* recovered WAL tip (the highest INTACT replayed LSN) so the post-recovery
* value is backed by durable on-disk state, NOT a volatile in-memory
* applied_lsn / last_known_committed_lsn. The Raft freshness gate (task 4.2,
* ha.c / cluster.c) consumes this so a truncated member cannot over-advertise
* a tip it does not durably hold.
*
* Precision: this is the durable LAST-LOG INDEX, NOT the commit position
* (committed_lsn). The post-election safe-commit point still comes from voter
* replies, never from this floor (CLAUDE.md). After a truncating restart the
* floor equals the lower intact tip, never a stale higher in-memory value and
* never the commit position.
*
* Thin wrapper over persist_durable_lsn: at and after recovery the durable
* watermark IS the recovered WAL tip (recovered on-disk entries are genuinely
* durable), so a separate field would be redundant. Lock-free atomic read;
* safe from any thread. Returns 0 when nothing is durable (empty first-boot
* WAL). */
uint64_t persist_durable_floor(persist_t* p);
/* Recovery-integrity verdict + flag (ha-commit-durability task 3.1, Req 2.1).
* persist_recovery_verdict returns the RECOVER_CLEAN / RECOVER_TRUNCATED
* verdict computed by the last persist_recover (RECOVER_CLEAN on a fresh