From 9344b8ed609e4b86210d453479834099248f97ce Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 1 Jul 2026 13:55:06 +0000 Subject: [PATCH] docs(raft-ha-core-safety): document term-blind bypass safe-by-guard + pin restart identity (task 6.2) The term-blind ha_apply_single_entry direct-persist branch (carried forward from task 2.4) is unreachable with a real non-zero term in any current production path: its only trigger, raft_send_append_entries, has zero production call sites (library simulator only). Both real senders (repl_broadcast, ha_stream_wal_to_standby) always emit a non-zero term for a present LSN. Documents this in ha.c and design.md rather than structurally retiring a dormant path (YAGNI), with an explicit flag for future re-evaluation if a heartbeat-driver task ever wires up raft_send_append_entries. Adds two regression pins: restart preserves (term,lsn) identity (replay_wal already reconstructs the term index via record_entry_term), and real senders never produce the term-blind wire shape. test_ha_core_safety 28/0, no regression, no ASAN reports. --- .kiro/specs/raft-ha-core-safety/design.md | 8 +- .kiro/specs/raft-ha-core-safety/tasks.md | 2 +- src/controller/ha.c | 55 +++- .../testsuite/test_ha_core_safety.c | 281 ++++++++++++++++++ 4 files changed, 338 insertions(+), 8 deletions(-) diff --git a/.kiro/specs/raft-ha-core-safety/design.md b/.kiro/specs/raft-ha-core-safety/design.md index 6a739bcd..a8fa79dc 100644 --- a/.kiro/specs/raft-ha-core-safety/design.md +++ b/.kiro/specs/raft-ha-core-safety/design.md @@ -235,11 +235,11 @@ A node is reported healthy/3-of-3 only when committed **and** applied state matc #### Unified Entry Ingest | Field | Detail | |-------|--------| -| Intent | Every replicated/catch-up entry passes `raft_on_append_entries_request` (logOk + `(term,lsn)` conflict + leader-term stamp). Defensively retire the term-blind direct-persist branch so no path can bypass the seam. | +| Intent | Every replicated/catch-up entry passes `raft_on_append_entries_request` (logOk + `(term,lsn)` conflict + leader-term stamp). The term-blind direct-persist branch is closed by reachability, not by removal — task 6.2 traced every production sender and found it unreachable with real data; documented SAFE-BY-GUARD rather than structurally retired (see task 6.2's Implementation Note below). | | Requirements | 1.2, 2.3, 3.1, 3.2, 3.4 | **Responsibilities & Constraints** -- Defensive: remove the `entry->term == 0` direct-persist branch (`ha_apply_single_entry`); route batch/single/catch-up ingest through the seam. (Not the cause of the observed divergence, but a latent bypass hazard.) +- Defensive: the `entry->term == 0` direct-persist branch (`ha_apply_single_entry`) is NOT structurally removed (task 6.2 decision). Every real sender (`repl_broadcast` for live replication, `ha_stream_wal_to_standby` for catch-up) always supplies `n_entries>=1` with a non-zero term, so this branch is unreachable with production data; the sole function that could ever drive it (`repl_send_append_entries`) is invoked only by the raft library's `raft_send_append_entries`, which has zero production callers today. It remains documented and guarded in `ha.c`, not retired. (Not the cause of the observed divergence, but a latent bypass hazard — now shown to be dormant, not live.) - Conflict repair on this (follower/standby) path uses the **follower-AE-conflict-truncate** contract (truncate the conflicting suffix per Raft Log Matching) — **not** the promotion clamp. logOk gates the truncate, so only genuinely-conflicting (never-majority-committed) entries are cut. The promotion clamp applies only to the leader-promotion path, which this component does not touch. - File I/O / truncation / restart never on a callback or iomgr thread — defer to delaymgr (existing pattern). @@ -253,8 +253,8 @@ A node is reported healthy/3-of-3 only when committed **and** applied state matc - Concurrency: ingest may run on iomgr; deferred truncate/persist follow the existing delaymgr one-shot pattern. **Implementation Notes** -- Integration: reuse `ha_apply_term_entry_via_library`/`ha_consume_append_entries`; retire `ha_apply_single_entry`'s direct-persist path. -- Validation: RED unit test injects an entry through the term-blind branch and asserts it now routes through the seam and is reconciled. +- Integration: reuse `ha_apply_term_entry_via_library`/`ha_consume_append_entries` for every reachable ingest path. +- Validation (task 6.2, closing task 2.4's deferral): rather than retiring `ha_apply_single_entry`'s direct-persist path, task 6.2 traced every production sender of a `wal_catchup_entry_t` message and found the term-blind branches (the batch-mode `sub->term==0` fallthrough and the unconditional `entry->n_entries==0` single-entry mode in `ha_apply_wal_catchup_entry`) unreachable with real data: `repl_broadcast` always sends `n_entries==1` with the real stamped term; `ha_stream_wal_to_standby` always batches `n_entries>=1` and sources each sub-entry's term via `persist_entry_term`, whose legacy-zero rule maps a stored term of 0 to `WAL_LEGACY_TERM` for any present LSN (never 0). The only function that could emit the term-blind shape, `repl_send_append_entries`, is driven solely by the raft library's `raft_send_append_entries`, which has zero production call sites (only the pure library simulator and an `XCE_TESTING` test helper call it). Decision: **document safe-by-guard, do not structurally retire** — retiring would mean building a term-0-safe seam variant for a path with no live trigger, which is speculative work the evidence does not warrant. The disposition and reachability trace are recorded as comments at `ha_apply_single_entry` and both term-blind branches in `ha.c`, and pinned by `test_2_3_termblind_route_payload_guarded`, `test_2_3_unrepairable_catchup_raises_nonpromotable_fence`, and `test_6_2_real_senders_never_produce_termblind_shape` (`test_ha_core_safety.c`). - Risks: the safety boundary on this path is **logOk**, not a durable-tip clamp — only entries that fail the `(term, lsn)` match are truncated, and a failed match proves the entry was never majority-committed (Leader Completeness). A unit test must assert that an entry which *matches* the leader is never truncated, and that a *conflicting* entry below local `durable_tip` *is* truncated. #### Standby Reconvergence diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 7b636b54..acdfcb3c 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -188,7 +188,7 @@ - _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_ - _Depends: 2.4_ -- [ ] 6.2 Verify the single-seam invariant and restart identity +- [x] 6.2 Verify the single-seam invariant and restart identity - Assert no ingest path (live replication, catch-up batch, snapshot post-install, rejoin) writes the WAL outside the reconcile seam, and that each entry's `(term, lsn)` identity survives a restart/recovery. **Carry-forward from 2.4:** the term-blind `ha_apply_single_entry` direct-persist branch was NOT structurally retired in 2.4 (its empty-over-full safety is enforced by the 2.2 payload guard and pinned by `test_2_3_termblind_route_payload_guarded`/`test_2_3_unrepairable_catchup_raises_nonpromotable_fence`). This task must either structurally retire that bypass (route term-0 catch-up through the seam, with its own RED first) OR formally document it as safe-by-persist-guard and adjust the invariant wording accordingly - Observable completion: an integration test confirms every ingest path reconciles through the seam and that entry terms are preserved across restart; the term-blind bypass is either removed (no direct-persist path remains) or documented safe-by-guard with the pin tests cited - _Requirements: 1.1, 1.2, 2.4, 3.1_ diff --git a/src/controller/ha.c b/src/controller/ha.c index 9675ae07..cb62ac76 100644 --- a/src/controller/ha.c +++ b/src/controller/ha.c @@ -2130,7 +2130,32 @@ static void ha_advance_applied_lsn(ha_state_t *ha, uint64_t new_lsn) * Payload bytes (raw_payload/raw_payload_len, extracted by xce_decode) * are passed through to apply_wal_entry so entity state is restored — * not just the LSN chain (task 7.3). - * Returns 0 on success, non-zero on error (caller logs and skips). */ + * Returns 0 on success, non-zero on error (caller logs and skips). + * + * raft-ha-core-safety task 6.2 — TERM-BLIND DIRECT-PERSIST BYPASS, SAFE-BY- + * GUARD (documented, not structurally retired): this function does NOT go + * through raft_on_append_entries_request's logOk/(term,lsn) reconcile — it + * calls persist_wal_write_replicated directly. Task 2.4 deferred deciding + * this bypass's fate to 6.2; 6.2's reachability trace (see the two call + * sites below and ha_apply_wal_catchup_entry) found it is UNREACHABLE with + * real production data today: every real sender (repl_broadcast for live + * replication; ha_stream_wal_to_standby for catch-up) always supplies + * n_entries>=1 with a non-zero term (persist_entry_term's legacy-zero rule + * maps a stored term of 0 to WAL_LEGACY_TERM for any present LSN — it never + * returns 0 for a real entry). The ONLY function that could ever drive a + * term==0 / n_entries==0 message into this path (repl_send_append_entries) + * is called solely from the raft library's raft_send_append_entries, which + * has zero production callers (only lib/raft/raft_sim.c and an XCE_TESTING + * helper call it) — so that trigger is dormant, not live, code. Even if this + * path were reached, the task-2.2 (term,lsn,payload-crc)-aware guard inside + * persist_wal_write_replicated already refuses to degrade a stored non-empty + * entry to empty and raises the sticky non-promotable fence on an + * unrepairable below-tip conflict — pinned by + * test_2_3_termblind_route_payload_guarded and + * test_2_3_unrepairable_catchup_raises_nonpromotable_fence + * (test_ha_core_safety.c). See ha_apply_wal_catchup_entry below and + * test_6_2_real_senders_never_produce_termblind_shape for the reachability + * evidence this comment summarizes. */ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e) { /* Persist-before-apply-before-ACK: durably write the replicated entry to @@ -2316,7 +2341,18 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p, * retires the term-0 direct-apply path for live/catch-up replication. * Entries stream in ascending LSN order, so the library's contiguity * (prev tuple) is satisfied in sequence; a not-yet-applicable entry - * (logOk reject) is left for the catch-up continuation. */ + * (logOk reject) is left for the catch-up continuation. + * + * raft-ha-core-safety task 6.2: the `sub->term != 0` fallthrough + * below is the term-blind direct-persist bypass (ha_apply_single_ + * entry) — documented SAFE-BY-GUARD, not structurally retired. + * ha_stream_wal_to_standby (the only real batch-catch-up sender) + * always stamps sub->term via persist_entry_term, which maps a + * legacy on-disk term of 0 to WAL_LEGACY_TERM for any present LSN, + * so a real streamed sub-entry is never term==0 — this fallthrough + * is unreachable with production data (see the ha_apply_single_ + * entry comment for the full reachability trace + the pin test + * test_6_2_real_senders_never_produce_termblind_shape). */ if (c && c->ha.raft_c == c && sub->term != 0) { if (ha_apply_term_entry_via_library(ha, p, c, ep, sub)) { last_good_lsn = sub->lsn; @@ -2356,7 +2392,20 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p, return 0; } - /* ── Single-entry mode (task 2.1 backward-compatible path) ────────────── */ + /* ── Single-entry mode (task 2.1 backward-compatible path) ────────────── + * raft-ha-core-safety task 6.2: this branch calls ha_apply_single_entry + * UNCONDITIONALLY — no term check at all — so it is the other half of the + * term-blind bypass documented at ha_apply_single_entry above. It is only + * ever reached with entry->n_entries==0, the 0-entry AppendEntries + * heartbeat shape built by repl_send_append_entries. That function is + * called solely from the raft library's raft_send_append_entries, which + * has ZERO production call sites (only lib/raft/raft_sim.c and an + * XCE_TESTING test helper call it) — so in the CURRENT codebase this + * branch is dormant/unreachable from any live sender, not a legacy- + * rollout shape carrying real data. Documented SAFE-BY-GUARD (not + * structurally retired): see the ha_apply_single_entry comment for the + * full reachability trace and the task-2.2 guard that would still bound + * the damage if this ever became reachable. */ int rc = ha_apply_single_entry(p, entry); if (rc != 0) { /* -EAGAIN = contiguity gap (bug-850): dropped quietly, backfilled by diff --git a/src/controller/testsuite/test_ha_core_safety.c b/src/controller/testsuite/test_ha_core_safety.c index bd829916..35392c18 100644 --- a/src/controller/testsuite/test_ha_core_safety.c +++ b/src/controller/testsuite/test_ha_core_safety.c @@ -55,6 +55,7 @@ #include "config_repl.h" /* ha_stream_wal_to_standby */ #include "ha.h" #include "persist.h" +#include "repl.h" /* repl_broadcast / repl_add_standby (task 6.2) */ #include "xce/cluster_def.h" /* cluster_t (follower AE-intake fixture, task 4.1) */ #include "request.h" /* xce_request_t */ #include "typemap.h" /* wal_catchup_entry_t */ @@ -62,6 +63,11 @@ #include "walcompare.h" #include "waldump.h" +/* test-only send-capture hook defined in repl.c (XCE_TESTING); mirrors the + * extern pattern test_repl_unit.c uses for the same hook (task 6.2). */ +extern int (*repl_broadcast_send_hook)(void *ep, wal_catchup_entry_t **entries, + int n_entries); + /* ── Fixtures ────────────────────────────────────────────────────────────── * A member is a real persist_t WAL on its own temp dir. This is the exact * receiver-side persistence surface the spec hardens, so a divergence @@ -113,6 +119,38 @@ static void hcs_member_down(hcs_member_t* m) m->p = NULL; } +/* Re-open an already-torn-down member's workdir as a FRESH persist_t plus an + * explicit persist_recover — simulates a controller restart/reboot over the + * SAME on-disk WAL (task 6.2, Req 2.4). Mirrors reopen_test_persist in + * test_repl_apply.c. m->p must be NULL (torn down via hcs_member_down) + * before calling; m->root/m->wal_dir are reused unchanged. */ +static int hcs_member_reopen(hcs_member_t* m) +{ + if (m->p) + return -EINVAL; + + m->p = calloc(1, sizeof(persist_t)); + if (!m->p) + return -ENOMEM; + + persist_config_t cfg = persist_config_defaults(); + cfg.wal_sync_ms = 5000; + cfg.sync_timeout_ms = 50; + cfg.max_wal_records = 1u << 30; + + persist_set_workdir(m->p, m->root, "ctrl"); + persist_init(m->p, &cfg, NULL); + int rc = persist_recover(m->p); + if (rc < 0) { + persist_fini(m->p); + free(m->p); + m->p = NULL; + return rc; + } + persist_start(m->p); + return 0; +} + /* ── Seam 1: payload-controllable WAL entry construction ──────────────────── * Write an entry with a caller-chosen (term, lsn, type, payload, len) through * the real replicated-write guard (persist.c:802). term is stamped via @@ -2374,6 +2412,239 @@ static void test_6_1_replication_and_catchup_preserve_term_payload_e2e(void) printf("test_6_1_replication_and_catchup_preserve_term_payload_e2e: done\n"); } +/* ═══════════════════════════════════════════════════════════════════════════ + * TASK 6.2 — single-seam invariant + restart identity + * + * Two obligations remain after 2.4/6.1: + * + * (a) RESTART IDENTITY (Req 2.4): an entry's (term, lsn) identity must + * survive a restart/recovery. persist.c's replay_wal already calls + * record_entry_term for every replayed entry (~persist.c:2731), so this + * is a regression PIN (test_6_2_restart_preserves_term_identity), not a + * RED — it also pins the legacy-zero (WAL_LEGACY_TERM) mapping surviving + * identically across the restart, not just a real assigned term. + * + * (b) SINGLE-SEAM INVARIANT / term-blind bypass disposition (carried + * forward from task 2.4's Implementation Note): ha_apply_single_entry + * (ha.c:2134) is a direct-persist route that skips + * raft_on_append_entries_request's logOk/(term,lsn) reconcile. + * ha_apply_wal_catchup_entry (ha.c:2270) still has TWO term-blind paths + * into it — the batch-mode fallthrough when a sub-entry's term==0 + * (ha.c ~2328), and the legacy n_entries==0 single-entry mode + * (ha.c ~2360), which is UNCONDITIONAL (no term check at all). + * + * REACHABILITY FINDING (this task, traced every production sender of a + * wal_catchup_entry_t message end to end): + * 1. repl_broadcast (repl.c:239, live WAL-append broadcast) ALWAYS + * sends n_entries==1 with entry.term = hdr->term — the term + * actually stamped on the just-appended entry. Routes through + * ha_apply_wal_catchup_entry's branch 1 (seam). + * 2. ha_stream_wal_to_standby (config_repl.c:2113, the catch-up + * streamer) ALWAYS builds a batch with n_entries >= 1 (checked + * before any batch is built — standby_lsn >= committed_lsn returns + * early) and stamps every sub-entry's term via + * persist_entry_term(p, hdr.lsn). entry_term_locked's legacy-zero + * rule (persist.c:450-459) maps a raw on-disk term of 0 to + * WAL_LEGACY_TERM (1) for any PRESENT lsn — persist_entry_term + * returns 0 ONLY for an lsn that was never recorded at all. A real + * streamed sub-entry's term is therefore NEVER 0. Routes through + * branch 2's seam call (ha_apply_term_entry_via_library). + * 3. repl_send_append_entries (repl.c:340) is the ONLY production + * function that can ever emit n_entries==0 (a 0-entry AE heartbeat) + * or a term==0 single entry. It is called ONLY from + * ha_cb_send_append_entries_stub (the wired on_send_append_entries + * callback, ha.c:1693/1815 s_real_cbs), which is called ONLY from + * the raft library's raft_send_append_entries (raft.c:595) or the + * XCE_TESTING-only helper ha_cb_send_append_entries_for_test. + * raft_send_append_entries has ZERO production call sites — every + * caller is in lib/raft/raft_sim.c (the pure library simulation + * harness). So the callback is registered but never DRIVEN by + * anything in the running controller: this is the only path that + * could ever reach ha_apply_wal_catchup_entry's n_entries==0 branch + * with a real message, and it is dormant/unreachable code today. + * + * CORROBORATION: the existing pin tests + * (test_2_3_termblind_route_payload_guarded, + * test_2_3_unrepairable_catchup_raises_nonpromotable_fence, above) can + * only exercise the term-blind branches by hand-building a + * wal_catchup_entry_t with term==0 and calling ha_apply_wal_catchup_entry + * directly with c==NULL — bypassing every real sender, because no real + * sender in the current codebase produces that shape. + * + * DECISION: DOCUMENT SAFE-BY-GUARD, do not structurally retire. The + * term-blind branches are (i) unreachable with real production data + * today (finding above), and (ii) even if reached, the task-2.2 + * (term,lsn,payload-crc)-aware guard in persist_wal_write_replicated + * already prevents an empty-over-real degrade and raises the sticky + * non-promotable fence on an unrepairable below-tip conflict — proven by + * the two pin tests cited above. Structurally retiring would mean + * inventing a term-0-safe seam variant (the exact coupling task 2.2's + * own design note deferred) for a path with no live trigger — spending + * effort/risk against YAGNI with no defect it would close. The + * corresponding comments at ha_apply_single_entry / the two term-blind + * branches in ha_apply_wal_catchup_entry (ha.c) record this disposition + * in the production source. test_6_2_real_senders_never_produce_ + * termblind_shape below PINS the reachability finding itself: it drives + * the two REAL senders with real WAL data (including a legacy + * term-0-stamped entry) and asserts every wire entry they emit satisfies + * n_entries>=1 && term!=0, so a future change that regresses a real + * sender into the term-blind shape is caught here. + * + * Requirements: 1.1, 1.2, 2.4, 3.1. + * ═══════════════════════════════════════════════════════════════════════════ */ + +/* (a) RESTART IDENTITY — an entry's (term, lsn) identity, INCLUDING the + * legacy-zero -> WAL_LEGACY_TERM mapping, must read back identically after a + * simulated controller restart (clean teardown + a FRESH persist_t replaying + * the same on-disk WAL via persist_recover). */ +static void test_6_2_restart_preserves_term_identity(void) +{ + hcs_member_t m; + TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&m, "62restart")); + + uint8_t full[HCS_FULL_LEN]; + hcs_fill_body(full); + + /* Entries spanning several distinct terms, including one stamped with the + * legacy term 0 at lsn 4 — Req 2.4 ("preserve each entry's original term + * identity") must cover the legacy-zero mapping too, not just a real + * assigned term. */ + enum { N = 5 }; + uint64_t terms[N + 1] = { 0, 3, 3, 7, 0 /* legacy */, 9 }; + for (uint64_t lsn = 1; lsn <= N; lsn++) { + TEST_ASSERT_EQUAL_INT_MESSAGE(0, + hcs_wal_put(&m, lsn, terms[lsn], WAL_JOB_SUBMIT, full, HCS_FULL_LEN), + "restart-identity build write must persist (real fs required — " + "bug-924)"); + } + atomic_store_explicit(&m.p->committed_lsn, (uint64_t)N, memory_order_release); + + /* Capture the BEFORE-restart term identity via the authoritative readback + * API (applies the legacy-0 -> WAL_LEGACY_TERM mapping uniformly). */ + uint64_t before[N + 1]; + for (uint64_t lsn = 1; lsn <= N; lsn++) + before[lsn] = persist_entry_term(m.p, lsn); + TEST_ASSERT_EQUAL_UINT64_MESSAGE((uint64_t)WAL_LEGACY_TERM, before[4], + "precondition: a term-0-stamped entry reads back as WAL_LEGACY_TERM, " + "not 0, even before the restart"); + + /* Simulate a restart: clean shutdown, then a FRESH persist_t replays the + * same on-disk WAL via persist_recover (record_entry_term rebuilds the + * per-LSN term index — persist.c ~2731). */ + hcs_member_down(&m); + TEST_ASSERT_EQUAL_INT(0, hcs_member_reopen(&m)); + + for (uint64_t lsn = 1; lsn <= N; lsn++) { + uint64_t after = persist_entry_term(m.p, lsn); + TEST_ASSERT_EQUAL_UINT64_MESSAGE(before[lsn], after, + "Req 2.4: an entry's (term, lsn) identity must survive a " + "restart/recovery — replay_wal's record_entry_term must " + "reconstruct the SAME term this LSN held before the restart"); + } + + /* Sanity: the restart replayed the WAL bytes too, not just the term + * index. */ + for (uint64_t lsn = 1; lsn <= N; lsn++) { + hcs_probe_t pr; + TEST_ASSERT_EQUAL_INT(0, hcs_wal_probe(&m, lsn, &pr)); + TEST_ASSERT_TRUE_MESSAGE(pr.present, + "every pre-restart entry must still be present after recovery"); + TEST_ASSERT_EQUAL_UINT32(HCS_FULL_LEN, pr.len); + } + + hcs_member_down(&m); + printf("test_6_2_restart_preserves_term_identity: done\n"); +} + +/* Capture buffer for the repl_broadcast send hook (live replication sender). */ +#define HCS62_BCAST_MAX 4 +static wal_catchup_entry_t g_hcs62_bcast_entries[HCS62_BCAST_MAX]; +static int g_hcs62_bcast_n; + +static int hcs_6_2_broadcast_hook(void* ep, wal_catchup_entry_t** entries, + int n_entries) +{ + (void)ep; + g_hcs62_bcast_n = n_entries; + for (int i = 0; i < n_entries && i < HCS62_BCAST_MAX; i++) + g_hcs62_bcast_entries[i] = *entries[i]; + return 0; +} + +/* (b) REACHABILITY PIN — the two real production senders (repl_broadcast for + * live replication, ha_stream_wal_to_standby for catch-up) can never produce + * the term-blind shape (n_entries==0, or a term==0 sub-entry) that would + * route through ha_apply_wal_catchup_entry's direct-persist bypass. This + * operationalizes the reachability finding in the block comment above: if a + * future change regressed a real sender into emitting that shape, this test + * would catch it. */ +static void test_6_2_real_senders_never_produce_termblind_shape(void) +{ + /* ── Sender 1: repl_broadcast (live WAL-append replication) ────────────── */ + replicator_t r; + TEST_ASSERT_EQUAL_INT(0, repl_init(&r)); + int dummy_ep = 0; + TEST_ASSERT_TRUE(repl_add_standby(&r, (void*)&dummy_ep, "standby", 7001) >= 0); + + uint8_t full[HCS_FULL_LEN]; + hcs_fill_body(full); + + wal_entry_hdr_t hdr; + memset(&hdr, 0, sizeof(hdr)); + hdr.lsn = 5; + hdr.type = WAL_JOB_SUBMIT; + hdr.term = 7; /* the real term stamped on this append by the leader */ + + g_hcs62_bcast_n = 0; + repl_broadcast_send_hook = hcs_6_2_broadcast_hook; + int brc = repl_broadcast(&r, &hdr, full, HCS_FULL_LEN, /*prev_log_term=*/6); + repl_broadcast_send_hook = NULL; + repl_fini(&r); + + TEST_ASSERT_EQUAL_INT(0, brc); + TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_hcs62_bcast_n, + "PIN: repl_broadcast always sends exactly ONE entry (n_entries==1) — " + "it can never produce the n_entries==0 heartbeat shape"); + TEST_ASSERT_NOT_EQUAL_MESSAGE(0u, g_hcs62_bcast_entries[0].term, + "PIN: repl_broadcast stamps the entry's REAL term (hdr->term) — it " + "can never emit the term==0 term-blind shape"); + + /* ── Sender 2: ha_stream_wal_to_standby (catch-up streamer) ────────────── */ + hcs_member_t src, rcv; + TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&src, "62src")); + TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&rcv, "62rcv")); + + /* lsn 1 is stamped with the LEGACY term 0 — the shape most likely to slip + * a term==0 sub-entry into a real catch-up batch, if it were possible. */ + TEST_ASSERT_EQUAL_INT(0, hcs_wal_put(&src, 1, /*term=*/0 /* legacy */, + WAL_JOB_SUBMIT, full, HCS_FULL_LEN)); + TEST_ASSERT_EQUAL_INT(0, hcs_wal_put(&src, 2, /*term=*/4, WAL_JOB_SUBMIT, + full, HCS_FULL_LEN)); + atomic_store_explicit(&src.p->committed_lsn, 2, memory_order_release); + + hcs_2_3_reset(&rcv, /*fault_lsn=*/0 /* no fault injected below */); + xce_wal_stream_request_hook = hcs_2_3_capture_hook; + int src_rc = ha_stream_wal_to_standby((endpoint_t*)NULL, src.wal_dir, src.p, + /*leader_term=*/9, /*standby_lsn=*/0, + /*committed_lsn=*/2); + xce_wal_stream_request_hook = NULL; + TEST_ASSERT_EQUAL_INT(0, src_rc); + + TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_sends_n, + "PIN: the streamer batched BOTH entries (n_entries>=1 in the wire " + "container) — catch-up never emits the n_entries==0 shape"); + for (int i = 0; i < g_sends_n; i++) + TEST_ASSERT_NOT_EQUAL_MESSAGE(0u, g_sends[i].term, + "PIN: every streamed sub-entry's term is non-zero — even an " + "on-disk legacy term==0 entry reads back as WAL_LEGACY_TERM via " + "persist_entry_term, so the streamer can never emit a term==0 " + "sub-entry into a real catch-up batch"); + + hcs_member_down(&src); + hcs_member_down(&rcv); + printf("test_6_2_real_senders_never_produce_termblind_shape: done\n"); +} + /* ── Suite entry point ─────────────────────────────────────────────────────*/ void test_ha_core_safety(void) @@ -2459,5 +2730,15 @@ void test_ha_core_safety(void) * instances: no term==0 on the wire from a live leader, no non-empty * entry ever lands as PLEN=0, and walcompare reports a clean prefix. */ RUN_TEST(test_6_1_replication_and_catchup_preserve_term_payload_e2e); + /* Task 6.2: single-seam invariant + restart identity. + * PIN — an entry's (term, lsn) identity, including the legacy-zero -> + * WAL_LEGACY_TERM mapping, survives a simulated restart/recovery. + * PIN — the two real production senders (repl_broadcast, + * ha_stream_wal_to_standby) can never produce the term-blind + * shape (n_entries==0 or a term==0 sub-entry); see the reachability + * finding + safe-by-guard decision recorded in the block comment + * above and in ha.c. */ + RUN_TEST(test_6_2_restart_preserves_term_identity); + RUN_TEST(test_6_2_real_senders_never_produce_termblind_shape); UNITY_END(); }