docs(raft-ha-core-safety): document term-blind bypass safe-by-guard + pin restart identity (task 6.2)
The term-blind ha_apply_single_entry direct-persist branch (carried forward from task 2.4) is unreachable with a real non-zero term in any current production path: its only trigger, raft_send_append_entries, has zero production call sites (library simulator only). Both real senders (repl_broadcast, ha_stream_wal_to_standby) always emit a non-zero term for a present LSN. Documents this in ha.c and design.md rather than structurally retiring a dormant path (YAGNI), with an explicit flag for future re-evaluation if a heartbeat-driver task ever wires up raft_send_append_entries. Adds two regression pins: restart preserves (term,lsn) identity (replay_wal already reconstructs the term index via record_entry_term), and real senders never produce the term-blind wire shape. test_ha_core_safety 28/0, no regression, no ASAN reports.
This commit is contained in:
parent
4e1e3e07fb
commit
9344b8ed60
4 changed files with 338 additions and 8 deletions
|
|
@ -235,11 +235,11 @@ A node is reported healthy/3-of-3 only when committed **and** applied state matc
|
|||
#### Unified Entry Ingest
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Every replicated/catch-up entry passes `raft_on_append_entries_request` (logOk + `(term,lsn)` conflict + leader-term stamp). Defensively retire the term-blind direct-persist branch so no path can bypass the seam. |
|
||||
| Intent | Every replicated/catch-up entry passes `raft_on_append_entries_request` (logOk + `(term,lsn)` conflict + leader-term stamp). The term-blind direct-persist branch is closed by reachability, not by removal — task 6.2 traced every production sender and found it unreachable with real data; documented SAFE-BY-GUARD rather than structurally retired (see task 6.2's Implementation Note below). |
|
||||
| Requirements | 1.2, 2.3, 3.1, 3.2, 3.4 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Defensive: remove the `entry->term == 0` direct-persist branch (`ha_apply_single_entry`); route batch/single/catch-up ingest through the seam. (Not the cause of the observed divergence, but a latent bypass hazard.)
|
||||
- Defensive: the `entry->term == 0` direct-persist branch (`ha_apply_single_entry`) is NOT structurally removed (task 6.2 decision). Every real sender (`repl_broadcast` for live replication, `ha_stream_wal_to_standby` for catch-up) always supplies `n_entries>=1` with a non-zero term, so this branch is unreachable with production data; the sole function that could ever drive it (`repl_send_append_entries`) is invoked only by the raft library's `raft_send_append_entries`, which has zero production callers today. It remains documented and guarded in `ha.c`, not retired. (Not the cause of the observed divergence, but a latent bypass hazard — now shown to be dormant, not live.)
|
||||
- Conflict repair on this (follower/standby) path uses the **follower-AE-conflict-truncate** contract (truncate the conflicting suffix per Raft Log Matching) — **not** the promotion clamp. logOk gates the truncate, so only genuinely-conflicting (never-majority-committed) entries are cut. The promotion clamp applies only to the leader-promotion path, which this component does not touch.
|
||||
- File I/O / truncation / restart never on a callback or iomgr thread — defer to delaymgr (existing pattern).
|
||||
|
||||
|
|
@ -253,8 +253,8 @@ A node is reported healthy/3-of-3 only when committed **and** applied state matc
|
|||
- Concurrency: ingest may run on iomgr; deferred truncate/persist follow the existing delaymgr one-shot pattern.
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: reuse `ha_apply_term_entry_via_library`/`ha_consume_append_entries`; retire `ha_apply_single_entry`'s direct-persist path.
|
||||
- Validation: RED unit test injects an entry through the term-blind branch and asserts it now routes through the seam and is reconciled.
|
||||
- Integration: reuse `ha_apply_term_entry_via_library`/`ha_consume_append_entries` for every reachable ingest path.
|
||||
- Validation (task 6.2, closing task 2.4's deferral): rather than retiring `ha_apply_single_entry`'s direct-persist path, task 6.2 traced every production sender of a `wal_catchup_entry_t` message and found the term-blind branches (the batch-mode `sub->term==0` fallthrough and the unconditional `entry->n_entries==0` single-entry mode in `ha_apply_wal_catchup_entry`) unreachable with real data: `repl_broadcast` always sends `n_entries==1` with the real stamped term; `ha_stream_wal_to_standby` always batches `n_entries>=1` and sources each sub-entry's term via `persist_entry_term`, whose legacy-zero rule maps a stored term of 0 to `WAL_LEGACY_TERM` for any present LSN (never 0). The only function that could emit the term-blind shape, `repl_send_append_entries`, is driven solely by the raft library's `raft_send_append_entries`, which has zero production call sites (only the pure library simulator and an `XCE_TESTING` test helper call it). Decision: **document safe-by-guard, do not structurally retire** — retiring would mean building a term-0-safe seam variant for a path with no live trigger, which is speculative work the evidence does not warrant. The disposition and reachability trace are recorded as comments at `ha_apply_single_entry` and both term-blind branches in `ha.c`, and pinned by `test_2_3_termblind_route_payload_guarded`, `test_2_3_unrepairable_catchup_raises_nonpromotable_fence`, and `test_6_2_real_senders_never_produce_termblind_shape` (`test_ha_core_safety.c`).
|
||||
- Risks: the safety boundary on this path is **logOk**, not a durable-tip clamp — only entries that fail the `(term, lsn)` match are truncated, and a failed match proves the entry was never majority-committed (Leader Completeness). A unit test must assert that an entry which *matches* the leader is never truncated, and that a *conflicting* entry below local `durable_tip` *is* truncated.
|
||||
|
||||
#### Standby Reconvergence
|
||||
|
|
|
|||
|
|
@ -188,7 +188,7 @@
|
|||
- _Boundary: Catch-up Payload Integrity, Unified Entry Ingest_
|
||||
- _Depends: 2.4_
|
||||
|
||||
- [ ] 6.2 Verify the single-seam invariant and restart identity
|
||||
- [x] 6.2 Verify the single-seam invariant and restart identity
|
||||
- Assert no ingest path (live replication, catch-up batch, snapshot post-install, rejoin) writes the WAL outside the reconcile seam, and that each entry's `(term, lsn)` identity survives a restart/recovery. **Carry-forward from 2.4:** the term-blind `ha_apply_single_entry` direct-persist branch was NOT structurally retired in 2.4 (its empty-over-full safety is enforced by the 2.2 payload guard and pinned by `test_2_3_termblind_route_payload_guarded`/`test_2_3_unrepairable_catchup_raises_nonpromotable_fence`). This task must either structurally retire that bypass (route term-0 catch-up through the seam, with its own RED first) OR formally document it as safe-by-persist-guard and adjust the invariant wording accordingly
|
||||
- Observable completion: an integration test confirms every ingest path reconciles through the seam and that entry terms are preserved across restart; the term-blind bypass is either removed (no direct-persist path remains) or documented safe-by-guard with the pin tests cited
|
||||
- _Requirements: 1.1, 1.2, 2.4, 3.1_
|
||||
|
|
|
|||
|
|
@ -2130,7 +2130,32 @@ static void ha_advance_applied_lsn(ha_state_t *ha, uint64_t new_lsn)
|
|||
* Payload bytes (raw_payload/raw_payload_len, extracted by xce_decode)
|
||||
* are passed through to apply_wal_entry so entity state is restored —
|
||||
* not just the LSN chain (task 7.3).
|
||||
* Returns 0 on success, non-zero on error (caller logs and skips). */
|
||||
* Returns 0 on success, non-zero on error (caller logs and skips).
|
||||
*
|
||||
* raft-ha-core-safety task 6.2 — TERM-BLIND DIRECT-PERSIST BYPASS, SAFE-BY-
|
||||
* GUARD (documented, not structurally retired): this function does NOT go
|
||||
* through raft_on_append_entries_request's logOk/(term,lsn) reconcile — it
|
||||
* calls persist_wal_write_replicated directly. Task 2.4 deferred deciding
|
||||
* this bypass's fate to 6.2; 6.2's reachability trace (see the two call
|
||||
* sites below and ha_apply_wal_catchup_entry) found it is UNREACHABLE with
|
||||
* real production data today: every real sender (repl_broadcast for live
|
||||
* replication; ha_stream_wal_to_standby for catch-up) always supplies
|
||||
* n_entries>=1 with a non-zero term (persist_entry_term's legacy-zero rule
|
||||
* maps a stored term of 0 to WAL_LEGACY_TERM for any present LSN — it never
|
||||
* returns 0 for a real entry). The ONLY function that could ever drive a
|
||||
* term==0 / n_entries==0 message into this path (repl_send_append_entries)
|
||||
* is called solely from the raft library's raft_send_append_entries, which
|
||||
* has zero production callers (only lib/raft/raft_sim.c and an XCE_TESTING
|
||||
* helper call it) — so that trigger is dormant, not live, code. Even if this
|
||||
* path were reached, the task-2.2 (term,lsn,payload-crc)-aware guard inside
|
||||
* persist_wal_write_replicated already refuses to degrade a stored non-empty
|
||||
* entry to empty and raises the sticky non-promotable fence on an
|
||||
* unrepairable below-tip conflict — pinned by
|
||||
* test_2_3_termblind_route_payload_guarded and
|
||||
* test_2_3_unrepairable_catchup_raises_nonpromotable_fence
|
||||
* (test_ha_core_safety.c). See ha_apply_wal_catchup_entry below and
|
||||
* test_6_2_real_senders_never_produce_termblind_shape for the reachability
|
||||
* evidence this comment summarizes. */
|
||||
static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
|
||||
{
|
||||
/* Persist-before-apply-before-ACK: durably write the replicated entry to
|
||||
|
|
@ -2316,7 +2341,18 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
* retires the term-0 direct-apply path for live/catch-up replication.
|
||||
* Entries stream in ascending LSN order, so the library's contiguity
|
||||
* (prev tuple) is satisfied in sequence; a not-yet-applicable entry
|
||||
* (logOk reject) is left for the catch-up continuation. */
|
||||
* (logOk reject) is left for the catch-up continuation.
|
||||
*
|
||||
* raft-ha-core-safety task 6.2: the `sub->term != 0` fallthrough
|
||||
* below is the term-blind direct-persist bypass (ha_apply_single_
|
||||
* entry) — documented SAFE-BY-GUARD, not structurally retired.
|
||||
* ha_stream_wal_to_standby (the only real batch-catch-up sender)
|
||||
* always stamps sub->term via persist_entry_term, which maps a
|
||||
* legacy on-disk term of 0 to WAL_LEGACY_TERM for any present LSN,
|
||||
* so a real streamed sub-entry is never term==0 — this fallthrough
|
||||
* is unreachable with production data (see the ha_apply_single_
|
||||
* entry comment for the full reachability trace + the pin test
|
||||
* test_6_2_real_senders_never_produce_termblind_shape). */
|
||||
if (c && c->ha.raft_c == c && sub->term != 0) {
|
||||
if (ha_apply_term_entry_via_library(ha, p, c, ep, sub)) {
|
||||
last_good_lsn = sub->lsn;
|
||||
|
|
@ -2356,7 +2392,20 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* ── Single-entry mode (task 2.1 backward-compatible path) ────────────── */
|
||||
/* ── Single-entry mode (task 2.1 backward-compatible path) ──────────────
|
||||
* raft-ha-core-safety task 6.2: this branch calls ha_apply_single_entry
|
||||
* UNCONDITIONALLY — no term check at all — so it is the other half of the
|
||||
* term-blind bypass documented at ha_apply_single_entry above. It is only
|
||||
* ever reached with entry->n_entries==0, the 0-entry AppendEntries
|
||||
* heartbeat shape built by repl_send_append_entries. That function is
|
||||
* called solely from the raft library's raft_send_append_entries, which
|
||||
* has ZERO production call sites (only lib/raft/raft_sim.c and an
|
||||
* XCE_TESTING test helper call it) — so in the CURRENT codebase this
|
||||
* branch is dormant/unreachable from any live sender, not a legacy-
|
||||
* rollout shape carrying real data. Documented SAFE-BY-GUARD (not
|
||||
* structurally retired): see the ha_apply_single_entry comment for the
|
||||
* full reachability trace and the task-2.2 guard that would still bound
|
||||
* the damage if this ever became reachable. */
|
||||
int rc = ha_apply_single_entry(p, entry);
|
||||
if (rc != 0) {
|
||||
/* -EAGAIN = contiguity gap (bug-850): dropped quietly, backfilled by
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@
|
|||
#include "config_repl.h" /* ha_stream_wal_to_standby */
|
||||
#include "ha.h"
|
||||
#include "persist.h"
|
||||
#include "repl.h" /* repl_broadcast / repl_add_standby (task 6.2) */
|
||||
#include "xce/cluster_def.h" /* cluster_t (follower AE-intake fixture, task 4.1) */
|
||||
#include "request.h" /* xce_request_t */
|
||||
#include "typemap.h" /* wal_catchup_entry_t */
|
||||
|
|
@ -62,6 +63,11 @@
|
|||
#include "walcompare.h"
|
||||
#include "waldump.h"
|
||||
|
||||
/* test-only send-capture hook defined in repl.c (XCE_TESTING); mirrors the
|
||||
* extern pattern test_repl_unit.c uses for the same hook (task 6.2). */
|
||||
extern int (*repl_broadcast_send_hook)(void *ep, wal_catchup_entry_t **entries,
|
||||
int n_entries);
|
||||
|
||||
/* ── Fixtures ──────────────────────────────────────────────────────────────
|
||||
* A member is a real persist_t WAL on its own temp dir. This is the exact
|
||||
* receiver-side persistence surface the spec hardens, so a divergence
|
||||
|
|
@ -113,6 +119,38 @@ static void hcs_member_down(hcs_member_t* m)
|
|||
m->p = NULL;
|
||||
}
|
||||
|
||||
/* Re-open an already-torn-down member's workdir as a FRESH persist_t plus an
|
||||
* explicit persist_recover — simulates a controller restart/reboot over the
|
||||
* SAME on-disk WAL (task 6.2, Req 2.4). Mirrors reopen_test_persist in
|
||||
* test_repl_apply.c. m->p must be NULL (torn down via hcs_member_down)
|
||||
* before calling; m->root/m->wal_dir are reused unchanged. */
|
||||
static int hcs_member_reopen(hcs_member_t* m)
|
||||
{
|
||||
if (m->p)
|
||||
return -EINVAL;
|
||||
|
||||
m->p = calloc(1, sizeof(persist_t));
|
||||
if (!m->p)
|
||||
return -ENOMEM;
|
||||
|
||||
persist_config_t cfg = persist_config_defaults();
|
||||
cfg.wal_sync_ms = 5000;
|
||||
cfg.sync_timeout_ms = 50;
|
||||
cfg.max_wal_records = 1u << 30;
|
||||
|
||||
persist_set_workdir(m->p, m->root, "ctrl");
|
||||
persist_init(m->p, &cfg, NULL);
|
||||
int rc = persist_recover(m->p);
|
||||
if (rc < 0) {
|
||||
persist_fini(m->p);
|
||||
free(m->p);
|
||||
m->p = NULL;
|
||||
return rc;
|
||||
}
|
||||
persist_start(m->p);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ── Seam 1: payload-controllable WAL entry construction ────────────────────
|
||||
* Write an entry with a caller-chosen (term, lsn, type, payload, len) through
|
||||
* the real replicated-write guard (persist.c:802). term is stamped via
|
||||
|
|
@ -2374,6 +2412,239 @@ static void test_6_1_replication_and_catchup_preserve_term_payload_e2e(void)
|
|||
printf("test_6_1_replication_and_catchup_preserve_term_payload_e2e: done\n");
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════════════════
|
||||
* TASK 6.2 — single-seam invariant + restart identity
|
||||
*
|
||||
* Two obligations remain after 2.4/6.1:
|
||||
*
|
||||
* (a) RESTART IDENTITY (Req 2.4): an entry's (term, lsn) identity must
|
||||
* survive a restart/recovery. persist.c's replay_wal already calls
|
||||
* record_entry_term for every replayed entry (~persist.c:2731), so this
|
||||
* is a regression PIN (test_6_2_restart_preserves_term_identity), not a
|
||||
* RED — it also pins the legacy-zero (WAL_LEGACY_TERM) mapping surviving
|
||||
* identically across the restart, not just a real assigned term.
|
||||
*
|
||||
* (b) SINGLE-SEAM INVARIANT / term-blind bypass disposition (carried
|
||||
* forward from task 2.4's Implementation Note): ha_apply_single_entry
|
||||
* (ha.c:2134) is a direct-persist route that skips
|
||||
* raft_on_append_entries_request's logOk/(term,lsn) reconcile.
|
||||
* ha_apply_wal_catchup_entry (ha.c:2270) still has TWO term-blind paths
|
||||
* into it — the batch-mode fallthrough when a sub-entry's term==0
|
||||
* (ha.c ~2328), and the legacy n_entries==0 single-entry mode
|
||||
* (ha.c ~2360), which is UNCONDITIONAL (no term check at all).
|
||||
*
|
||||
* REACHABILITY FINDING (this task, traced every production sender of a
|
||||
* wal_catchup_entry_t message end to end):
|
||||
* 1. repl_broadcast (repl.c:239, live WAL-append broadcast) ALWAYS
|
||||
* sends n_entries==1 with entry.term = hdr->term — the term
|
||||
* actually stamped on the just-appended entry. Routes through
|
||||
* ha_apply_wal_catchup_entry's branch 1 (seam).
|
||||
* 2. ha_stream_wal_to_standby (config_repl.c:2113, the catch-up
|
||||
* streamer) ALWAYS builds a batch with n_entries >= 1 (checked
|
||||
* before any batch is built — standby_lsn >= committed_lsn returns
|
||||
* early) and stamps every sub-entry's term via
|
||||
* persist_entry_term(p, hdr.lsn). entry_term_locked's legacy-zero
|
||||
* rule (persist.c:450-459) maps a raw on-disk term of 0 to
|
||||
* WAL_LEGACY_TERM (1) for any PRESENT lsn — persist_entry_term
|
||||
* returns 0 ONLY for an lsn that was never recorded at all. A real
|
||||
* streamed sub-entry's term is therefore NEVER 0. Routes through
|
||||
* branch 2's seam call (ha_apply_term_entry_via_library).
|
||||
* 3. repl_send_append_entries (repl.c:340) is the ONLY production
|
||||
* function that can ever emit n_entries==0 (a 0-entry AE heartbeat)
|
||||
* or a term==0 single entry. It is called ONLY from
|
||||
* ha_cb_send_append_entries_stub (the wired on_send_append_entries
|
||||
* callback, ha.c:1693/1815 s_real_cbs), which is called ONLY from
|
||||
* the raft library's raft_send_append_entries (raft.c:595) or the
|
||||
* XCE_TESTING-only helper ha_cb_send_append_entries_for_test.
|
||||
* raft_send_append_entries has ZERO production call sites — every
|
||||
* caller is in lib/raft/raft_sim.c (the pure library simulation
|
||||
* harness). So the callback is registered but never DRIVEN by
|
||||
* anything in the running controller: this is the only path that
|
||||
* could ever reach ha_apply_wal_catchup_entry's n_entries==0 branch
|
||||
* with a real message, and it is dormant/unreachable code today.
|
||||
*
|
||||
* CORROBORATION: the existing pin tests
|
||||
* (test_2_3_termblind_route_payload_guarded,
|
||||
* test_2_3_unrepairable_catchup_raises_nonpromotable_fence, above) can
|
||||
* only exercise the term-blind branches by hand-building a
|
||||
* wal_catchup_entry_t with term==0 and calling ha_apply_wal_catchup_entry
|
||||
* directly with c==NULL — bypassing every real sender, because no real
|
||||
* sender in the current codebase produces that shape.
|
||||
*
|
||||
* DECISION: DOCUMENT SAFE-BY-GUARD, do not structurally retire. The
|
||||
* term-blind branches are (i) unreachable with real production data
|
||||
* today (finding above), and (ii) even if reached, the task-2.2
|
||||
* (term,lsn,payload-crc)-aware guard in persist_wal_write_replicated
|
||||
* already prevents an empty-over-real degrade and raises the sticky
|
||||
* non-promotable fence on an unrepairable below-tip conflict — proven by
|
||||
* the two pin tests cited above. Structurally retiring would mean
|
||||
* inventing a term-0-safe seam variant (the exact coupling task 2.2's
|
||||
* own design note deferred) for a path with no live trigger — spending
|
||||
* effort/risk against YAGNI with no defect it would close. The
|
||||
* corresponding comments at ha_apply_single_entry / the two term-blind
|
||||
* branches in ha_apply_wal_catchup_entry (ha.c) record this disposition
|
||||
* in the production source. test_6_2_real_senders_never_produce_
|
||||
* termblind_shape below PINS the reachability finding itself: it drives
|
||||
* the two REAL senders with real WAL data (including a legacy
|
||||
* term-0-stamped entry) and asserts every wire entry they emit satisfies
|
||||
* n_entries>=1 && term!=0, so a future change that regresses a real
|
||||
* sender into the term-blind shape is caught here.
|
||||
*
|
||||
* Requirements: 1.1, 1.2, 2.4, 3.1.
|
||||
* ═══════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
/* (a) RESTART IDENTITY — an entry's (term, lsn) identity, INCLUDING the
|
||||
* legacy-zero -> WAL_LEGACY_TERM mapping, must read back identically after a
|
||||
* simulated controller restart (clean teardown + a FRESH persist_t replaying
|
||||
* the same on-disk WAL via persist_recover). */
|
||||
static void test_6_2_restart_preserves_term_identity(void)
|
||||
{
|
||||
hcs_member_t m;
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&m, "62restart"));
|
||||
|
||||
uint8_t full[HCS_FULL_LEN];
|
||||
hcs_fill_body(full);
|
||||
|
||||
/* Entries spanning several distinct terms, including one stamped with the
|
||||
* legacy term 0 at lsn 4 — Req 2.4 ("preserve each entry's original term
|
||||
* identity") must cover the legacy-zero mapping too, not just a real
|
||||
* assigned term. */
|
||||
enum { N = 5 };
|
||||
uint64_t terms[N + 1] = { 0, 3, 3, 7, 0 /* legacy */, 9 };
|
||||
for (uint64_t lsn = 1; lsn <= N; lsn++) {
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
|
||||
hcs_wal_put(&m, lsn, terms[lsn], WAL_JOB_SUBMIT, full, HCS_FULL_LEN),
|
||||
"restart-identity build write must persist (real fs required — "
|
||||
"bug-924)");
|
||||
}
|
||||
atomic_store_explicit(&m.p->committed_lsn, (uint64_t)N, memory_order_release);
|
||||
|
||||
/* Capture the BEFORE-restart term identity via the authoritative readback
|
||||
* API (applies the legacy-0 -> WAL_LEGACY_TERM mapping uniformly). */
|
||||
uint64_t before[N + 1];
|
||||
for (uint64_t lsn = 1; lsn <= N; lsn++)
|
||||
before[lsn] = persist_entry_term(m.p, lsn);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE((uint64_t)WAL_LEGACY_TERM, before[4],
|
||||
"precondition: a term-0-stamped entry reads back as WAL_LEGACY_TERM, "
|
||||
"not 0, even before the restart");
|
||||
|
||||
/* Simulate a restart: clean shutdown, then a FRESH persist_t replays the
|
||||
* same on-disk WAL via persist_recover (record_entry_term rebuilds the
|
||||
* per-LSN term index — persist.c ~2731). */
|
||||
hcs_member_down(&m);
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_member_reopen(&m));
|
||||
|
||||
for (uint64_t lsn = 1; lsn <= N; lsn++) {
|
||||
uint64_t after = persist_entry_term(m.p, lsn);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(before[lsn], after,
|
||||
"Req 2.4: an entry's (term, lsn) identity must survive a "
|
||||
"restart/recovery — replay_wal's record_entry_term must "
|
||||
"reconstruct the SAME term this LSN held before the restart");
|
||||
}
|
||||
|
||||
/* Sanity: the restart replayed the WAL bytes too, not just the term
|
||||
* index. */
|
||||
for (uint64_t lsn = 1; lsn <= N; lsn++) {
|
||||
hcs_probe_t pr;
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_wal_probe(&m, lsn, &pr));
|
||||
TEST_ASSERT_TRUE_MESSAGE(pr.present,
|
||||
"every pre-restart entry must still be present after recovery");
|
||||
TEST_ASSERT_EQUAL_UINT32(HCS_FULL_LEN, pr.len);
|
||||
}
|
||||
|
||||
hcs_member_down(&m);
|
||||
printf("test_6_2_restart_preserves_term_identity: done\n");
|
||||
}
|
||||
|
||||
/* Capture buffer for the repl_broadcast send hook (live replication sender). */
|
||||
#define HCS62_BCAST_MAX 4
|
||||
static wal_catchup_entry_t g_hcs62_bcast_entries[HCS62_BCAST_MAX];
|
||||
static int g_hcs62_bcast_n;
|
||||
|
||||
static int hcs_6_2_broadcast_hook(void* ep, wal_catchup_entry_t** entries,
|
||||
int n_entries)
|
||||
{
|
||||
(void)ep;
|
||||
g_hcs62_bcast_n = n_entries;
|
||||
for (int i = 0; i < n_entries && i < HCS62_BCAST_MAX; i++)
|
||||
g_hcs62_bcast_entries[i] = *entries[i];
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* (b) REACHABILITY PIN — the two real production senders (repl_broadcast for
|
||||
* live replication, ha_stream_wal_to_standby for catch-up) can never produce
|
||||
* the term-blind shape (n_entries==0, or a term==0 sub-entry) that would
|
||||
* route through ha_apply_wal_catchup_entry's direct-persist bypass. This
|
||||
* operationalizes the reachability finding in the block comment above: if a
|
||||
* future change regressed a real sender into emitting that shape, this test
|
||||
* would catch it. */
|
||||
static void test_6_2_real_senders_never_produce_termblind_shape(void)
|
||||
{
|
||||
/* ── Sender 1: repl_broadcast (live WAL-append replication) ────────────── */
|
||||
replicator_t r;
|
||||
TEST_ASSERT_EQUAL_INT(0, repl_init(&r));
|
||||
int dummy_ep = 0;
|
||||
TEST_ASSERT_TRUE(repl_add_standby(&r, (void*)&dummy_ep, "standby", 7001) >= 0);
|
||||
|
||||
uint8_t full[HCS_FULL_LEN];
|
||||
hcs_fill_body(full);
|
||||
|
||||
wal_entry_hdr_t hdr;
|
||||
memset(&hdr, 0, sizeof(hdr));
|
||||
hdr.lsn = 5;
|
||||
hdr.type = WAL_JOB_SUBMIT;
|
||||
hdr.term = 7; /* the real term stamped on this append by the leader */
|
||||
|
||||
g_hcs62_bcast_n = 0;
|
||||
repl_broadcast_send_hook = hcs_6_2_broadcast_hook;
|
||||
int brc = repl_broadcast(&r, &hdr, full, HCS_FULL_LEN, /*prev_log_term=*/6);
|
||||
repl_broadcast_send_hook = NULL;
|
||||
repl_fini(&r);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, brc);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_hcs62_bcast_n,
|
||||
"PIN: repl_broadcast always sends exactly ONE entry (n_entries==1) — "
|
||||
"it can never produce the n_entries==0 heartbeat shape");
|
||||
TEST_ASSERT_NOT_EQUAL_MESSAGE(0u, g_hcs62_bcast_entries[0].term,
|
||||
"PIN: repl_broadcast stamps the entry's REAL term (hdr->term) — it "
|
||||
"can never emit the term==0 term-blind shape");
|
||||
|
||||
/* ── Sender 2: ha_stream_wal_to_standby (catch-up streamer) ────────────── */
|
||||
hcs_member_t src, rcv;
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&src, "62src"));
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&rcv, "62rcv"));
|
||||
|
||||
/* lsn 1 is stamped with the LEGACY term 0 — the shape most likely to slip
|
||||
* a term==0 sub-entry into a real catch-up batch, if it were possible. */
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_wal_put(&src, 1, /*term=*/0 /* legacy */,
|
||||
WAL_JOB_SUBMIT, full, HCS_FULL_LEN));
|
||||
TEST_ASSERT_EQUAL_INT(0, hcs_wal_put(&src, 2, /*term=*/4, WAL_JOB_SUBMIT,
|
||||
full, HCS_FULL_LEN));
|
||||
atomic_store_explicit(&src.p->committed_lsn, 2, memory_order_release);
|
||||
|
||||
hcs_2_3_reset(&rcv, /*fault_lsn=*/0 /* no fault injected below */);
|
||||
xce_wal_stream_request_hook = hcs_2_3_capture_hook;
|
||||
int src_rc = ha_stream_wal_to_standby((endpoint_t*)NULL, src.wal_dir, src.p,
|
||||
/*leader_term=*/9, /*standby_lsn=*/0,
|
||||
/*committed_lsn=*/2);
|
||||
xce_wal_stream_request_hook = NULL;
|
||||
TEST_ASSERT_EQUAL_INT(0, src_rc);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_sends_n,
|
||||
"PIN: the streamer batched BOTH entries (n_entries>=1 in the wire "
|
||||
"container) — catch-up never emits the n_entries==0 shape");
|
||||
for (int i = 0; i < g_sends_n; i++)
|
||||
TEST_ASSERT_NOT_EQUAL_MESSAGE(0u, g_sends[i].term,
|
||||
"PIN: every streamed sub-entry's term is non-zero — even an "
|
||||
"on-disk legacy term==0 entry reads back as WAL_LEGACY_TERM via "
|
||||
"persist_entry_term, so the streamer can never emit a term==0 "
|
||||
"sub-entry into a real catch-up batch");
|
||||
|
||||
hcs_member_down(&src);
|
||||
hcs_member_down(&rcv);
|
||||
printf("test_6_2_real_senders_never_produce_termblind_shape: done\n");
|
||||
}
|
||||
|
||||
/* ── Suite entry point ─────────────────────────────────────────────────────*/
|
||||
|
||||
void test_ha_core_safety(void)
|
||||
|
|
@ -2459,5 +2730,15 @@ void test_ha_core_safety(void)
|
|||
* instances: no term==0 on the wire from a live leader, no non-empty
|
||||
* entry ever lands as PLEN=0, and walcompare reports a clean prefix. */
|
||||
RUN_TEST(test_6_1_replication_and_catchup_preserve_term_payload_e2e);
|
||||
/* Task 6.2: single-seam invariant + restart identity.
|
||||
* PIN — an entry's (term, lsn) identity, including the legacy-zero ->
|
||||
* WAL_LEGACY_TERM mapping, survives a simulated restart/recovery.
|
||||
* PIN — the two real production senders (repl_broadcast,
|
||||
* ha_stream_wal_to_standby) can never produce the term-blind
|
||||
* shape (n_entries==0 or a term==0 sub-entry); see the reachability
|
||||
* finding + safe-by-guard decision recorded in the block comment
|
||||
* above and in ha.c. */
|
||||
RUN_TEST(test_6_2_restart_preserves_term_identity);
|
||||
RUN_TEST(test_6_2_real_senders_never_produce_termblind_shape);
|
||||
UNITY_END();
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue