wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3)

Browse source 
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.

A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.

Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.

New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.

Closes out task-group 5 (restart apply-correctness verification).
This commit is contained in:
Rongsong Shen 2026-07-04 13:33:03 +00:00
parent 099ee1a216
commit d3da3ca8bd
5 changed files with 616 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.kiro/specs/raft-apply-decoupling/tasks.md
View file

@ -106,7 +106,7 @@
- _Requirements: 6.1, 6.2_
- _Depends: 4.1_
- [ ] 5. Restart apply-correctness verification
- [x] 5. Restart apply-correctness verification
- [x] 5.1 (P) Compute a post-replay expectation from the committed log tip and compare it against materialized state
- Derive a lightweight expectation (e.g., last-applied position and a count or checksum of materialized entities) from the recovered log
- Compare the expectation against the actual state produced by replay through the shared apply dispatch function
@ -121,7 +121,7 @@
- _Requirements: 8.3, 8.4_
- _Depends: 5.1_
- [ ] 5.3 Unit test the DIVERGENT verdict path stays alive and keeps retrying
- [x] 5.3 Unit test the DIVERGENT verdict path stays alive and keeps retrying
- Force a DIVERGENT verdict and confirm the process does not exit or abort
- Confirm the fence clears once a subsequent recovery reaches a consistent state
- Observable: the test process remains running and retrying after a forced DIVERGENT verdict, never terminating
@ -193,6 +193,7 @@
- **1.2**: implemented `commit_resolve()` + wired it into all 6 commit.c-internal paths that can advance `committed_lsn`/`writable_lsn` (the COMMITTED/majority-ack outcome). Chose `VTASK_LIVE_BLOCK` for dispatch (never admission-gated, unlike `VTASK_REQ_BLOCK` which can fast-reject — unacceptable for "exactly-once, never unresolved"). **Reviewer caught a real task-graph gap, not a defect in 1.2 itself**: converting `COMMIT_MODE_WAIT` away from a blocking wait also removes the *only* mechanism that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline) — nothing yet arms an equivalent timeout for async-registered continuations, and nothing yet wires demotion/log-truncation detection to call `commit_resolve(FAILED, ...)`. Without this, a pending async command during quorum loss would hang forever instead of failing — worse than today's behavior. Split out as new tasks **1.3** (timeout arming) and **1.4** (demotion/truncation wiring), inserted before the renumbered **1.5** (was 1.3, concurrency test — now exercises the real detection paths from 1.3/1.4 instead of calling `commit_resolve()` directly). Task 3.2 now also depends on 1.3/1.4, since its own "failure/timeout outcome replies" observable is meaningless without them.
- **1.3**: added a dedicated per-commit-engine sweep thread (not the shared `delaymgr` — would violate boundary + delaymgr's 200ms poll granularity vs. `DEFAULT_SYNC_TIMEOUT_MS`=500ms would add timing jitter). Race rule: majority-satisfied always wins over an already-expired deadline when both are true under the same lock hold. **Round 1 REJECTED**: `commit_sweep_once()` computed the next-deadline snapshot under `ce->lock`, unlocked to dispatch, then slept on the *stale* snapshot — a `commit_submit_async()` registration/broadcast landing during that unlocked dispatch window was lost (no thread parked on the condvar to receive it), so the sweep thread could sleep on an absent/late deadline indefinitely (worst case: unbounded delay in a low-traffic quorum-loss scenario, violating the "identical failure timing" no-regression bar). **Fixed** by holding `ce->lock` continuously across scan→wait decisions (releasing only for the unlocked dispatch step) and always re-scanning under the lock immediately before ever calling `cond_wait`/`cond_timedwait` — the standard recheck-then-wait idiom. Round 2 reviewer independently reproduced the round-1 bug shape to confirm the fix actually closes it, not just narrows it. **Precedent for 1.4**: any future code that needs to interrupt the sweep thread's wait (e.g., a demotion/truncation handler resolving a pending LSN out-of-band) must go through the same lock+broadcast discipline `commit_submit_async()` uses, or it risks the identical lost-wakeup class of bug.
- **1.4**: added `commit_resolve_pending_above(ce, lsn_floor, rc)` (snapshot-then-`commit_resolve()`-per-entry, no parallel bookkeeping). Truncation: a single new `on_truncate`/`truncate_cb_ctx` callback slot on `persist_t` (same inversion-of-control shape as the existing `on_durable_advance` hook) fires from `persist_truncate_wal_above` after releasing `p->lock`, only when a truncation actually removed entries — this ONE wrapper covers all three real truncation call sites (`raft_cb_become_leader`'s post-election truncate, the library's `on_truncate_from` conflict-reconcile, `ha_demotion_convergence_fn`'s post-demotion truncate), verified by the reviewer reading all three. Demotion: `raft_cb_become_follower` calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` (lsn_floor=0 → fails everything — a demoted node can never authoritatively confirm anything pending; verified against the CP posture). **Known residual, non-blocking**: a fourth internal truncation path (`persist.c` inside `persist_wal_write_replicated`'s tip-conflict-repair branch, STANDBY replicated-apply only) calls `truncate_wal_above_locked()` directly, bypassing the new wrapper — safe today only because demotion resolves ALL pending continuations before role flips to STANDBY (so nothing can still be registered by the time that path runs), but this is an *implicit* ordering invariant, not an asserted one. Flag for whoever next touches the STANDBY replicated-apply path: add a defensive comment/assertion there, or route it through the wrapper too, before relying on it further. Also: bug-1407 was believed closed at this point (70/70 across 3 runs) — **see 1.5's note below, this was a false reading, not a real fix**.
- **5.3**: closes task-group 5. Took 3 review rounds — the heaviest task in the entire series, and the only one that surfaced a genuine CRITICAL production safety bug (not a doc/log issue like 5.1/5.2's rounds). **Round 1 (test-only) REJECTED**: while writing tests for "the fence stays alive" and "the fence clears on a subsequent consistent recovery," review discovered the sticky `recovery_unrecoverable` flag is trigger-agnostic — it has ZERO provenance about WHY it was set. Task 5.2 correctly reused this flag for the new `apply_divergent` trigger, but TWO PRE-EXISTING, live, running-process call sites in ha.c already clear this SAME flag based purely on STRUCTURAL conditions (never checking the apply verdict): `promotion_fence_timeout_fn` (a self-re-arming periodic timer) and the promotion-eligibility gate in `raft_cb_become_leader`. **Concrete, reachable attack path**: a node's local apply dispatch has a defect (materialized state diverges from the committed log) while its WAL bytes are otherwise fine (structurally healthy — orthogonal to the apply defect by construction). The apply-divergent fence sets correctly at boot. But WHILE THE PROCESS KEEPS RUNNING, the periodic timer sees the structural conditions were always fine and silently calls `persist_clear_recovery_unrecoverable()` — releasing a fence that was never actually resolved. If this node then wins a later election, the promotion gate now sees `unrecoverable==false` and promotes it to primary — **a genuinely divergent node serving authoritative state**, a direct Req 8.4 / CLAUDE.md "never serve divergent state" violation. **Round 2 fixed the code**: added `apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT` as an additional guard ANDed into BOTH clear-call-site conditions — `promotion_fence_timeout_fn` (line ~876) and `raft_cb_become_leader`'s gate (line ~1130) — so an apply-divergence-triggered fence can now ONLY be cleared by a fresh process restart's clean `persist_recover()` (since `apply_verdict` has exactly two production write sites: `persist_init()`'s reset-to-`APPLY_CLEAN` and `persist_recover()`'s boot-time computation — no live path resets it). Verified via full trace + RED→GREEN for site 1. **Round 2 REJECTED anyway**: the fix was logically sound, but review PROVED site 2's guard had ZERO test coverage — by deleting only that guard and re-running the ENTIRE 298-test regression sweep across 13 groups, every single test still passed. An unverified defense-in-depth guard on promotion/election-safety code doesn't meet the bar. **Round 3 (test-only) closed it**: added a test that structurally isolates site 2 from site 1 — proven not to be exercising the already-covered sticky-flag path, since `ha_promotion_gate_decision()` never populates `apply_divergent` in its own inputs, so ONLY the round-2-added standalone guard at the call site can catch this scenario. RED→GREEN independently reproduced by two separate reviewers reverting the guard and confirming the exact predicted test failure. **Final closed picture, verified end-to-end across all 3 rounds**: (a) apply-divergence sets the fence correctly (5.2); (b) survives the structural timer's periodic re-check (round 2); (c) survives the promotion gate's re-check even in the hypothetical where the sticky flag itself somehow reads false (round 3); (d) only clears via a fresh process restart replaying cleanly — confirmed by round 1's own two-separate-`persist_t`-instances test. **Forward note for task 6, already flagged in-repo (test_apply_correctness_verification.c comments), not a defect today**: this entire safety picture rests on `apply_verdict` only ever being recomputed once, at boot. If task 6's live apply pump ever adds a live re-verification of apply-correctness (not currently planned, but worth checking), the "restart-only" assumption these 3 rounds of guards depend on would need re-examination at that time. **Lesson for the whole series**: reusing an existing mechanism correctly at the logic/decision level (5.2's `apply_divergent` short-circuit) does not automatically make every OTHER consumer of that same mechanism's output (ha.c's two independent, pre-existing live-clear call sites, written before this feature existed and with no reason to have considered a future trigger) still behave safely — a shared, trigger-agnostic sticky flag is exactly the kind of implicit coupling that needs an explicit audit of ALL its readers/writers whenever a new trigger is added, not just the one you're actively wiring.
- **5.2**: routed `APPLY_DIVERGENT` (task 5.1) into the EXISTING loud-fence mechanism as a new INPUT, not a new fencing path — added `bool apply_divergent` to the existing `recover_policy_inputs_t`; `persist_recover_decision()` now unconditionally short-circuits to the pre-existing `RECOVER_DECISION_FENCE_UNRECOVERABLE` enum value when it's true (correct and design-mandated: a divergent local materialization can't be trusted for a peer/catch-up comparison either, so it must always fence regardless of peer state); reuses the same sticky `recovery_unrecoverable` flag and the same `persist_recover_loud_fence_if_unrecoverable()` function — confirmed by both review rounds to be the identical pre-existing symbols, zero new fence mechanism. Confirmed the existing promotion-gate consultation in ha.c (~line 1078, `persist_recovery_unrecoverable(&c->persist)`) already independently reads this flag — zero ha.c changes needed. Zero new self-termination paths (verified by grep both rounds) — stays alive, keeps retrying, per the CLAUDE.md process-lifecycle invariant. Took 2 review rounds. **Round 1 REJECTED — real, test-reproduced defect, not documentation**: the operator-facing ERROR() log's fixed boilerplate clause ("no healthy peer reachable to catch up; durable_floor_lsn=X < cluster committed high-water hwm=Y") fired UNCONDITIONALLY for every fence decision, including the new apply_divergent trigger — but for that trigger the fields are either hardcoded placeholders or explicitly-set values that made the clause LITERALLY FALSE in test output ("100 < 100", "2 < 0" — numerically impossible claims presented as fact to an operator during a real incident). Before this task, `RECOVER_DECISION_FENCE_UNRECOVERABLE` was only reachable via the structural "no healthy peer" branch, so the clause was truthful by construction; the new unconditional short-circuit broke that invariant. This directly undermines Req 8.3's "loud, operator-visible" requirement — a log an operator can't trust during an incident is worse than no log. **Round 2 fixed** by branching the ERROR() call itself (not just the inner reason string) on `apply_divergent` — that branch now states only the real, verified reason with no peer/HWM claim; the structural-fence branch is byte-identical to before. New regression test uses a genuine (non-mocked) stdout-capture mechanism relying on `controller_unit_test` never calling `tlog_init()` (verified true across all ~85 files in the test binary — confirmed the vendored tinylog fallback path used is a real raw `write(STDOUT_FILENO,...)`, not a stub) — both review rounds independently reverted the fix and reproduced the exact false-clause failure before re-confirming GREEN. **Lesson, consistent with task 2.2's earlier pattern**: reusing an existing decision path correctly at the LOGIC level doesn't automatically make every SIDE EFFECT of that path (like a log message assuming context from the old, narrower set of triggers) still valid for a newly-added trigger — check accompanying log/error text for assumptions that were true only because of the old, narrower reachability, not just the returned enum/decision value itself.
- **5.1**: added a new `apply_verdict_t` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) computed alongside the EXISTING, untouched structural `recover_verdict_t` (CRC) verdict — a dual-check design: (1) an entity-count check comparing `WAL_JOB_SUBMIT` entries scanned during replay against the live `c->jobmgr` table's actual growth (`count_jobmgr_entries()`), genuinely independent since it queries live state rather than trusting `apply_committed_entry()`'s return code — verified real by both review rounds via a forced job-count-mismatch test; (2) a position check comparing an "expected" LSN against `p->last_applied_lsn` (new `_Atomic uint64_t` field, per design.md, forward-looking infrastructure for task 6's live pump). Took 2 review rounds. **Round 1 REJECTED — real, live-today bug, not documentation**: the position check's "expected" value was computed as `recovered_tip = current_lsn - 1` (the PHYSICAL WAL-scan tip, which `replay_wal()` advances for every entry including `WAL_CHECKPOINT` markers and unknown types) instead of the LOGICAL apply cursor. `persist_checkpoint()` durably fsyncs the checkpoint marker in one step, then persists the advanced `checkpoint_lsn` to the on-disk snapshot in a LATER step — a crash between those two steps (a real, unavoidable crash-safety window this codebase already engineers hard for, cf. bug-661/ha-commit-durability) leaves a trailing checkpoint marker as the highest-LSN entry on next boot, causing `recovered_tip` to exceed the true `last_applied_lsn` and falsely report `APPLY_DIVERGENT` on a perfectly clean recovery — a direct Req 8.2 violation, reachable TODAY on any node that crashes at exactly the wrong moment during a checkpoint. **Round 2 fixed** by reading `p->last_applied_lsn` directly (the value `replay_wal()` already correctly advances only for entries actually dispatched through `apply_committed_entry()`, skipping checkpoint-marker/unknown-type entries) instead of `recovered_tip`; `recovered_tip` itself is untouched and still correctly feeds `durable_lsn`, which does need the physical tip. New regression test genuinely reproduces the trigger condition (real `persist_checkpoint()`, then `unlink()`s the exact snapshot file it wrote, landing on-disk state in the identical place a real crash would) — both review rounds independently reverted the fix and reproduced the false-DIVERGENT failure before re-confirming GREEN. **Known, disclosed, accepted scope gap (not a defect, flagged for whoever extends this check)**: the entity-count check covers jobs only. Queues are correctly excluded (config-derived, `WAL_QUEUE_CTRL` never creates a new one, verified by both review rounds). **Workers are NOT covered but CAN diverge**: `worker_from_capnp()`/`restore_worker()` genuinely creates a new worker on a `WAL_WORKER_CTRL`/`HOOK_DETAIL_WORKER_JOIN` entry when absent from `workermgr` — a worker-creation divergence is currently invisible to this verification. Extend `persist_verify_apply_correctness()` with a worker-count check (mirroring `count_jobmgr_entries()`) if/when this needs to be closed. **Self-disclosed, accepted limitation**: because `replay_wal()`'s single-threaded caller in `persist_recover()` derives both the "expected" and "actual" position values from the same atomic within moments of each other on the same thread, the position check is diagnostically tautological FOR THIS SPECIFIC CALL SITE today (though the entity-count half remains genuinely independent) — this will change once task 6's live apply pump gives expectation and actual state a real chance to diverge across threads/time; task 5.1's boundary does not require solving that now.
- **4.2**: pure test-only close-out of task-group 4 — one new test `test_fact_state_unchanged_throughout_quorum_loss_outage` in `test_commit_barrier_reply.c` (30→31), approved first round, zero production changes. Drives a genuine `repl_remove_standby`-based quorum-loss window through all 4 real fact-receipt handlers, polls live job state + `committed_lsn` at 5 time-spread points (real 20ms sleeps, ~100ms total) rather than once, asserts a real hook-spy never sees `AFTER_EVENT` fire, and confirms durable WAL presence via direct on-disk segment read (`cbrp_count_jobstate_entries`) — combining and strengthening two pre-existing tests that each covered only half (4.1's own single-point-in-time check under normal conditions; an older pre-4.1 test that proved WAL durability but never inspected live state). **Static/inspection half of Req 6.2 confirmed independently by the reviewer, re-reading 4.1's landed diff**: none of the four `record_*_fact` functions call `apply_committed_entry`/`restore_job`/mutate the real job object — `record_job_fail_fact` specifically operates on a throwaway `job_t scratch = *job` shallow copy. **Architecturally significant finding for task 6 planning**: facts have NO commit-timeout/failure concept at the receipt layer, unlike commands (task 3.2's `commit_submit_async`). `persist_submit_job_fact()` → `wal_hook_append_classified(WAL_MODE_RETAIN)` → `commit_submit(..., COMMIT_MODE_RETAIN, 0)`, and `COMMIT_MODE_RETAIN`'s branch in commit.c is an immediate, unconditional return with NO continuation registered (explicit comment: "RETAIN never honors the timeout") — `commit_resolve()` never fires for a retained fact. A fact's outage is therefore open-ended by construction; its eventual commit is discovered only when `commit_advance_locked()`/`commit_on_ack` notices `committed_lsn` can advance past it via the ACK-registry watermarks, purely. **Task 6's apply pump must rely entirely on `committed_lsn` advancement to discover newly-eligible retained facts — there is no per-fact signal to wait on.** Confirmed neither of 4.1's two flagged hazards (double-append, merge-staleness) is triggered by this test, since it never drives apply. **Task-group 4 (fact apply-after-commit split) is now complete**: 4.1 (receipt converted to durable-append-only, two critical hazards discovered and documented for task 6) → 4.2 (retention proven under sustained quorum loss, fact-vs-command architectural asymmetry documented). **Both task-groups 1-4 are now done; task 6 (ordered apply pump) is the critical next milestone that must resolve, before going live: the double-append hazard, the merge-staleness hazard, AND design around facts' lack of a per-entry commit signal (commit_advance/committed_lsn-driven discovery only).** Groups 5 (restart verification) and 6 (apply pump) remain; 5.1 depends only on 2.3 (already done) so could run in parallel with 6, per tasks.md's dependency graph — continuing in numeric order (5 next) unless reprioritized.

65
src/controller/ha.c
View file

@ -859,10 +859,21 @@ static int promotion_fence_timeout_fn(void *arg)
? (repl_active_standby_count(ha->repl) > 0)
: false;
/* Caught up: not truncated AND durable position reached the boundary →
* release the fence + clear the sticky flag; the member is eligible on its
* next election (Req 2.6). */
if (!truncated && !below_hwm) {
/* raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4): a fence
* set for an APPLY_DIVERGENT reason must NEVER be live-cleared by this
* purely-structural re-check. apply_verdict is only ever recomputed by
* persist_recover() (a boot-time-once replay, cluster.c) — more WAL
* bytes arriving or a healthy peer appearing cannot prove a local
* apply-dispatch defect is fixed, so this periodic timer must treat an
* apply-divergent fence as untouchable until the next full restart
* replays cleanly (persist_recover_decision fences apply_divergent
* unconditionally for the same reason — see persist.c). */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* Caught up: not truncated AND durable position reached the boundary AND
* the current apply verdict is not DIVERGENT → release the fence + clear
* the sticky flag; the member is eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
@ -1077,6 +1088,16 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool unrecoverable = persist_recovery_unrecoverable(&c->persist);
/* raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4):
* same guard as promotion_fence_timeout_fn above — a fence set for
* an APPLY_DIVERGENT reason must never be live-cleared, and this
* gate must never promote while the current apply verdict is
* divergent, REGARDLESS of what `unrecoverable` currently reads
* (defense-in-depth: it must not rely on the other call site alone
* having kept `unrecoverable` accurate). apply_verdict is only ever
* recomputed by a fresh boot-time persist_recover() replay. */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
* channels promoted below). */
@ -1106,26 +1127,30 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable) {
/* ELIGIBLE and not sticky-unrecoverable: release the fence (a
* caught-up member regains eligibility, Req 2.6), clear the sticky
* flag if set, and cancel any pending re-alert timer. Proceed to
* authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent) {
/* ELIGIBLE, not sticky-unrecoverable, AND not apply-divergent:
* release the fence (a caught-up member regains eligibility,
* Req 2.6), clear the sticky flag if set, and cancel any pending
* re-alert timer. Proceed to authoritative promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE, UNRECOVERABLE, or sticky-unrecoverable: refuse promotion —
* do NOT become CTRL_PRIMARY. Fence the member and arm the catch-up
* re-alert timer. The member STAYS ALIVE, non-promotable, retrying
* catch-up (Req 2.2/2.4). NEVER self-exit, never fence silently. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + arming "
"catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4)",
(int)gate, unrecoverable, truncated, durable_floor,
promotion_safe_lsn, peer_reachable);
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, or apply-divergent:
* refuse promotion — do NOT become CTRL_PRIMARY. Fence the member
* and arm the catch-up re-alert timer. The member STAYS ALIVE,
* non-promotable, retrying catch-up (Req 2.2/2.4). NEVER
* self-exit, never fence silently. An apply-divergent member is
* refused unconditionally (Req 8.4) — no peer/HWM condition can
* clear that trigger; only a fresh, clean restart replay can. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d "
"apply_divergent=%d) — recovery_truncated=%d durable_floor=%"
PRIu64 " cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"arming catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4; raft-apply-decoupling "
"Req 8.4)",
(int)gate, unrecoverable, apply_divergent, truncated,
durable_floor, promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;

367
src/controller/testsuite/test_apply_correctness_verification.c
View file

@ -814,6 +814,370 @@ static void test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause(void
"PASS\n");
}
/* ── raft-apply-decoupling task 5.3 (Req 8.4): the DIVERGENT verdict path
* stays alive and keeps retrying, and the fence clears once a subsequent
* recovery reaches a consistent state.
*
* Task 5.2's own tests already establish, implicitly, that the process
* survives a forced DIVERGENT verdict (persist_recover() and
* persist_recover_loud_fence_if_unrecoverable() both RETURN normally, and
* the whole controller_unit_test binary keeps running every later test).
* This section adds a MORE EXPLICIT, direct pin on that guarantee, plus
* NEW coverage of the fence-clearing half, which 5.2 explicitly deferred
* ("this type only classifies, it does not fence" / clearing was never
* exercised anywhere in this file). ─────────────────────────────────── */
/*
* More explicit "stays alive" pin than 5.2's implicit evidence. Forces a
* REAL, end-to-end APPLY_DIVERGENT verdict via an actual persist_recover()
* replay (the duplicate-jobid trigger, same as
* test_real_replay_divergence_sets_unrecoverable_fence), then performs
* MULTIPLE, clearly UNRELATED, real operations on the SAME now-fenced
* persist_t afterward, in the SAME test function and thread -- proving
* control genuinely returned to the caller and the process remains fully
* live and functional, not merely "hasn't crashed yet by the time one
* boolean was checked."
*
* The follow-up WAL append models a real operational fact: fencing
* (recovery_unrecoverable) only refuses PROMOTION (ha.c's promotion gate,
* ~line 1078); it does not stop the member from continuing to operate as an
* ordinary, alive standby that keeps appending/replicating entries and
* keeps retrying catch-up, per the controller-never-self-terminate
* invariant (src/controller/CLAUDE.md "Process Lifecycle").
*/
static void test_apply_divergent_process_stays_alive_and_functional_after_fence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_tacv_alive_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Phase 1: duplicate-jobid WAL -- the same genuine divergence trigger as
* test_real_replay_divergence_sets_unrecoverable_fence. */
{
persist_t* p = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 2; i++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 76001;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
/* Phase 2: recover into divergence, then prove the process stays alive
* and functional with several explicit, unrelated follow-up steps. */
{
persist_t* p2 = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
tacv_build_cluster();
p2->cluster = &g_tacv_c;
/* persist_recover() itself RETURNING (rc==0, not a crash / not a
* jump to an abort handler) is the first-order proof of life;
* everything below is this task's requested MORE EXPLICIT pin on
* top of that. */
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p2),
"fixture sanity: the duplicate-jobid replay must genuinely "
"diverge");
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"the fence must be set before this test's stays-alive pin means "
"anything");
/* Explicit pin #1: repeated, independent reads across several
* statements -- each one a real function call plus atomic load that
* requires a live, unaborted thread to execute. Not a single
* fire-and-forget check. */
for (int i = 0; i < 5; i++) {
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"the fence must remain observably set across repeated "
"reads -- each read requires the calling thread to still "
"be alive");
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p2),
"the verdict must remain stable across repeated reads");
}
/* Explicit pin #2: a genuinely unrelated, real persist operation
* performed AFTER the fence trips -- concrete proof the process is
* fully functional (file I/O, WAL locking, fsync path all still
* operate), not merely "didn't crash by the time a boolean was
* checked." */
{
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 76099;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
persist_wal_async(p2, WAL_JOB_SUBMIT, payload, (uint32_t)plen),
"a fenced-but-alive member must still be able to append WAL "
"entries -- fencing refuses PROMOTION only, never ordinary "
"operation");
free(payload);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, persist_flush(p2),
"the WAL flush/fsync path must still be fully operational "
"after the fence trips");
}
/* Explicit pin #3: the fence itself is unperturbed by ordinary
* continued operation -- it stays sticky exactly as designed; only
* an explicit clear (tested separately below) releases it. */
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"ordinary continued operation after the fence trips must not "
"silently clear it");
tacv_teardown_cluster();
tacv_teardown_persist(p2);
}
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_apply_divergent_process_stays_alive_and_functional_after_fence: "
"PASS\n");
}
/*
* "Confirm the fence clears once a subsequent recovery reaches a consistent
* state" (task 5.3 text) -- the direct, mechanical half: persist_clear_
* recovery_unrecoverable() is the SAME generic, trigger-agnostic mechanism
* the pre-existing STRUCTURAL fence already uses and already proves clears
* correctly (test_commit_durability.c test_boot_policy_row4_truncated_no_
* peer_loud_fence / row4b). Nothing in persist_recover_loud_fence_if_
* unrecoverable()'s apply_divergent branch (persist.c) creates a second,
* parallel, or harder-to-clear fence flag -- it is the exact same sticky
* `recovery_unrecoverable` atomic used everywhere else. This test is the
* apply_divergent-triggered analogue of the row4/row4b precedent, closing
* the gap that no existing test in this file ever called persist_clear_
* recovery_unrecoverable() at all.
*/
static void test_apply_divergent_fence_clears_via_existing_clear_mechanism(void)
{
persist_t p;
memset(&p, 0, sizeof(p));
recover_policy_inputs_t in = {
.verdict = RECOVER_CLEAN,
.durable_below_hwm = false,
.healthy_peer_reachable = true,
.catch_up_succeeded = true,
.durable_floor_lsn = 100,
.cluster_hwm_lsn = 100,
.apply_divergent = true,
};
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&p),
"sanity: the sticky flag starts clear on a fresh persist_t");
recover_decision_t d = persist_recover_loud_fence_if_unrecoverable(&p, &in);
TEST_ASSERT_EQUAL_INT_MESSAGE(
RECOVER_DECISION_FENCE_UNRECOVERABLE, d,
"sanity: apply_divergent must decide the loud, non-terminating "
"fence");
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(&p),
"sanity: apply_divergent must set the sticky fence before this "
"test's clear assertion means anything");
/* Req 8.4 / CLAUDE.md "clear the fenced state once a committed-
* consistent state is reached": the SAME existing, generic clear
* primitive must release an apply_divergent-triggered fence exactly as
* it releases a structural one. */
persist_clear_recovery_unrecoverable(&p);
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&p),
"the apply_divergent-triggered fence must clear via the same "
"existing mechanism used for the structural fence -- no parallel, "
"harder-to-clear fence exists for this trigger");
printf("test_apply_divergent_fence_clears_via_existing_clear_mechanism: "
"PASS\n");
}
/*
* "Confirm the fence clears once a subsequent recovery reaches a consistent
* state" -- the end-to-end, restart-shaped half, and an important
* ARCHITECTURAL FINDING documented here rather than papered over:
*
* Unlike the pre-existing STRUCTURAL fence (recovery-truncated / behind-HWM),
* which has a LIVE self-heal path -- ha.c's promotion-fence-timeout re-check
* and promotion-gate re-evaluation periodically re-read the SAME live
* persist_t's structural getters (persist_recovery_truncated /
* persist_durable_floor) WHILE THE PROCESS STAYS UP, and call persist_clear_
* recovery_unrecoverable() the moment they observe the log has genuinely
* caught up (ha.c:865-873, ha.c:1109-1116) -- the apply_divergent trigger has
* NO analogous LIVE re-verification today. Task 5.1 established that
* persist_verify_apply_correctness() runs exactly once, at boot-time replay
* inside persist_recover(); nothing re-invokes it later in the life of the
* SAME process/persist_t (that is task 6's ordered-apply-pump territory, out
* of this task's boundary). Confirmed by inspection: persist_recover()'s
* APPLY_CLEAN branch never calls persist_clear_recovery_unrecoverable() (zero
* call sites of that function anywhere in persist.c -- grep-verified; its
* only callers are the two ha.c sites above), and this is CORRECT BY DESIGN,
* not a bug: task 5.2's own decision table fences apply_divergent
* UNCONDITIONALLY specifically because a local apply defect cannot be proven
* fixed by more WAL bytes arriving (persist_recover_decision's own comment),
* so it must never be downgraded the live way a structurally-behind log can.
*
* What IS real and available today is restart-based self-heal: the sticky
* flag lives only on a single persist_t instance (never persisted to disk),
* defaults to false on every fresh persist_init(), and persist_recover()
* only ever sets it true when it ACTUALLY diverges. This test proves that
* shape directly: one instance that genuinely diverges stays fenced, a
* separate, later instance that replays a genuinely CONSISTENT log is never
* fenced, and the first instance's own fence is untouched by the second,
* independent instance's clean outcome (ruling out a vacuously-always-false
* flag proving nothing).
*/
static void test_apply_divergent_fence_does_not_survive_a_clean_restart(void)
{
char tmp_divergent[256];
char tmp_clean[256];
strcpy(tmp_divergent, "/tmp/xce_tacv_restart_bad_XXXXXX");
strcpy(tmp_clean, "/tmp/xce_tacv_restart_ok_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_divergent));
TEST_ASSERT_NOT_NULL(mkdtemp(tmp_clean));
tacv_build_cluster();
/* "Boot 1": a duplicate-jobid WAL genuinely diverges and fences. */
{
persist_t* p = tacv_make_fresh_persist(tmp_divergent);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 2; i++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 77001;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
persist_t* p_boot1 = tacv_make_fresh_persist(tmp_divergent);
TEST_ASSERT_NOT_NULL(p_boot1);
p_boot1->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p_boot1));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p_boot1),
"fixture sanity: boot 1 must genuinely diverge");
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p_boot1),
"boot 1 must be fenced");
/* "Boot 2": a separate, later boot -- distinct jobids, no duplication --
* replays a genuinely consistent log. A fresh persist_t always starts
* unfenced (persist_init defaults recovery_unrecoverable=false); the
* point under test is that a CONSISTENT replay leaves it that way. */
{
persist_t* p = tacv_make_fresh_persist(tmp_clean);
TEST_ASSERT_NOT_NULL(p);
for (uint64_t jobid = 78001; jobid <= 78002; jobid++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
persist_t* p_boot2 = tacv_make_fresh_persist(tmp_clean);
TEST_ASSERT_NOT_NULL(p_boot2);
p_boot2->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p_boot2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p_boot2),
"fixture sanity: boot 2 must genuinely replay clean");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p_boot2),
"a subsequent (restart) recovery that reaches a consistent state "
"must not be fenced -- the fence does not survive past a clean "
"restart (Req 8.4)");
/* The FIRST boot's own fence is untouched by the second, independent
* boot's clean outcome -- proves this is a real, per-instance sticky
* transition, not a vacuously-always-false flag. */
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(p_boot1),
"an independent instance reaching a clean recovery must not "
"retroactively clear a DIFFERENT, still-divergent instance's fence");
tacv_teardown_persist(p_boot1);
tacv_teardown_persist(p_boot2);
tacv_teardown_cluster();
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp_divergent);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp_divergent);
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp_clean);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp_clean);
printf("test_apply_divergent_fence_does_not_survive_a_clean_restart: "
"PASS\n");
}
void test_apply_correctness_verification(void)
{
RUN_TEST(test_clean_replay_produces_apply_clean_alongside_crc_verdict);
@ -824,4 +1188,7 @@ void test_apply_correctness_verification(void)
RUN_TEST(test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive);
RUN_TEST(test_real_replay_divergence_sets_unrecoverable_fence);
RUN_TEST(test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause);
RUN_TEST(test_apply_divergent_process_stays_alive_and_functional_after_fence);
RUN_TEST(test_apply_divergent_fence_clears_via_existing_clear_mechanism);
RUN_TEST(test_apply_divergent_fence_does_not_survive_a_clean_restart);
}

83
src/controller/testsuite/test_commit_durability.c
View file

@ -2315,6 +2315,85 @@ static void test_promotion_fence_cancel_stops_realert(void)
printf("test_promotion_fence_cancel_stops_realert: PASS\n");
}
/* ── raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4) ───────────
* CROSS-CONTAMINATION REGRESSION: an APPLY_DIVERGENT-triggered fence must
* NEVER be live-cleared by this purely-structural re-check timer, even when
* the structural conditions it actually inspects (truncated / durable_floor
* vs HWM) become healthy. Round-1 review found BOTH ha.c live-clear call
* sites (this timer, ha.c ~867, and the promotion-eligibility gate in
* raft_cb_become_leader, ha.c ~1115) blind to WHY the sticky flag was set —
* they only ever inspected structural getters. A node whose local apply
* dispatch is defective (materialized state diverges from the committed log
* — task 5.1's APPLY_DIVERGENT verdict) can have a perfectly healthy WAL
* (not truncated, durable_floor >= HWM) by construction, since the two
* properties are orthogonal. Before the fix, this test drives exactly that
* cross-contamination and asserts the fence does NOT clear.
*
* This mirrors test_promotion_fence_released_on_catchup's harness and
* catch-up simulation EXACTLY, differing only in also marking the fence as
* apply_divergent-triggered up front — proving the fix discriminates on the
* apply verdict, not merely on the structural fields. */
static void test_promotion_fence_apply_divergent_survives_structural_catchup(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_fence_apply_div_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t* c = dur_make_fence_cluster(tmp, /*truncated=*/true,
/*durable_floor=*/4);
TEST_ASSERT_NOT_NULL(c);
/* Mark the fence as APPLY_DIVERGENT-triggered, exactly as persist_recover()
* does at boot (task 5.1/5.2) — the WAL-structural fields above
* (truncated/durable_floor) were already set independently above, proving
* the two conditions are orthogonal by construction. */
c->persist.apply_verdict = APPLY_DIVERGENT;
atomic_store_explicit(&c->persist.recovery_unrecoverable, true,
memory_order_release);
ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
uint64_t);
ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
/*durable_floor=*/4, /*hwm=*/10);
TEST_ASSERT_EQUAL_INT(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
/* Simulate the EXACT SAME structural catch-up as
* test_promotion_fence_released_on_catchup: log now intact and durable
* past the HWM. The apply_verdict / apply-divergent fence trigger is
* UNTOUCHED by this — it can only ever be recomputed by a fresh
* persist_recover() boot-time replay (task 5.1), never by a live
* structural mutation like this one. */
c->persist.recovery_verdict = RECOVER_CLEAN;
atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
/* Wait long enough for the window to fire. */
usleep(300 * 1000);
/* THE REGRESSION ASSERTION (Req 8.4): even though the structural
* conditions the timer inspects are now fully healthy, an
* APPLY_DIVERGENT-triggered fence must NOT be released — a local apply
* defect cannot be proven fixed by more WAL bytes arriving. Before the
* fix this assertion FAILS (the timer clears the flag purely on the
* structural check, oblivious to apply_verdict). */
TEST_ASSERT_TRUE_MESSAGE(
persist_recovery_unrecoverable(&c->persist),
"5.3 round-2: an APPLY_DIVERGENT-triggered fence must survive a "
"purely-structural catch-up re-check — only a fresh, clean "
"persist_recover() restart may clear it (Req 8.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
"5.3 round-2: the member must remain fenced (non-promotable) while "
"apply_divergent, regardless of structural catch-up");
ha_set_promotion_fence_window_ms(0);
dur_destroy_fence_cluster(c);
printf("test_promotion_fence_apply_divergent_survives_structural_catchup: "
"PASS\n");
}
#endif /* XCE_TESTING */
/* ------------------------------------------------------------------ */
@ -2375,6 +2454,10 @@ void test_commit_durability(void)
RUN_TEST(test_promotion_fence_unrecoverable_stays_alive_loud);
RUN_TEST(test_promotion_fence_released_on_catchup);
RUN_TEST(test_promotion_fence_cancel_stops_realert);
/* task 5.3 round-2 remediation (Req 8.4): cross-contamination regression —
* an APPLY_DIVERGENT-triggered fence must not clear on structural catch-up
* alone. */
RUN_TEST(test_promotion_fence_apply_divergent_survives_structural_catchup);
#else
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
#endif

118
src/controller/testsuite/test_convergence_verification.c
View file

@ -1187,6 +1187,119 @@ static void test_promotion_skips_registration_when_peer_null(void)
printf("test_promotion_skips_registration_when_peer_null: checked\n");
}
/* ── raft-apply-decoupling task 5.3 (round-3 remediation, Req 8.4) ──────────
*
* COVERAGE GAP closed here: round-2 added an `!apply_divergent` conjunct to
* BOTH the promotion_fence_timeout_fn re-alert timer (ha.c ~876) AND the
* raft_cb_become_leader pre-promotion eligibility gate (ha.c ~1130,
* `gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent`).
* The round-2 reviewer proved the SECOND site had zero test coverage: reverting
* ONLY that conjunct left the entire suite (298 tests across 13 groups,
* including every test that drives raft_cb_become_leader /
* raft_cb_become_leader_with_cluster_for_test) fully green. This test targets
* exactly the scenario that conjunct alone must catch: a node that is
* structurally ELIGIBLE (ha_promotion_gate_decision returns
* RECOVER_DECISION_ELIGIBLE — not truncated, durable_floor >= the cluster
* commit high-water mark) and NOT sticky-fenced (persist_recovery_unrecoverable
* is explicitly false, i.e. the OTHER guard has nothing to contribute), whose
* apply_verdict is nonetheless APPLY_DIVERGENT — a local materialized-state
* defect that only a fresh, clean persist_recover() boot replay may clear
* (task 5.1/5.2), never a peer/HWM condition.
*
* Fixture: identical to test_promotion_registers_standbys_when_peer_set above
* (n_peers=2, n_standbys=0, appended_lsn=0) — that sibling test's own
* assertion (repl_active_standby_count == 2 after promotion) is only
* reachable if this exact fixture already drives ha_promotion_gate_decision to
* RECOVER_DECISION_ELIGIBLE with unrecoverable == false, so reusing it here
* isolates the apply_divergent conjunct as the ONLY variable changed.
*
* RED (site-2 guard reverted): the gate promotes anyway — role flips to
* CTRL_PRIMARY and the vote endpoints get registered as standbys.
* GREEN (current code): promotion is refused — role stays non-PRIMARY, the
* member is fenced (in_catch_up=1, never-self-exit — it stays alive and keeps
* retrying catch-up), and no standbys are registered.
*/
static void test_promotion_refused_when_apply_divergent_even_if_eligible(void)
{
/* n_standbys=0, appended_lsn=0: same structurally-healthy fixture as
* test_promotion_registers_standbys_when_peer_set (durable_floor=0,
* recovery_truncated=false, and — with vote_granted left at its
* calloc-zero default — ha_compute_safe_committed_lsn's only counted
* voter is self at self_lsn=0, so promotion_safe_lsn=0). durable_floor(0)
* >= promotion_safe_lsn(0) and not truncated ⇒ genuinely ELIGIBLE. */
cv_setup("applydiv", /*n_peers=*/2, /*n_standbys=*/0,
/*appended_lsn=*/0, /*n_vcpus=*/8);
/* Precondition (Req 8.4 scenario, explicit per task spec): NOT fenced by
* the sticky flag — isolates the apply_divergent conjunct from the
* already-covered `unrecoverable` conjunct. */
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(&g_cv.persist),
"fixture precondition: sticky recovery_unrecoverable must be clear so "
"this test isolates the apply_divergent guard, not the sticky-flag "
"guard (Req 8.4)");
/* Force the apply-divergent verdict directly — exactly as task 5.1's
* persist_recover() replay leaves it after detecting a materialized-state
* mismatch, and exactly as
* test_promotion_fence_apply_divergent_survives_structural_catchup
* (test_commit_durability.c) sets it for the sibling (site-1) call site. */
g_cv.persist.apply_verdict = APPLY_DIVERGENT;
ha_elect_ctx_t *ctx = calloc(1, sizeof(*ctx) + 2 * sizeof(ctx->votes[0]));
TEST_ASSERT_NOT_NULL(ctx);
ctx->n_peers = 2;
pthread_mutex_init(&ctx->lock, NULL);
sem_init(&ctx->done, 0, 0);
xce_candidate_t peer0 = {0}, peer1 = {0};
cv_set_vote(ctx, 0, &peer0, "sb0.example", 7700);
cv_set_vote(ctx, 1, &peer1, "sb1.example", 7701);
g_cv.ha.standby_endpoints = calloc(1, sizeof(zarray_t));
TEST_ASSERT_NOT_NULL(g_cv.ha.standby_endpoints);
zarray_init(g_cv.ha.standby_endpoints, 4, NULL);
atomic_store_explicit(&g_cv.ha.elect_ctx, (void *)ctx, memory_order_release);
/* Act: real promotion attempt with a live, structurally-healthy cluster. */
raft_cb_become_leader_with_cluster_for_test(&g_cv, /*term=*/1);
/* Assert (Req 8.4): refused despite genuine structural ELIGIBLE +
* unrecoverable==false — the apply_divergent conjunct alone must block
* promotion. */
TEST_ASSERT_NOT_EQUAL_MESSAGE(
(int32_t)CTRL_PRIMARY,
atomic_load_explicit(&g_cv.ha.role, memory_order_acquire),
"raft-apply-decoupling task 5.3 (round-3 coverage, Req 8.4): a "
"structurally-eligible, non-sticky-fenced node whose apply_verdict is "
"APPLY_DIVERGENT must NOT be promoted to CTRL_PRIMARY");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load_explicit(&g_cv.ha.in_catch_up, memory_order_acquire),
"a refused apply-divergent promotion must fence the member "
"(in_catch_up=1) and keep it alive, retrying catch-up — never "
"self-exit (Req 8.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_active_standby_count(&g_cv.persist.repl),
"promotion refusal must happen BEFORE the standby-registration block "
"— no vote endpoint may be registered when the node never became "
"primary");
/* Cleanup */
atomic_store_explicit(&g_cv.ha.elect_ctx, NULL, memory_order_release);
if (g_cv.ha.standby_endpoints) {
zarray_fini(g_cv.ha.standby_endpoints);
free(g_cv.ha.standby_endpoints);
g_cv.ha.standby_endpoints = NULL;
}
pthread_mutex_destroy(&ctx->lock);
sem_destroy(&ctx->done);
free(ctx);
cv_teardown();
printf("test_promotion_refused_when_apply_divergent_even_if_eligible: "
"checked\n");
}
#endif /* XCE_TESTING */
/* ── Suite entry point ──────────────────────────────────────────────────── */
@ -1223,6 +1336,11 @@ void test_convergence_verification(void)
RUN_TEST(test_promotion_registers_standbys_when_peer_set);
RUN_TEST(test_promotion_skips_registration_when_peer_null);
printf("\n>> Promotion eligibility gate site-2 coverage "
"(raft-apply-decoupling task 5.3 round-3, Req 8.4)\n\n");
RUN_TEST(test_promotion_refused_when_apply_divergent_even_if_eligible);
#else
printf("test_convergence_verification: SKIP (non-test build)\n");
#endif