test(raft-apply-decoupling): durable fact retention under sustained quorum loss (task 4.2)
Add test_fact_state_unchanged_throughout_quorum_loss_outage to test_commit_barrier_reply.c (30->31), no production changes. Drives a real repl_remove_standby quorum-loss window through all 4 fact-receipt handlers, polls live state and committed_lsn at 5 time-spread points (not once), asserts a real hook-spy never sees AFTER_EVENT fire, and confirms durable WAL presence via direct on-disk read. Documents an architectural asymmetry for task 6 planning: facts have no commit-timeout concept (COMMIT_MODE_RETAIN registers no continuation, "never honors the timeout") unlike commands' commit_submit_async -- the apply pump must discover newly-committed retained facts purely via committed_lsn advancement, not any per-fact signal. Closes out task-group 4 (fact apply-after-commit split).
This commit is contained in:
parent
23cbfb48e1
commit
665b29f5b0
2 changed files with 202 additions and 2 deletions
|
|
@ -90,7 +90,7 @@
|
|||
- _Requirements: 2.1, 2.3_
|
||||
- _Depends: 3.2_
|
||||
|
||||
- [ ] 4. Fact apply-after-commit split
|
||||
- [x] 4. Fact apply-after-commit split
|
||||
- [x] 4.1 (P) Change worker-fact receipt handling to validate and durably append only
|
||||
- Remove the state mutation and the post-mutation notification hook firing from the receipt-time handlers for job start, finish, fail, and update reports
|
||||
- Keep the existing pre-mutation validation hook firing at receipt time, unchanged
|
||||
|
|
@ -99,7 +99,7 @@
|
|||
- _Depends: 2.1_
|
||||
- _Boundary: Fact Ingestion — job.c on_job_start/on_job_finish/on_job_fail/on_job_update receipt handlers only, disjoint from handle_jobsubmit_
|
||||
|
||||
- [ ] 4.2 Unit test durable retention of facts submitted while majority is unreachable
|
||||
- [x] 4.2 Unit test durable retention of facts submitted while majority is unreachable
|
||||
- Submit facts while majority is unreachable and confirm they are durably appended and that job state remains unchanged for the duration of the outage
|
||||
- Confirm no local-only apply code path exists by inspection of the receipt handler from 4.1 (no runtime check needed — the guarantee is structural)
|
||||
- Observable: the test demonstrates zero state mutation during a simulated quorum-loss window
|
||||
|
|
@ -193,6 +193,7 @@
|
|||
- **1.2**: implemented `commit_resolve()` + wired it into all 6 commit.c-internal paths that can advance `committed_lsn`/`writable_lsn` (the COMMITTED/majority-ack outcome). Chose `VTASK_LIVE_BLOCK` for dispatch (never admission-gated, unlike `VTASK_REQ_BLOCK` which can fast-reject — unacceptable for "exactly-once, never unresolved"). **Reviewer caught a real task-graph gap, not a defect in 1.2 itself**: converting `COMMIT_MODE_WAIT` away from a blocking wait also removes the *only* mechanism that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline) — nothing yet arms an equivalent timeout for async-registered continuations, and nothing yet wires demotion/log-truncation detection to call `commit_resolve(FAILED, ...)`. Without this, a pending async command during quorum loss would hang forever instead of failing — worse than today's behavior. Split out as new tasks **1.3** (timeout arming) and **1.4** (demotion/truncation wiring), inserted before the renumbered **1.5** (was 1.3, concurrency test — now exercises the real detection paths from 1.3/1.4 instead of calling `commit_resolve()` directly). Task 3.2 now also depends on 1.3/1.4, since its own "failure/timeout outcome replies" observable is meaningless without them.
|
||||
- **1.3**: added a dedicated per-commit-engine sweep thread (not the shared `delaymgr` — would violate boundary + delaymgr's 200ms poll granularity vs. `DEFAULT_SYNC_TIMEOUT_MS`=500ms would add timing jitter). Race rule: majority-satisfied always wins over an already-expired deadline when both are true under the same lock hold. **Round 1 REJECTED**: `commit_sweep_once()` computed the next-deadline snapshot under `ce->lock`, unlocked to dispatch, then slept on the *stale* snapshot — a `commit_submit_async()` registration/broadcast landing during that unlocked dispatch window was lost (no thread parked on the condvar to receive it), so the sweep thread could sleep on an absent/late deadline indefinitely (worst case: unbounded delay in a low-traffic quorum-loss scenario, violating the "identical failure timing" no-regression bar). **Fixed** by holding `ce->lock` continuously across scan→wait decisions (releasing only for the unlocked dispatch step) and always re-scanning under the lock immediately before ever calling `cond_wait`/`cond_timedwait` — the standard recheck-then-wait idiom. Round 2 reviewer independently reproduced the round-1 bug shape to confirm the fix actually closes it, not just narrows it. **Precedent for 1.4**: any future code that needs to interrupt the sweep thread's wait (e.g., a demotion/truncation handler resolving a pending LSN out-of-band) must go through the same lock+broadcast discipline `commit_submit_async()` uses, or it risks the identical lost-wakeup class of bug.
|
||||
- **1.4**: added `commit_resolve_pending_above(ce, lsn_floor, rc)` (snapshot-then-`commit_resolve()`-per-entry, no parallel bookkeeping). Truncation: a single new `on_truncate`/`truncate_cb_ctx` callback slot on `persist_t` (same inversion-of-control shape as the existing `on_durable_advance` hook) fires from `persist_truncate_wal_above` after releasing `p->lock`, only when a truncation actually removed entries — this ONE wrapper covers all three real truncation call sites (`raft_cb_become_leader`'s post-election truncate, the library's `on_truncate_from` conflict-reconcile, `ha_demotion_convergence_fn`'s post-demotion truncate), verified by the reviewer reading all three. Demotion: `raft_cb_become_follower` calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` (lsn_floor=0 → fails everything — a demoted node can never authoritatively confirm anything pending; verified against the CP posture). **Known residual, non-blocking**: a fourth internal truncation path (`persist.c` inside `persist_wal_write_replicated`'s tip-conflict-repair branch, STANDBY replicated-apply only) calls `truncate_wal_above_locked()` directly, bypassing the new wrapper — safe today only because demotion resolves ALL pending continuations before role flips to STANDBY (so nothing can still be registered by the time that path runs), but this is an *implicit* ordering invariant, not an asserted one. Flag for whoever next touches the STANDBY replicated-apply path: add a defensive comment/assertion there, or route it through the wrapper too, before relying on it further. Also: bug-1407 was believed closed at this point (70/70 across 3 runs) — **see 1.5's note below, this was a false reading, not a real fix**.
|
||||
- **4.2**: pure test-only close-out of task-group 4 — one new test `test_fact_state_unchanged_throughout_quorum_loss_outage` in `test_commit_barrier_reply.c` (30→31), approved first round, zero production changes. Drives a genuine `repl_remove_standby`-based quorum-loss window through all 4 real fact-receipt handlers, polls live job state + `committed_lsn` at 5 time-spread points (real 20ms sleeps, ~100ms total) rather than once, asserts a real hook-spy never sees `AFTER_EVENT` fire, and confirms durable WAL presence via direct on-disk segment read (`cbrp_count_jobstate_entries`) — combining and strengthening two pre-existing tests that each covered only half (4.1's own single-point-in-time check under normal conditions; an older pre-4.1 test that proved WAL durability but never inspected live state). **Static/inspection half of Req 6.2 confirmed independently by the reviewer, re-reading 4.1's landed diff**: none of the four `record_*_fact` functions call `apply_committed_entry`/`restore_job`/mutate the real job object — `record_job_fail_fact` specifically operates on a throwaway `job_t scratch = *job` shallow copy. **Architecturally significant finding for task 6 planning**: facts have NO commit-timeout/failure concept at the receipt layer, unlike commands (task 3.2's `commit_submit_async`). `persist_submit_job_fact()` → `wal_hook_append_classified(WAL_MODE_RETAIN)` → `commit_submit(..., COMMIT_MODE_RETAIN, 0)`, and `COMMIT_MODE_RETAIN`'s branch in commit.c is an immediate, unconditional return with NO continuation registered (explicit comment: "RETAIN never honors the timeout") — `commit_resolve()` never fires for a retained fact. A fact's outage is therefore open-ended by construction; its eventual commit is discovered only when `commit_advance_locked()`/`commit_on_ack` notices `committed_lsn` can advance past it via the ACK-registry watermarks, purely. **Task 6's apply pump must rely entirely on `committed_lsn` advancement to discover newly-eligible retained facts — there is no per-fact signal to wait on.** Confirmed neither of 4.1's two flagged hazards (double-append, merge-staleness) is triggered by this test, since it never drives apply. **Task-group 4 (fact apply-after-commit split) is now complete**: 4.1 (receipt converted to durable-append-only, two critical hazards discovered and documented for task 6) → 4.2 (retention proven under sustained quorum loss, fact-vs-command architectural asymmetry documented). **Both task-groups 1-4 are now done; task 6 (ordered apply pump) is the critical next milestone that must resolve, before going live: the double-append hazard, the merge-staleness hazard, AND design around facts' lack of a per-entry commit signal (commit_advance/committed_lsn-driven discovery only).** Groups 5 (restart verification) and 6 (apply pump) remain; 5.1 depends only on 2.3 (already done) so could run in parallel with 6, per tasks.md's dependency graph — continuing in numeric order (5 next) unless reprioritized.
|
||||
- **4.1**: converted the four worker-fact receipt handlers (`on_job_start`, `on_job_finish`, `on_job_fail`, `on_job_postdone` — NOT literally named `on_job_update`, task text's naming was approximate) to validate-and-durably-append-only: BEFORE_EVENT validation hook unchanged, AFTER_EVENT (post-mutation) hook no longer fires at receipt, live state mutation removed. **Key discovery**: the durable WAL append was NOT a separate call — it lived INSIDE the very `persist_job_hook_fun` AFTER_EVENT subscriber this task needed to silence, so a new `persist_submit_job_fact()` (persist.c) was added that calls the same unmodified `wal_hook_append_classified(WAL_MODE_RETAIN, wal_jobstate_t{jobid,state}, ...)` directly — verified wire-format-identical to before, `apply_committed_entry()`'s existing switch (task 2.2) routes it unchanged. `mark_job_start`/`mark_job_finish` (single caller each, verified via diff against 41d3620) renamed in place to `record_job_start_fact`/`record_job_finish_fact`; `mark_job_fail`/`mark_job_update` (shared with `job_ctrl`'s OUT-OF-BOUNDARY client-command path, 2 and 3 additional call sites) left untouched, with new durable-append-only siblings `record_job_fail_fact`/`record_job_update_fact` added instead — `job_ctrl`/`do_jobcontrol_op` confirmed byte-identical to pre-task by the reviewer. Approved first round. **Real, permanent regression accepted, not fixed (Req 5.1 required the removal)**: `on_job_postdone`'s `finish_time = time(NULL)` anomaly backstop is gone with no wire-format-level equivalent — concretely, a job hitting this rare no-prior-exit-status path now has `finish_time` stuck at 0 forever, which `job_should_retain()` (persist.c) evaluates as immediately GC-eligible instead of retained for the configured window. Narrow, rare-path, but real. **TWO CRITICAL HAZARDS DISCOVERED BY REVIEW, MUST BE RESOLVED BEFORE TASK 6 WIRES apply_committed_entry() INTO A LIVE PATH — currently inert only because replay runs before persist_start() registers hooks (cluster.c: persist_recover() at line ~1451 vs persist_start() at ~1694)**:
|
||||
1. **Latent double-append**: `persist_job_hook_fun` is STILL registered on `(XCTL_HOOK_JOB, HOOK_AFTER_EVENT)` with no detail filter for `HOOK_DETAIL_JOB_UPDATE`, and does NOT early-return for that detail (it only early-returns for the WAIT-mode submit path). The moment task 6 wires `apply_committed_entry()` into live commit-advance, EVERY fact `apply_committed_entry()` applies will fire `HOOK_DETAIL_JOB_UPDATE`, which `persist_job_hook_fun` will catch and append AGAIN — a duplicate, mis-typed WAL entry per live-applied fact. **Task 6.1/6.2 must retire or detail-filter `persist_job_hook_fun`'s fact-handling branch before going live**, since no live receipt path fires AFTER_EVENT for facts anymore (this task removed the only other source).
|
||||
2. **Merge-staleness on deferred apply**: `record_job_fail_fact`/`record_job_update_fact` precompute a FINAL target `state` at receipt time from a snapshot of current job state and bake it verbatim into the WAL entry (their own doc comments claim this "can never drift from what apply time will eventually compute" — that claim was TRUE only when receipt and apply were the same event; it becomes FALSE the moment they're separated by a commit round-trip, which is this entire feature's purpose). `apply_committed_entry()` routes all job-state facts through the generic `apply_job_update_transition`, which does a blind `job->job.state = state` overwrite — not the type-specific `|=` merge `apply_job_fail_transition` uses. Once two facts for the same job are in flight simultaneously (now newly possible, since apply is deferred), the second one to apply can silently clobber bits the first one set. Root cause is task 2.2's wire-format/generic-routing reality, not this task's diff — but no task in the current plan (4.2, 5.x, 6.x) tests or fixes this. **Task 6 planning must resolve this before wiring live apply** — likely requires either wire-format enrichment (baking the merge operation itself into the WAL entry, not just the precomputed result) or an apply-time re-derivation strategy.
|
||||
|
|
|
|||
|
|
@ -2638,6 +2638,202 @@ static void test_receipt_facts_durably_append_without_live_mutation(void)
|
|||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* Task 4.2 (Req 6.1, 6.2): durable retention of facts submitted while
|
||||
* majority is unreachable, held for the SUSTAINED DURATION of the outage —
|
||||
* not just a single immediate check.
|
||||
*
|
||||
* This closes a gap neither existing test covers on its own:
|
||||
* - test_receipt_facts_durably_append_without_live_mutation (task 4.1,
|
||||
* above) proves live-state-unchanged for all four fact types, but
|
||||
* under ordinary (majority-reachable) conditions, and only checks
|
||||
* state once, immediately after each receipt call returns.
|
||||
* - test_fact_retention_across_quorum_loss (pre-existing, from
|
||||
* ha-commit-barrier-rollout task 4.3, further above) drives a genuine
|
||||
* quorum-loss window and proves WAL-level durability/no-duplication,
|
||||
* but never inspects live job state at all — it predates task 4.1's
|
||||
* receipt/apply split, when receipt-time mutation was still the norm
|
||||
* for the behavior it was pinning.
|
||||
*
|
||||
* This test combines both: a genuine simulated quorum-loss window (same
|
||||
* repl_remove_standby mechanism as test_fact_retention_across_quorum_loss)
|
||||
* driving all four real fact-receipt handlers from task 4.1, with the
|
||||
* job's live in-memory state polled at MULTIPLE points spread across the
|
||||
* outage — not merely before/after a single call — to prove the guarantee
|
||||
* holds for the outage's entire duration, sealed by one final check after
|
||||
* the last poll.
|
||||
*
|
||||
* Per persist.c's wal_hook_append_classified (task 4.1's persist_submit_
|
||||
* job_fact routes through it): COMMIT_MODE_RETAIN "never honors the
|
||||
* timeout" and commit_submit() returns 0 immediately for RETAIN entries —
|
||||
* facts have NO commit-timeout/failure concept at the receipt layer,
|
||||
* unlike commands (task 3.2's commit_submit_async). A fact's entry simply
|
||||
* sits durably appended, uncommitted, for as long as majority stays
|
||||
* unreachable; nothing ever resolves it as FAILED the way a command's
|
||||
* registered continuation would. This test's "outage" is therefore
|
||||
* genuinely open-ended by construction (committed_lsn structurally cannot
|
||||
* advance past it while both standbys are disconnected), not a race
|
||||
* against a timeout that might independently fire.
|
||||
*/
|
||||
static void test_fact_state_unchanged_throughout_quorum_loss_outage(void)
|
||||
{
|
||||
static unsigned int qo_pids[1] = { 8501 };
|
||||
job_pidinfo_t start;
|
||||
xce_jobfinish_t fin;
|
||||
job_failinfo_t failinfo;
|
||||
uint64_t lsn_before, lsn_after;
|
||||
uint64_t committed_at_poll;
|
||||
int initial_state, poll_state, final_state;
|
||||
uint32_t st_start = 0, st_fin = 0, st_fail = 0, st_upd = 0;
|
||||
int cnt_start, cnt_fin, cnt_fail, cnt_upd;
|
||||
int i;
|
||||
enum { OUTAGE_POLL_COUNT = 5, OUTAGE_POLL_INTERVAL_MS = 20 };
|
||||
|
||||
cbrp_setup(5000);
|
||||
memset(&g_job_submit, 0, sizeof(g_job_submit));
|
||||
cbrp_make_job(85, &g_job_submit);
|
||||
|
||||
initial_state = cbrp_job_live_state(85);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_PEND, initial_state,
|
||||
"precondition: a freshly created job starts PEND");
|
||||
|
||||
g_cbrp_before_fires = 0;
|
||||
g_cbrp_after_fires = 0;
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_add_hook(&g_hc.hooksys, XCE_CONTROLLER_HOOK_JOB, cbrp_job_hook_spy,
|
||||
NULL));
|
||||
|
||||
/* Act 1 — QUORUM LOST: disconnect both standby registry slots
|
||||
* (repl_remove_standby mirrors the RDHUP-disconnect registry effect,
|
||||
* same mechanism test_fact_retention_across_quorum_loss uses). n_peers
|
||||
* stays 3 in the engine: self alone is 1 of 3, never a majority, so
|
||||
* committed_lsn structurally cannot advance past this point for as
|
||||
* long as both standbys remain disconnected. */
|
||||
repl_remove_standby(&g_hc.persist.repl, &g_hep_a);
|
||||
repl_remove_standby(&g_hc.persist.repl, &g_hep_b);
|
||||
|
||||
pthread_mutex_lock(&g_hc.persist.lock);
|
||||
lsn_before = g_hc.persist.current_lsn;
|
||||
pthread_mutex_unlock(&g_hc.persist.lock);
|
||||
|
||||
/* Act 2 — submit all four fact types for the SAME job while majority
|
||||
* is unreachable, through the real receipt handlers from task 4.1
|
||||
* (identical call sequence to test_receipt_facts_durably_append_
|
||||
* without_live_mutation above, now under a genuine quorum-loss
|
||||
* window instead of ordinary conditions). */
|
||||
memset(&start, 0, sizeof(start));
|
||||
start.jobid = 85;
|
||||
start.npids = 1;
|
||||
start.pids = &qo_pids[0];
|
||||
CBRP_DRIVE_FACT(on_job_start(&g_hc, &start), "on_job_start(85)");
|
||||
|
||||
memset(&fin, 0, sizeof(fin));
|
||||
fin.jobid = 85;
|
||||
fin.exit_code = 0; /* -> JOB_STAT_DONE */
|
||||
CBRP_DRIVE_FACT(on_job_finish(&g_hc, &fin), "on_job_finish(85)");
|
||||
|
||||
memset(&failinfo, 0, sizeof(failinfo));
|
||||
failinfo.jobid = 85;
|
||||
failinfo.stage = JOB_EXEC_STAGE_POST_EXEC;
|
||||
failinfo.code = 1;
|
||||
failinfo.detail = (char *)"synthetic fail during simulated quorum loss";
|
||||
CBRP_DRIVE_FACT(on_job_fail(&g_hc, &failinfo), "on_job_fail(85)");
|
||||
|
||||
CBRP_DRIVE_FACT(on_job_postdone(&g_hc, &g_req_ep, 85),
|
||||
"on_job_postdone(85)");
|
||||
|
||||
pthread_mutex_lock(&g_hc.persist.lock);
|
||||
lsn_after = g_hc.persist.current_lsn;
|
||||
pthread_mutex_unlock(&g_hc.persist.lock);
|
||||
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before + 4, lsn_after,
|
||||
"each of the four facts must append exactly one WAL entry while "
|
||||
"majority is unreachable (Req 6.1)");
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
atomic_load(&g_hc.persist.committed_lsn) < lsn_before,
|
||||
"no fact entry may commit while majority is unreachable (Req 6.1)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
|
||||
"AFTER_EVENT must never fire for an uncommitted fact — no "
|
||||
"reaction (slot free, dispatch, notify) may escape the outage "
|
||||
"(Req 6.1, 6.2)");
|
||||
|
||||
/* Act 3 — the core observable this task adds: poll live job state AND
|
||||
* the commit watermark repeatedly across a simulated outage window,
|
||||
* not just once immediately after receipt. Every single poll must see
|
||||
* the job unchanged and the outage still genuinely in effect — proving
|
||||
* the no-mutation guarantee holds for the outage's sustained duration,
|
||||
* not merely at one instant. */
|
||||
for (i = 0; i < OUTAGE_POLL_COUNT; i++) {
|
||||
cbrp_sleep_ms(OUTAGE_POLL_INTERVAL_MS);
|
||||
|
||||
poll_state = cbrp_job_live_state(85);
|
||||
committed_at_poll = atomic_load(&g_hc.persist.committed_lsn);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(initial_state, poll_state,
|
||||
"job state must remain unchanged at every point during the "
|
||||
"quorum-loss outage (Req 6.1, 6.2) — there is no local-only "
|
||||
"apply path that could mutate it while majority is "
|
||||
"unreachable");
|
||||
TEST_ASSERT_TRUE_MESSAGE(committed_at_poll < lsn_before,
|
||||
"the outage must still be genuinely in effect at this poll — "
|
||||
"committed_lsn must not have advanced while both standbys "
|
||||
"remain disconnected");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
|
||||
"AFTER_EVENT must still not have fired at this poll point — "
|
||||
"no reaction may fire mid-outage (Req 6.1, 6.2)");
|
||||
}
|
||||
|
||||
/* Assert — nothing dropped, nothing duplicated: all four facts are on
|
||||
* disk exactly once with the correct post-transition state, confirmed
|
||||
* AFTER the full polled outage window (not merely right after Act 2). */
|
||||
cnt_start = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 85,
|
||||
&st_start);
|
||||
cnt_fin = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 85,
|
||||
&st_fin);
|
||||
cnt_fail = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 85,
|
||||
&st_fail);
|
||||
cnt_upd = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 85,
|
||||
&st_upd);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_start,
|
||||
"job 85 START fact must remain durably retained exactly once "
|
||||
"throughout the whole outage window (Req 6.1)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_fin,
|
||||
"job 85 FINISH fact must remain durably retained exactly once "
|
||||
"throughout the whole outage window (Req 6.1)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_fail,
|
||||
"job 85 FAIL fact must remain durably retained exactly once "
|
||||
"throughout the whole outage window (Req 6.1)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, cnt_upd,
|
||||
"job 85 UPDATE fact must remain durably retained exactly once "
|
||||
"throughout the whole outage window (Req 6.1)");
|
||||
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_start);
|
||||
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_fin);
|
||||
TEST_ASSERT_TRUE_MESSAGE((st_upd & JOB_STAT_PDONE) != 0,
|
||||
"the recorded UPDATE fact must carry JOB_STAT_PDONE");
|
||||
TEST_ASSERT_TRUE_MESSAGE((st_upd & JOB_STAT_EXIT) != 0,
|
||||
"the recorded UPDATE fact must carry the backstop JOB_STAT_EXIT "
|
||||
"bit postdone computes when no prior exit status was set");
|
||||
(void)st_fail; /* FAIL's merge result depends on current state at the
|
||||
* (never-reached) apply step; only its durable
|
||||
* retention count is this test's concern. */
|
||||
|
||||
/* Final check: after the ENTIRE outage window, job state is STILL
|
||||
* unchanged and no reaction has fired — sealing the "for the duration
|
||||
* of the outage" observable end-to-end. */
|
||||
final_state = cbrp_job_live_state(85);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(initial_state, final_state,
|
||||
"job state must still be unchanged at the end of the outage "
|
||||
"window (Req 6.1, 6.2)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
|
||||
"AFTER_EVENT must never have fired across the entire outage "
|
||||
"window (Req 6.1, 6.2)");
|
||||
|
||||
cbrp_teardown();
|
||||
printf("test_fact_state_unchanged_throughout_quorum_loss_outage: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ── Test group entry point ──────────────────────────────────────────── */
|
||||
|
||||
void test_commit_barrier_reply(void)
|
||||
|
|
@ -2695,5 +2891,8 @@ void test_commit_barrier_reply(void)
|
|||
RUN_TEST(test_guard_fail_after_finish_preserves_perr_merge);
|
||||
/* Fact receipt handlers durably append only (task 4.1, Req 5.1) */
|
||||
RUN_TEST(test_receipt_facts_durably_append_without_live_mutation);
|
||||
/* Durable retention held for the outage's sustained duration under
|
||||
* genuine quorum loss (task 4.2, Req 6.1, 6.2) */
|
||||
RUN_TEST(test_fact_state_unchanged_throughout_quorum_loss_outage);
|
||||
UNITY_END();
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue