fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact traffic against a live 3-node cluster, killing the primary mid-stream and asserting zero committed-state divergence and zero phantom fact reactions. Building this scenario surfaced bug-1416: persist_recover()'s boot-time apply-correctness verification can race the live apply pump and the standby live-replication apply path, both of which mutate the same last_applied_lsn/ job-table state the verification re-reads live -- producing a false APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with a new persist_t.recovery_complete gate that defers only the in-memory apply (never the durability write or replication ACK) while a recovery pass is in flight, closed from the earliest safe point in myapp_run() so an election- retry candidate can't apply live traffic before persist_recover() ever runs. Round-2 adversarial review independently verified the ACK/durability paths stay ungated, reproduced the myapp.c fix as load-bearing via revert-and- reproduce, and confirmed real-divergence detection still fires. Live 3-node factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
parent
5f98d1b0ef
commit
a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
|
|
@ -12388,5 +12388,179 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-03T21:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1409",
|
||||
"timestamp": "2026-07-04T17:00:00.000Z",
|
||||
"error_message": "test_demotion_convergence hung/spun (18.5M+ log lines, timeout kill) once commit.c's commit_resolve_pending unconditionally called the new persist_apply_pump_kick",
|
||||
"file": "src/controller/persist.c",
|
||||
"root_cause": "apply_pump_drain()'s recheck-then-clear retry loop ignored persist_apply_pump_run()'s return code: on a genuine read failure (e.g. committed_lsn advanced in a test fixture with no real WAL bytes on disk for that LSN, rc=-ENOENT), last_applied_lsn never advances, so the very next re-check inside the SAME drain loop sees identical unresolved work and immediately reclaims the scheduled flag and retries forever -- a tight busy loop with no backoff, hammering disk reads and the logger.",
|
||||
"fix": "apply_pump_drain() now checks persist_apply_pump_run()'s rc immediately after clearing apply_pump_scheduled; a negative rc returns immediately instead of re-checking for more work and reclaiming the flag. The failure is already logged loudly inside persist_apply_pump_run itself. A future, genuinely NEW kick (a later real commit-advance event) retries then, at that event's natural cadence, never in a busy spin.",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.1",
|
||||
"apply-pump",
|
||||
"busy-loop",
|
||||
"infinite-loop",
|
||||
"commit-resolve-pending",
|
||||
"regression-sweep"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T17:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1410",
|
||||
"timestamp": "2026-07-04T17:00:00.000Z",
|
||||
"error_message": "test_apply_pump's spy hook recorded jobid=0 for HOOK_DETAIL_JOB_UPDATE/HOOK_DETAIL_JOB_NEW instead of the real jobid",
|
||||
"file": "src/controller/testsuite/test_apply_pump.c",
|
||||
"root_cause": "The hook's per-subscriber `data`/cls pointer (bound at xce_add_hook registration time) is separate from the va_list's own variadic arguments. apply_committed_entry's notify calls pass (cluster_t*, payload) as the actual varargs -- the spy hook read the FIRST va_arg as the payload pointer directly, skipping the leading cluster_t* argument every real dispatch call (job.c, persist.c) always sends first.",
|
||||
"fix": "Added `(void)va_arg(ap, cluster_t*);` before extracting the real payload (xlist_t*/job_t*) in both HOOK_DETAIL_JOB_NEW and HOOK_DETAIL_JOB_UPDATE cases, mirroring persist_job_hook_fun's own established va_arg consumption order.",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.1",
|
||||
"test-only",
|
||||
"va_arg",
|
||||
"hook-system"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T17:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1411",
|
||||
"timestamp": "2026-07-04T17:00:00.000Z",
|
||||
"error_message": "AddressSanitizer global-buffer-overflow in dataq_enq when test_apply_pump.c called persist_submit_job_fact with a real 3-member repl setup",
|
||||
"file": "src/controller/testsuite/test_apply_pump.c",
|
||||
"root_cause": "repl_broadcast tried to enqueue onto the dummy standby `int` stand-ins (g_tap_hep_a/b, opaque identity handles only) as if they were real endpoint_t* objects, overflowing past the 4-byte int into adjacent globals -- the fixture omitted the repl_broadcast_send_hook interception test_commit_barrier_reply.c's cbrp_setup already uses to keep repl_broadcast from ever dereferencing those dummy pointers.",
|
||||
"fix": "Added the identical repl_broadcast_send_hook = tap_drop_send_hook (a no-op stub returning success) in tap_setup/tap_teardown, mirroring test_commit_barrier_reply.c's cbrp_drop_send_hook exactly.",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.1",
|
||||
"test-only",
|
||||
"asan",
|
||||
"repl-broadcast",
|
||||
"fixture"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T17:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1412",
|
||||
"timestamp": "2026-07-04T17:00:00.000Z",
|
||||
"error_message": "test_apply_pump.c failed to compile: 'unknown type name RETAIN', stray bytes, __gnuc_va_list errors cascading from an unrelated location",
|
||||
"file": "src/controller/testsuite/test_apply_pump.c",
|
||||
"root_cause": "A doc comment contained the literal substring 'WAL_JOB_*/RETAIN' -- the '*' immediately followed by '/' forms a real C block-comment terminator, closing the file's top doc comment many lines early and turning the rest of the comment text into stray code tokens.",
|
||||
"fix": "Reworded to 'a WAL_JOB_SUBMIT WAIT-mode entry mixed with WAL_JOB_{START,FINISH} RETAIN-mode facts', removing the accidental */ sequence. Lesson: never write '<TOKEN>*/<TOKEN>' inside a C block comment, even when '*' is meant as a wildcard/glob character.",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.1",
|
||||
"test-only",
|
||||
"comment-terminator",
|
||||
"build-break"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T17:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1413",
|
||||
"timestamp": "2026-07-04T17:00:00.000Z",
|
||||
"error_message": "test_apply_pump's burst-ordering test hung the NEXT test (longjmp-past-teardown) after a wrong off-by-one assertion (expected lsn_before+3, actual watermark is lsn_before+2 for 3 appended entries) failed",
|
||||
"file": "src/controller/testsuite/test_apply_pump.c",
|
||||
"root_cause": "A TEST_ASSERT failure mid-test longjmps past the rest of the function body, including the explicit tap_teardown() call at the end -- leaving the fsync thread/tmpdir/hooksys from the failed test alive when the next test's tap_setup() ran, hanging it. Root trigger was a genuine off-by-one in the test's own math (current_lsn before N appends is the FIRST assigned lsn, not one less than it; the watermark after N appends is first_lsn + N - 1, not first_lsn + N).",
|
||||
"fix": "Fixed the off-by-one (first_lsn + 2 == watermark for 3 appends starting at first_lsn). Restructured all three tests in the file to capture every assertion-relevant observable into local variables BEFORE any TEST_ASSERT call, and to call tap_teardown() unconditionally before running any assertion -- the exact pattern already established in this codebase for task 3.1's identical hazard (tasks.md Implementation Notes, task 3.1 round 2/3).",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.1",
|
||||
"test-only",
|
||||
"longjmp-past-teardown",
|
||||
"off-by-one",
|
||||
"test-hygiene"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1408"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T17:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1414",
|
||||
"timestamp": "2026-07-04T18:00:00.000Z",
|
||||
"error_message": "raft-apply-decoupling task 6.1 round 1 REJECTED: FINISH (LSN N) then a POST_EXEC-stage FAIL (LSN N+1) for the same job, with FAIL received before FINISH applies, corrupts the job to RUN|EXIT|PERR (0x106) instead of the correct DONE|PERR (0x108) once both apply in LSN order -- DONE is silently lost and a stray RUN bit appears.",
|
||||
"file": "src/controller/job.c, src/controller/persist.c, src/include/wal_types.h, src/include/persist.h, src/controller/include/cluster.h",
|
||||
"root_cause": "record_job_fail_fact (job.c) and record_job_update_fact (postdone) precomputed a FINAL absolute state at RECEIPT time (scratch-copy merge for FAIL, inline backstop+PDONE merge for postdone) and baked it into wal_jobstate_t.state -- correct only when receipt and apply are the same event. Once task 6.1 deferred apply past the commit barrier, a second in-flight fact for the same job (e.g. FINISH) could apply BETWEEN this fact's receipt and its own apply, making the receipt-time snapshot stale by the time apply_committed_entry's generic apply_job_update_transition() did its blind absolute overwrite -- silently clobbering whatever the other fact had just applied.",
|
||||
"fix": "Repurposed wal_jobstate_t._pad (previously always-zero, unused) as a discriminator: _pad==0 still means legacy absolute state (job_ctrl's mark_job_fail/mark_job_update live paths, which mutate+capture under one lock atomically -- no staleness risk, unchanged); a nonzero _pad (WAL_JOB_FAIL_STAGE_FLAG|stage for FAIL, WAL_JOB_UPDATE_POSTDONE_MARK for postdone) marks the entry for apply-time recomputation. apply_committed_entry (persist.c) now dispatches WAL_JOB_FAIL/WAL_JOB_UPDATE on this bit, calling the REAL apply_job_fail_transition() / new apply_job_postdone_transition() (job.c) against the job's ACTUAL current state at apply time instead of a stale baked value. New persist_submit_job_fact_ex(p,type,jobid,state,pad) primitive added; persist_submit_job_fact becomes a pad=0 thin wrapper so zero existing call sites needed changes. Also fixed 2 pre-existing test_commit_barrier_reply.c tests that asserted the old wire format directly (cbrp_count_jobstate_entries reading state bits) -- extended the helper to a _ex variant exposing _pad, updated assertions to check the new marker plus a direct call to the guard core on a scratch job_t. New regression test test_apply_pump_fail_racing_finish_merges_against_current_state in test_apply_pump.c drives FINISH then FAIL through the real receipt handlers with both entries durably appended but unapplied, then a single covering ACK; RED-confirmed byte-for-byte (Expected 264 Was 262) against the reverted round-1 code, GREEN after the fix.",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.1",
|
||||
"round-2",
|
||||
"job-state-corruption",
|
||||
"merge-staleness",
|
||||
"deferred-apply",
|
||||
"wal-wire-format",
|
||||
"wal_jobstate_t",
|
||||
"_pad-discriminator"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1413"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T18:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1415",
|
||||
"timestamp": "2026-07-04T19:30:00.000Z",
|
||||
"error_message": "raft-apply-decoupling task 6.2 review finding (NOT reproduced under the test suite's pool-less fixtures, NOT fixed): a real vtaskmgr pool (n_vcpus > 1) can run the commit continuation's dispatch (job_commit_resolve_adapter -> jobsubmit_commit_continuation -> persist_register_pending_apply) and the apply pump's kick (persist_apply_pump_kick -> apply_pump_drain -> persist_apply_pump_run) on two DIFFERENT worker threads with no completion-order guarantee, even though commit_resolve_pending() enqueues them in that order. If the apply-pump task happens to finish first, persist_take_pending_apply() finds nothing registered yet for that lsn (0, correct no-op per its own contract), applies the entry, and advances last_applied_lsn past it -- then when the continuation task finally runs and calls persist_register_pending_apply() for that now-already-applied lsn, the registration is permanently orphaned: the walk only ever revisits [last_applied_lsn+1..committed_lsn], so nothing will ever call persist_take_pending_apply() for that lsn again. Net effect: the job is correctly constructed and visible in c->jobmgr, but the client that submitted it never receives a reply -- silently, forever.",
|
||||
"file": "src/controller/commit.c (commit_resolve_pending, commit_resolve/dispatch_continuation), src/controller/persist.c (persist_apply_pump_kick, persist_apply_pump_run)",
|
||||
"root_cause": "commit_resolve_pending() (commit.c ~793-835, landed by task 1.2/6.1) resolves every ready continuation via commit_resolve() -> dispatch_continuation(), which for a real pool (ce->pool != NULL) enqueues the continuation onto vtaskmgr via vtask_enq_classified(VTASK_LIVE_BLOCK) and returns WITHOUT waiting for it to run; commit_resolve_pending() then calls persist_apply_pump_kick(ce->p, ce->pool), which enqueues a SECOND, independent VTASK_LIVE_BLOCK task onto the SAME pool. vtaskmgr.c's dispatcher (vdispatcher_entry) hands queued tasks to whichever vcpu worker thread is next idle (pick_idle_vcpu/pick_eligible_task) -- with n_vcpus > 1 (the only realistic production configuration; every test fixture in this spec deliberately never calls commit_set_pool, so ce->pool stays NULL and both dispatches run synchronously inline instead, which is why no test in this suite reproduces the race), two tasks enqueued back-to-back can be picked up by two different idle vcpus at nearly the same time, and their COMPLETION order depends on each task's own runtime (a disk-reading WAL-scan-and-apply walk vs. a calloc+mutex-guarded list insert) -- not their enqueue order. This is a pre-existing structural gap from task 1.2 (async continuation dispatch) + task 6.1 (apply-pump-kick wired into the same choke point, same pool) that had no observable consequence until task 6.2 added a real consumer (persist_take_pending_apply) whose registration can now race the very entry it targets.",
|
||||
"fix": "FIXED round 2 (task 6.2): persist_register_pending_apply() (persist.c) now self-heals option (b) from this entry's own proposed fixes -- before linking a new node onto pending_applies, it checks `lsn <= persist_last_applied_lsn(p)`; if the apply pump's walk has already applied `lsn` (last_applied_lsn already at or past it), `on_applied(ctx, lsn)` fires immediately and synchronously on the CALLING thread instead of being linked for a walk pass that will never revisit that lsn. Race-safe in both interleavings: when the walk has not reached `lsn` yet, the check is false and registration proceeds exactly as before (unchanged behavior, verified against the full existing regression suite); when the walk already passed `lsn`, its own persist_take_pending_apply() call for that lsn already ran and found nothing (that is precisely how we know registration is late), so the immediate inline fire can never double-fire alongside it. New deterministic regression test `test_apply_pump_late_pending_apply_registration_self_heals` (test_apply_pump.c) wires a REAL multi-vcpu vtaskmgr_t (n_vcpus=4) via commit_set_pool -- the one test in this spec to do so -- and forces the exact bad interleaving via a barrier (the test's stand-in commit continuation polls persist_last_applied_lsn() until the apply pump has already applied the lsn, THEN registers), rather than relying on natural timing luck. Verified genuine RED (fails with 'Expected 1 Was 0' against the unfixed check, confirming the hang is real and reproducible) -> GREEN (fires exactly once, resource released exactly once, no orphaned registration) after the fix. Full regression sweep (test_apply_pump, test_commit_barrier_reply, test_apply_reapply_idempotency, test_commit_resolve, test_commit_admission_pool, test_demotion_convergence) shows zero new failures -- the only 2 failures present (test_jobsubmit_returns_before_commit_and_registers_pending_apply, test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements, both in test_commit_barrier_reply.c) are pre-existing/expected per task 6.1's own Implementation Notes (their assertion text predates task 6's apply pump landing; task 7.3's job to update), confirmed identical before and after this fix by reverting to the pre-round-2 baseline and re-running.",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-6.2",
|
||||
"vtaskmgr",
|
||||
"race-condition",
|
||||
"orphaned-registration",
|
||||
"lost-reply",
|
||||
"commit_resolve_pending",
|
||||
"persist_apply_pump_kick",
|
||||
"fixed",
|
||||
"self-healing"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1409"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-04T23:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1416",
|
||||
"timestamp": "2026-07-05T06:15:00.000Z",
|
||||
"error_message": "[xce-test-ha] FAIL: factstream round 1: cluster did not reconverge to 3-of-3 after restarting ctrl-1 -- root cause: persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=72 new_job_count=23); persist_recover_loud_fence_if_unrecoverable: UNRECOVERABLE, staying alive/non-promotable",
|
||||
"file": "src/controller/persist.c (persist_verify_apply_correctness / persist_recover, cluster.c cluster_init_phase2)",
|
||||
"root_cause": "Found live via the new raft-apply-decoupling task 7.2 kill-mid-fact-stream soak scenario (--factstream), reproduced 4/4 runs (ASAN debug build, paced-wave ASAN build, release build, and a cleanup-suppressed debug postmortem run) in this sandboxed environment. cluster_init_phase2() calls cluster_open_listener() (which starts accepting network I/O, including replicated/catch-up WAL traffic from an already-serving primary) BEFORE persist_recover() runs its boot-time apply-correctness verification (task 5.1, Req 8.1/8.2). Under a slow/contended boot (observed: persist_recover firing ~36s after process start instead of the normal ~5s, plausibly due to heavy concurrent nix-develop/test-process CPU contention in this sandbox), a standby's listener can already be live and processing real traffic for tens of seconds before its own persist_recover() call finally executes. persist_verify_apply_correctness() (persist.c ~3597) reads p->last_applied_lsn and live-queries count_jobmgr_entries(p->cluster) AFTER replay_wal completes -- both of these are also mutated by the LIVE apply pump (raft-apply-decoupling task 6.1, wired through commit_resolve_pending) reacting to replicated commit-advance events. If live traffic is already flowing and being applied through the live pump DURING the extended pre-persist_recover window, the live-read 'actual' values used by persist_verify_apply_correctness can reflect state beyond what persist_recover's own single WAL-scan pass computed as 'expected', producing a DIVERGENT verdict even though no real apply defect exists -- confirmed via ctrlstatus probes showing committedLsn identical (79) and no actual data loss across all 3 nodes; the fence engaged correctly per its own fail-loud design (stayed alive, non-promotable, kept retrying) but two of three nodes (including the node that went on to win the next election, term=5, first_boot=1) were fenced at the exact moment persist_recover ran. This matches task 5.1's own documented residual-risk note (tasks.md Implementation Notes, task 5.1 entry): 'the position check is diagnostically tautological for THIS SPECIFIC call site today... this will change once task 6's live apply pump gives expectation and actual state a real chance to diverge across threads/time' -- that predicted future divergence has now materialized under a slow-boot + concurrent-live-traffic combination that only a scenario driving traffic this early/aggressively after cluster boot (the factstream scenario) exercises; existing soak scenarios (chaotic/sequential/force-snapshot) submit far fewer jobs, more slowly, and do not reliably land traffic inside a slow node's pre-recovery window. REFINED after further reproduction (10/10 runs total, across ASAN/release builds and job counts from 8 to 20): the trigger is specifically the RESTART/catch-up path, not the initial multi-node cold boot. A standalone control test (3 nodes, no kill, 20s settle before any traffic) submitted jobs and drove jstart/jfinish/jfail cleanly with zero APPLY_DIVERGENT -- confirming the race needs a node that was DOWN while the cluster kept moving (i.e. an actual failover), not merely a fast cold boot: the returning node's persist_recover()/cluster_init_phase2 vtask must complete while it is ALSO receiving a compressed catch-up replication burst covering everything it missed while down, and that burst's volume is inherent to 'kill mid-fact-stream' (the whole point of the scenario) -- a run-ha-soak.sh-side pacing fix cannot eliminate the trigger the way it did for the (separately real but distinct) initial-boot instance the settle-delay mitigation addresses. Reproduced identically down to 8 fact-stream jobs with 2-job/1s-gap waves, so the trigger threshold is low. This plausibly explains why no prior soak scenario in this spec's history (chaotic/sequential/force-snapshot, all using NJOBS=3 by default, un-paced but small) ever surfaced it: none combine a real kill-during-traffic failover with this much job+fact volume until raft-apply-decoupling task 7.2's factstream scenario. Also independently added test.c's print_jobinfo state_raw hex field (task 7.2) and confirmed via a live probe that apply_job_fail_transition correctly applies EXIT|PERR bits even when job_state_str's RUN-checked-first priority ordering hides them in the pretty-printed state string (job showed 'state: RUN' / 'state_raw: 0x106' after a jstart+jfail-without-jfinish sequence) -- a separate, minor, likely pre-existing (not introduced by raft-apply-decoupling) display-only quirk in job_state_str, also not fixed here (out of boundary), but which the factstream phantom-check was updated to work around by reading state_raw instead of the ambiguous pretty string.",
|
||||
"fix": "FIXED (round 2, raft-apply-decoupling task 7.2 review remediation). Root cause confirmed exactly as hypothesized in the prior entry's candidate #2: persist_verify_apply_correctness() compared a WAL-scan-derived 'expected' snapshot against a LIVE re-read of last_applied_lsn/count_jobmgr_entries(), while TWO live mutation paths could write those same values concurrently -- the live apply pump (persist_apply_pump_run, driven by persist_apply_pump_kick off any of commit.c's 6 committed_lsn-advance sites) and the standby live-replication apply path (ha_apply_single_entry / ha_apply_term_entry_via_library in ha.c). Fixed via a new _Atomic bool persist_t.recovery_complete gate (persist.h/persist.c): persist_apply_pump_kick() now no-ops while the gate is closed; ha_apply_single_entry/ha_apply_term_entry_via_library now skip ONLY the in-memory apply_wal_entry() call while closed (durability write + replication ACK + applied_lsn advance are never gated, preserving the existing persist-before-apply-before-ACK contract and bug-1043's durability/materialization decoupling precedent). persist_recover() closes the gate at entry and reopens it (plus issues one deliberate persist_apply_pump_kick catch-up pass) right before returning, regardless of verdict -- covering any direct unit-test caller. ROUND 2 FINDING during live verification: this alone was insufficient -- a node stuck retrying its own election as a losing/timing-out candidate can receive and durably apply live-replicated entries via the ungated (persist_init default=true) path for tens of seconds BEFORE cluster_init_phase2() ever calls persist_recover() for the first time, reproducing the identical false-DIVERGENT symptom via an earlier entry point that persist_recover()'s own bracket cannot see. Closed by adding persist_mark_recovery_pending(&c.persist) in myapp_run() (myapp.c), called immediately after persist_init()/persist_start() succeed and BEFORE ha_start_election_async()/cluster_run() can process any network traffic -- closing the gate from the earliest safe moment in the real boot sequence. persist_init()'s default stays TRUE (ungated) so none of this spec's ~20 existing unit-test fixtures that construct a persist_t directly (bypassing myapp_run entirely) are affected; persist_mark_recovery_pending is never called outside myapp.c. Verified: live 3-node factstream soak (2 rounds, kill+restart both rounds) now PASSes clean with zero APPLY_DIVERGENT/UNRECOVERABLE across 5 persist_recover() calls (2 initial-boot delayed by the SAME pre-existing election-instability condition that originally exposed the round-2 gap, 2 post-kill restarts, all APPLY_CLEAN); broader controller_unit_test sweep (23 groups incl. test_apply_correctness_verification's 3 real-divergence tests, test_apply_pump/_integration, test_ha_core_safety, test_demotion_convergence) shows zero new failures (only the 2 pre-existing, documented task-6.1/6.2 test_commit_barrier_reply.c failures remain, task 7.3's job). Real-divergence detection independently re-confirmed intact (test_apply_divergent_decision_forces_unconditional_fence, test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive, test_real_replay_divergence_sets_unrecoverable_fence all still PASS).",
|
||||
"tags": [
|
||||
"raft-apply-decoupling",
|
||||
"task-7.2",
|
||||
"xce-test-ha",
|
||||
"factstream",
|
||||
"persist_recover",
|
||||
"APPLY_DIVERGENT",
|
||||
"false-positive-candidate",
|
||||
"boot-race",
|
||||
"concurrency",
|
||||
"live-regression-found",
|
||||
"fixed",
|
||||
"recovery-complete-gate",
|
||||
"myapp-boot-sequencing"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1409",
|
||||
"bug-1415"
|
||||
],
|
||||
"occurrences": 11,
|
||||
"last_seen": "2026-07-05T07:45:00.000Z"
|
||||
}
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue