docs(raft-ha-core-safety): document delaymgr-checkpoint blocker on task 5.10a

Records this session's investigation: walcompare's per-member floor
fix (f3c86a6) and the --force-snapshot soak scaffolding (9f4292d) were
built specifically to close 5.10a with direct live evidence, but hit
a new, separate blocker -- schedule_checkpoint() fires correctly but
its delaymgr-scheduled checkpoint_task_fn never executes in a live
deployment, so no snapshot is ever installed to test against. Task
5.10a stays open pending a dedicated delaymgr investigation.
This commit is contained in:
Rongsong Shen 2026-07-02 03:20:25 +00:00
parent 9f4292de29
commit 4696f43a74

View file

@ -175,6 +175,7 @@
- [ ] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag [5.10a, OPEN] + election-escalation tail [5.10b, FIX LANDED]
- _5.10b DONE (bug-1310, commit pending): RDHUP failover DEBOUNCE. Root cause = a dead primary's fd RDHUPs faster than an election settles; the 5.6 point-in-time guard misses RDHUPs that land after a lost election (role briefly back to STANDBY) → term++ per RDHUP (48 begin_failovers, terms 6→24). Fix: ha_state_t.last_failover_ms stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2×election_timeout of the last trigger. Unit-proven (test_rdhup_within_cooldown_is_debounced); soak 10/10 elect, terms 4→14 bounded, begin_failover 14/14/12 (was 48), no regression — but the stochastic storm did not recur this run (debounce fired 0×), so soak-reproduction of the storm is still pending. 5.10a (divergence catch-up lag) remains OPEN below._
- _(a) committed_divergence>0 from the just-restarted node's catch-up LAG (walcompare `present=2`: entry on the 2 live nodes, not yet backfilled on the restarted one). Transient, settles; NOT lost data. Closed by entity-bearing snapshots / the full raft-apply-decoupling apply seam (EXTERNAL — ha-commit-durability / ha-convergence-recovery). (b) Election-escalation TAIL: under an adversarial overlap (kill inside a split-vote window) a round can still stall + ladder terms (observed round-4 term 6→25 before electing) despite 5.7's jitter — liveness held (elected next round) but degraded. Harden: cap/dampen term escalation or widen the jitter under repeated same-round failovers._
- _2026-07-01 investigation (post task 3.2 + 7.2, this session): task 7.2's own 3-round-clean soak history never triggered a real snapshot install (waldump confirmed `snap/` empty every run — all rejoins caught up via incremental WAL, never a full checkpoint), so 5.10a's exact scenario remained UNEXERCISED despite the acceptance gate passing. Two enabling fixes landed to close this gap for real: (1) `walcompare` gained per-member checkpoint-floor awareness (commit `f3c86a6`) — it previously had no concept of a member's own compaction floor and would always flag a snapshot-caught-up member's pre-checkpoint range as MISSING/divergent regardless of correctness, which is the exact false-positive `ha-entity-snapshots` task 7.1 diagnosed (`committed_divergence=11`); now proven via 2 dedicated unit tests. (2) a `--max-wal-records` CLI flag + a new `--force-snapshot` soak mode (commit `9f4292d`) were built to deterministically force a real snapshot-triggered empty-rejoin in the soak (submit past the record-count checkpoint threshold, wipe a node's data dir, restart, assert the log shows a real "HA: snapshot loaded" line before trusting the verdict). **BLOCKED on a newly-discovered, separate issue**: live-instrumented with temporary diagnostic logging (since reverted) and confirmed `schedule_checkpoint()` correctly fires once `wal_record_count` reaches the configured threshold, but the delaymgr-scheduled `checkpoint_task_fn` never executes afterward — no checkpoint ever completes, no snapshot file is ever written, in a live multi-node deployment. This appears to be a deeper issue in the delaymgr scheduling path itself (or its wiring for this specific call shape via `persist->cluster->delaymgr`), not in this session's CLI/soak-scaffolding work, and has production implications beyond this test scenario (WAL segments may never be trimmed automatically today). Needs its own dedicated investigation (likely `src/lib` delaymgr) before `--force-snapshot` can produce a real PASS and close 5.10a with direct live evidence._
- Observable completion: a 10-round chaotic soak verdict is committed_divergence=0 with no per-round election stall — the green 7.2 gate
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
- _Boundary: catch-up backfill (EXTERNAL) + election-liveness tail (5.7 family)_