fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)

Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
Rongsong Shen 2026-07-07 03:03:40 +00:00
parent 5f98d1b0ef
commit a5e29a6ffa
17 changed files with 74793 additions and 156 deletions

View file

@ -1,7 +1,13 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-07-03T03:23:30.967Z
> Files: 569 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-07-05T07:39:59.974Z
> Files: 600 tracked | Anatomy hits: 0 | Misses: 0
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/
- `append_bug_1414.py` (~1114 tok)
- `append_bugs.py` (~1412 tok)
- `run_regression.sh` (~181 tok)
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/c9cdeb23-6930-42d0-adab-2651b711793f/scratchpad/
@ -219,10 +225,10 @@
## .claude/skills/xce-test-ha/
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~13692 tok)
- `SKILL.md` — XCE HA Failover Soak Test (~3703 tok)
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~18920 tok)
- `SKILL.md` — XCE HA Failover Soak Test (~4553 tok)
- `soak-verdict-selftest.sh` — soak-verdict-selftest.sh — deterministic unit test for the verdict parser in (~1857 tok)
- `soak-verdict.sh` — soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. (~1194 tok)
- `soak-verdict.sh` — soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. (~1383 tok)
## .codegraph/
@ -462,7 +468,12 @@
## .kiro/specs/raft-apply-decoupling/
- `brief.md` — Brief: raft-apply-decoupling (~2239 tok)
- `design.md` — Design Document: raft-apply-decoupling (~9210 tok)
- `facts-in-raft.md` — Design Note: Routing Facts Through Raft (apply-after-commit for RETAIN events) (~5569 tok)
- `requirements.md` — Requirements Document (~4629 tok)
- `research.md` — Research & Design Decisions (~4600 tok)
- `spec.json` (~129 tok)
- `tasks.md` — Implementation Plan (~20719 tok)
## .kiro/specs/raft-commit-safety/
@ -614,6 +625,10 @@
- `#raft.tla#` — ------------------------------- MODULE raft --------------------------------- (~5162 tok)
- `raft.tla` — ------------------------------- MODULE raft --------------------------------- (~5350 tok)
## reports/
- `jun29-jul04-2026.md` — Work Summary: June 30 – July 4, 2026 (~857 tok)
## src/
- `flake.nix` (~571 tok)
@ -889,24 +904,42 @@
- `CLAUDE.md` — XCE Controller (~3373 tok)
- `cluster.c` — include "config.h" (~18098 tok)
- `commit.c` — Declares is (~12825 tok)
- `config_repl.c` — define _GNU_SOURCE (~28945 tok)
- `config_repl.h` — pragma once (~6155 tok)
- `ha.c` — define _GNU_SOURCE (~100992 tok)
- `ha.c` — define _GNU_SOURCE (~102306 tok)
- `ha.h` — ifndef HA_H (~26001 tok)
- `job.c` — include <errno.h> (~9864 tok)
- `job.c` — include <errno.h> (~16424 tok)
- `main.c` — include <argp.h> (~3035 tok)
- `meson.build` (~1491 tok)
- `myapp.c` — define _GNU_SOURCE (~4920 tok)
- `persist.c` — define _GNU_SOURCE (~43339 tok)
- `meson.build` (~1580 tok)
- `myapp.c` — define _GNU_SOURCE (~5215 tok)
- `persist.c` — define _GNU_SOURCE (~60786 tok)
## src/controller/include/
- `cluster.h` — if !defined(_CLUSTER_H_) (~2610 tok)
- `commit.h` — pragma once (~3635 tok)
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1817 tok)
- `controller_test.c` — include <argp.h> (~1986 tok)
- `snapdiff.c` — define _GNU_SOURCE (~833 tok)
- `snapdiff.h` — pragma once (~569 tok)
- `snapverify.c` — define _GNU_SOURCE (~2505 tok)
- `snapverify.h` — pragma once (~645 tok)
- `test_apply_committed_entry.c` — Declares directly (~4141 tok)
- `test_apply_correctness_verification.c` — Task 5.1-5.3: post-replay apply verdict (CLEAN/DIVERGENT), routes through existing loud-fence; 11 tests incl. explicit stays-alive pin + fence-clear-via-restart + direct persist_clear_recovery_unrecoverable coverage (~17000 tok)
- `test_apply_pump_integration.c` — Declares hook (~16913 tok)
- `test_apply_pump.c` — Task 6.1 ordered apply pump tests (4, pool=NULL) + task 6.2 round-2 bug-1415 regression `test_apply_pump_late_pending_apply_registration_self_heals` (5th test, own tpr_* fixture with a REAL multi-vcpu vtaskmgr_t via commit_set_pool, forces apply-pump-wins-race via a barrier) (~17700 tok)
- `test_apply_reapply_idempotency.c` — Declares xce_job_t (~4001 tok)
- `test_apply_seam.c` (~2936 tok)
- `test_apply_seam.c` — raft-apply-decoupling task 2.1: calls apply_job_start/finish/fail/update_transition() directly on a bare jobmgr+job_t (no cluster_t), bypassing on_job_start/finish/fail, to prove the extracted guard cores work outside the receipt path (~2900 tok)
- `test_commit_barrier_reply.c` — Declares uint8_t (~38815 tok)
- `test_commit_durability.c` — define _GNU_SOURCE (~31376 tok)
- `test_commit_resolve.c` (~14018 tok)
- `test_config_repl.c` — define _GNU_SOURCE (~42756 tok)
- `test_convergence_verification.c` — define _GNU_SOURCE (~16987 tok)
- `test_demotion_convergence.c` — define _GNU_SOURCE (~12767 tok)
- `test_e2e_hooks.c` — define _GNU_SOURCE (~2184 tok)
- `test_egress_filter.c` — Declares xce_job_t (~5072 tok)
- `test_ha_core_safety.c` — Declares char (~36309 tok)
@ -917,12 +950,15 @@
- `test_ha_vote_gate.c` — define _GNU_SOURCE (~14165 tok)
- `test_persist_hooks.c` — define _GNU_SOURCE (~4305 tok)
- `test_persist_snapshot_entities.c` — unsigned: queue (~14897 tok)
- `test_persist_truncate.c` — define _GNU_SOURCE (~5031 tok)
- `test_replay_apply_seam.c` — Declares through (~3707 tok)
- `test_snapdiff.c` — define _GNU_SOURCE (~1033 tok)
- `test_snapverify.c` — define _GNU_SOURCE (~2008 tok)
- `test_walcompare.c` — define _GNU_SOURCE (~6845 tok)
- `test.c` — include <argp.h> (~28321 tok)
- `test.c` — include <argp.h> (~29726 tok)
- `walcompare.c` — define _GNU_SOURCE (~5176 tok)
- `walcompare.h` — pragma once (~1474 tok)
- `waldump.c` — define _GNU_SOURCE (~5404 tok)
## src/include/
@ -931,8 +967,9 @@
- `delay.h` — if !defined(_DELAY_H_) (~414 tok)
- `myconf.h` — if !defined(_MYCONF_H_) (~560 tok)
- `netio.h` — if !defined(_NETIO_H_) (~2313 tok)
- `persist.h` — pragma once (~11003 tok)
- `persist.h` — pragma once (~20431 tok)
- `typemap.h` — if !defined(_TYPEMAP_H_) (~4717 tok)
- `wal_types.h` — pragma once (~1996 tok)
## src/lib/
@ -942,9 +979,15 @@
- `request.c` — include <errno.h> (~15919 tok)
- `vtaskmgr.c` — include <errno.h> (~3822 tok)
## src/lib/raft/
- `raft_sim.c` — ifdef XCE_TESTING (~28490 tok)
- `raft.h` — ifndef RAFT_H (~17581 tok)
## src/lib/testsuite/
- `test_ctrlstatus.c` (~4821 tok)
- `test_raft.c` — include <errno.h> (~98108 tok)
- `test_wal_wire.c` — include <errno.h> (~4238 tok)
## src/schema/