lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.wolf/hooks/_session.json
Rongsong Shen a5e29a6ffa fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2) 
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
2026-07-07 03:03:40 +00:00

470 lines
No EOL
15 KiB
JSON
Raw Blame History

{
"session_id": "session-2026-07-05-2050",
"started": "2026-07-05T03:50:02.712Z",
"files_read": {
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/requirements.md": {
"count": 3,
"tokens": 4629,
"first_read": "2026-07-05T05:13:49.214Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md": {
"count": 12,
"tokens": 20719,
"first_read": "2026-07-05T05:14:01.912Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/design.md": {
"count": 1,
"tokens": 9210,
"first_read": "2026-07-05T05:14:02.728Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c": {
"count": 4,
"tokens": 28490,
"first_read": "2026-07-05T05:14:31.226Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft.h": {
"count": 1,
"tokens": 17581,
"first_read": "2026-07-05T05:15:27.928Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T05:20:15.107Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md": {
"count": 4,
"tokens": 4553,
"first_read": "2026-07-05T05:28:34.593Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh": {
"count": 18,
"tokens": 18920,
"first_read": "2026-07-05T05:28:47.075Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c": {
"count": 12,
"tokens": 29726,
"first_read": "2026-07-05T05:29:07.259Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/job.c": {
"count": 5,
"tokens": 16424,
"first_read": "2026-07-05T05:30:09.491Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client.c": {
"count": 2,
"tokens": 0,
"first_read": "2026-07-05T05:30:41.791Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/cluster.c": {
"count": 4,
"tokens": 18098,
"first_read": "2026-07-05T05:30:54.661Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client_internal.h": {
"count": 2,
"tokens": 0,
"first_read": "2026-07-05T05:31:49.690Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/sched.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T05:32:31.884Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c": {
"count": 6,
"tokens": 5404,
"first_read": "2026-07-05T05:33:41.172Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/wal_types.h": {
"count": 2,
"tokens": 1996,
"first_read": "2026-07-05T05:33:51.925Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xce/types.h": {
"count": 3,
"tokens": 0,
"first_read": "2026-07-05T05:34:30.060Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/soak-verdict.sh": {
"count": 3,
"tokens": 1383,
"first_read": "2026-07-05T05:37:16.616Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c": {
"count": 20,
"tokens": 60786,
"first_read": "2026-07-05T06:14:23.860Z"
},
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/full.diff": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T06:47:57.965Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/utils.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T06:51:04.731Z"
},
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/soak.diff": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T06:59:21.281Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
"count": 9,
"tokens": 102306,
"first_read": "2026-07-05T07:06:14.209Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h": {
"count": 5,
"tokens": 19990,
"first_read": "2026-07-05T07:09:10.020Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xcectx.h": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T07:12:14.838Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/vtaskmgr.c": {
"count": 1,
"tokens": 3822,
"first_read": "2026-07-05T07:19:54.733Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/controller_test.c": {
"count": 2,
"tokens": 1986,
"first_read": "2026-07-05T07:25:28.359Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/myapp.c": {
"count": 4,
"tokens": 5215,
"first_read": "2026-07-05T07:31:53.169Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_apply_correctness_verification.c": {
"count": 4,
"tokens": 17000,
"first_read": "2026-07-05T08:08:11.151Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md": {
"count": 1,
"tokens": 380,
"first_read": "2026-07-05T08:09:36.337Z"
},
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/tacv_out.log": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T12:13:07.549Z"
}
},
"files_written": [
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/reports/jun29-jul04-2026.md",
"action": "create",
"tokens": 979,
"at": "2026-07-05T03:57:01.756Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 40,
"at": "2026-07-05T05:20:38.611Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 38,
"at": "2026-07-05T05:20:58.465Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 26,
"at": "2026-07-05T05:21:03.400Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 25,
"at": "2026-07-05T05:21:17.748Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md",
"action": "edit",
"tokens": 25,
"at": "2026-07-05T05:25:03.542Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md",
"action": "edit",
"tokens": 896,
"at": "2026-07-05T05:25:21.590Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 46,
"at": "2026-07-05T05:41:04.648Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 97,
"at": "2026-07-05T05:41:12.545Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 825,
"at": "2026-07-05T05:41:30.455Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 566,
"at": "2026-07-05T05:41:44.220Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c",
"action": "edit",
"tokens": 415,
"at": "2026-07-05T05:42:04.666Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c",
"action": "edit",
"tokens": 376,
"at": "2026-07-05T05:42:14.873Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/soak-verdict.sh",
"action": "edit",
"tokens": 363,
"at": "2026-07-05T05:42:57.027Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 450,
"at": "2026-07-05T05:44:13.870Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 344,
"at": "2026-07-05T05:44:41.636Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 54,
"at": "2026-07-05T05:44:48.489Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 166,
"at": "2026-07-05T05:45:10.893Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 2683,
"at": "2026-07-05T05:46:02.987Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 568,
"at": "2026-07-05T05:46:28.157Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 871,
"at": "2026-07-05T05:48:15.351Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 228,
"at": "2026-07-05T05:48:37.033Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 776,
"at": "2026-07-05T05:53:15.669Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 17,
"at": "2026-07-05T05:53:37.328Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 318,
"at": "2026-07-05T05:53:47.430Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 141,
"at": "2026-07-05T05:54:00.332Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 374,
"at": "2026-07-05T06:16:57.614Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 613,
"at": "2026-07-05T06:22:34.527Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 13,
"at": "2026-07-05T06:23:51.836Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 210,
"at": "2026-07-05T06:24:37.656Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 92,
"at": "2026-07-05T06:24:53.835Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 245,
"at": "2026-07-05T06:33:11.363Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 603,
"at": "2026-07-05T06:34:12.298Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 160,
"at": "2026-07-05T06:34:36.760Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 993,
"at": "2026-07-05T07:02:41.181Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
"action": "edit",
"tokens": 680,
"at": "2026-07-05T07:16:17.567Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
"action": "edit",
"tokens": 292,
"at": "2026-07-05T07:16:28.893Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 160,
"at": "2026-07-05T07:16:43.698Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 133,
"at": "2026-07-05T07:16:50.545Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 514,
"at": "2026-07-05T07:17:19.666Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 362,
"at": "2026-07-05T07:19:06.213Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 403,
"at": "2026-07-05T07:19:17.398Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 313,
"at": "2026-07-05T07:21:05.529Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
"action": "edit",
"tokens": 504,
"at": "2026-07-05T07:21:38.054Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
"action": "edit",
"tokens": 311,
"at": "2026-07-05T07:21:56.305Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
"action": "edit",
"tokens": 486,
"at": "2026-07-05T07:39:27.113Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 164,
"at": "2026-07-05T07:39:49.040Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/myapp.c",
"action": "edit",
"tokens": 384,
"at": "2026-07-05T07:39:59.983Z"
}
],
"edit_counts": {
"reports/jun29-jul04-2026.md": 1,
"src/lib/raft/raft_sim.c": 4,
".kiro/specs/raft-apply-decoupling/tasks.md": 2,
"src/controller/testsuite/test.c": 5,
"src/controller/testsuite/waldump.c": 2,
".claude/skills/xce-test-ha/soak-verdict.sh": 1,
".claude/skills/xce-test-ha/run-ha-soak.sh": 16,
".claude/skills/xce-test-ha/SKILL.md": 4,
"src/include/persist.h": 3,
"src/controller/persist.c": 7,
"src/controller/ha.c": 2,
"src/controller/myapp.c": 1
},
"anatomy_hits": 20,
"anatomy_misses": 11,
"repeated_reads_warned": 104,
"cerebrum_warnings": 0,
"stop_count": 8
}
Reference in a new issue View git blame Copy permalink