chore: update OpenWolf session state (xce-audit-invariants controller pass)

This commit is contained in:
Rongsong Shen 2026-07-19 08:14:19 +00:00
parent d8d1901901
commit e1f4d676de
6 changed files with 6009 additions and 28 deletions

View file

@ -1,6 +1,6 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-07-18T13:30:08.564Z
> Auto-maintained by OpenWolf. Last scanned: 2026-07-18T15:39:26.395Z
> Files: 684 tracked | Anatomy hits: 0 | Misses: 0
## ../../../../../../tmp/claude-1000/
@ -255,7 +255,7 @@
## .claude/skills/xce-audit-invariants/
- `SKILL.md` — XCE Design-Invariant Audit & Fix (~2126 tok)
- `SKILL.md` — XCE Design-Invariant Audit & Fix (~2311 tok)
- `SKILL.md` — XCE Design-Invariant Audit & Fix — audit documented invariants, fix confirmed violations, prove with build+unit+live HA soak (~1350 tok)
## .claude/skills/xce-build/
@ -628,14 +628,14 @@
- `api-standards.md` — API Standards (~1291 tok)
- `build-deployment.md` — Build & Deployment (~1451 tok)
- `error-handling.md` — Error Handling Standards (~1026 tok)
- `ha-async-io.md` — HA Protocol & Async I/O (~3864 tok)
- `ha-async-io.md` — HA Protocol & Async I/O (~4180 tok)
- `persistence.md` — WAL & Persistence Durability (~2600 tok)
- `product.md` — Product Overview (~1246 tok)
- `roadmap.md` — Roadmap (~10840 tok)
- `product.md` — Product Overview (~1429 tok)
- `roadmap.md` — Roadmap (~11258 tok)
- `security.md` — Security Standards (~804 tok)
- `structure.md` — Project Structure (~1170 tok)
- `tech.md` — Technology Stack (~1155 tok)
- `testing.md` — Testing Standards (~741 tok)
- `testing.md` — Testing Standards (~936 tok)
## .remember/
@ -968,14 +968,14 @@
## src/controller/
- `apiserver.c` — include <errno.h> (~5962 tok)
- `CLAUDE.md` — XCE Controller (~3373 tok)
- `CLAUDE.md` — XCE Controller (~3491 tok)
- `cluster.c` — include "config.h" (~18669 tok)
- `commit.c` — Declares is (~12825 tok)
- `config_repl.c` — define _GNU_SOURCE (~30643 tok)
- `config_repl.c` — define _GNU_SOURCE (~32213 tok)
- `config_repl.h` — pragma once (~6155 tok)
- `ha_convergence.c` — Declares char (~5779 tok)
- `ha_convergence.c` — Declares char (~7830 tok)
- `ha.c` — define _GNU_SOURCE (~111605 tok)
- `ha.h` — ifndef HA_H (~27638 tok)
- `ha.h` — ifndef HA_H; +demotion_convergence_inflight field, ha_set_demotion_offload_for_test/ha_set_catchup_offload_for_test decls (2026-07-18) (~31200 tok)
- `job.c` — include <errno.h> (~24947 tok)
- `join.c` — include "config.h" (~2795 tok)
- `main.c` — include <argp.h> (~3035 tok)
@ -1007,9 +1007,9 @@
- `test_commit_barrier_rollout.c` — Declares explicitly (~26498 tok)
- `test_commit_durability.c` — define _GNU_SOURCE (~32879 tok)
- `test_commit_resolve.c` (~14018 tok)
- `test_config_repl.c` — define _GNU_SOURCE (~42756 tok)
- `test_convergence_verification.c` — define _GNU_SOURCE (~18715 tok)
- `test_demotion_convergence.c` — define _GNU_SOURCE (~12767 tok)
- `test_config_repl.c` — define _GNU_SOURCE; +test_catchup_offload_probe_runs_promptly / test_catchup_inline_probe_blocks_RED (2026-07-18, bug-1451 RED/GREEN via real single-worker delaymgr + probe task) (~48500 tok)
- `test_convergence_verification.c` — define _GNU_SOURCE; pre-existing ha_walack_on_ack/ha_catchup_continuation_fn fixture, NO slowlane_init — the surface that broke when catchup offload defaulted ON (see bug-1451) (~18715 tok)
- `test_demotion_convergence.c` — define _GNU_SOURCE; dc_setup +slowlane_init, +test_demotion_offload_task_returns_promptly / test_demotion_inline_task_blocks_RED (2026-07-18, bug-1450 RED/GREEN) (~18600 tok)
- `test_e2e_capnp.c` — include <stdint.h> (~3729 tok)
- `test_e2e_checkpoint.c` — define _GNU_SOURCE (~3544 tok)
- `test_e2e_gc.c` — define _GNU_SOURCE (~1678 tok)

View file

@ -13434,5 +13434,85 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-18T13:00:00.000Z"
},
{
"id": "bug-1450",
"timestamp": "2026-07-18",
"error_message": "xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_demotion_convergence_fn scheduled directly as a delaymgr task body with no vtask/slow-lane hop: it truncates the WAL (persist_truncate_wal_above, file I/O under p->lock) then calls ha_self_restart (tlog_exit synchronous log flush + execv) -- all inline on the delaymgr thread, the SAME single thread that drives heartbeats and election timeouts cluster-wide for the whole cluster.",
"file": "src/controller/ha_convergence.c, src/controller/ha.h",
"root_cause": "ha_demotion_convergence_fn (ha_convergence.c) ran its full body -- truncate + self-restart -- synchronously on whatever thread invoked it (the delaymgr worker in production). A prior fix attempt (2026-07-10, see bug-1429) tried moving this to a VTASK_LIVE_BLOCK vtask via c->ha.pool and deadlocked controller_unit_test, because c->ha.pool is never initialized/started by this file's lightweight unit-test fixtures (dc_setup), so a vtask enqueued VTASK_LIVE_BLOCK blocked forever waiting for a worker that never existed; that attempt was reverted.",
"fix": "Split ha_demotion_convergence_fn into a dispatcher (fast admin work: single-flight CAS on new ha_state_t field demotion_convergence_inflight, then slowlane_submit) and a new demotion_lane_fn slow-lane worker body running the actual truncate+self-restart (extracted into ha_demotion_convergence_do_run) -- the SAME established slow-lane pattern already used for snapshot send / config-push / peer connect in ha.c (ha_snapshot_send_dispatch, ha_config_push_offloadable, ha_connect_to_peer), not the vtask pool that deadlocked before. New feature flag g_drive_async_demotion_offload (ha_set_demotion_offload_for_test / ha_demotion_offload_enabled, ha.h/ha_convergence.c) gates the offload. IMPORTANT: defaulting this flag to ON (1) was tried first and reproduced a NEW real hang -- raft_cb_become_follower (which schedules this task) is exercised by a very broad set of pre-existing controller_unit_test fixtures across many files that never call slowlane_init, so the offloaded work was silently queued on an uninitialized c->slowlane (queued, never run -- no live workers) and downstream code waiting on the now-never-happening completion hung. Reverted the default to OFF (0), matching the ORIGINAL rollout discipline of the three reference offloads (all shipped OFF-by-default, flipped ON only in a later dedicated compatibility-audit pass -- raft-drive-async task 3.4). The mechanism is correct and covered by dedicated tests (test_demotion_offload_task_returns_promptly / test_demotion_inline_task_blocks_RED in test_demotion_convergence.c) that explicitly flip the flag ON against a fixture with a real slowlane_init'd lane. Flipping the cluster-wide default to ON is a follow-up compatibility audit across the full test suite, explicitly out of scope for this fix. Verified: standalone controller_unit_test binary run directly (bypassing meson) reports 976 Tests 0 Failures 0 Ignored, no hang/timeout.",
"tags": [
"async-io",
"thread-role-violation",
"delaymgr",
"slowlane",
"ha_convergence.c",
"demotion",
"raft-drive-async-pattern",
"xce-audit-invariants",
"controller-tests"
],
"related_bugs": [
"bug-1429",
"bug-1451"
],
"occurrences": 1,
"last_seen": "2026-07-18"
},
{
"id": "bug-1451",
"timestamp": "2026-07-18",
"error_message": "xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_catchup_continuation_fn scheduled directly as a delaymgr task body with no vtask/slow-lane hop: catchup_stream_range -> ha_stream_wal_to_standby does blocking WAL file reads + socket writes to a standby directly on the delaymgr thread. Flagged as the MORE URGENT of the two violations -- catch-up can stream a variable, potentially large amount of WAL data, so it can block delaymgr (and therefore heartbeats to every OTHER peer) for an unbounded, unpredictable time, a real spurious-election risk.",
"file": "src/controller/config_repl.c, src/controller/ha.h",
"root_cause": "ha_catchup_continuation_fn (config_repl.c) ran its full body -- re-derive target, then catchup_stream_range's blocking WAL reads/socket writes -- synchronously on whatever thread invoked it (the delaymgr worker in production). Source comment at config_repl.c confirmed a prior VTASK_LIVE_BLOCK vtask-pool attempt here (2026-07-10) also deadlocked controller_unit_test and was reverted (bug-1429), tracked as a known open violation.",
"fix": "Split ha_catchup_continuation_fn into a dispatcher (fast admin work: slowlane_submit keyed by peer_idx+1) and a new catchup_lane_fn slow-lane worker body running the original streaming logic unchanged -- the SAME established slow-lane pattern used for snapshot/config-push/reconnect in ha.c, not the vtask pool. c->ha.catchup_inflight[peer_idx] (set by ha_walack_on_ack before scheduling) remains the authoritative per-standby single-flight guard; the lane's own per-key dedup is a redundant backstop, mirroring ha_connect_to_peer's reconnect offload. New feature flag g_drive_async_catchup_offload (ha_set_catchup_offload_for_test / ha_catchup_offload_enabled, ha.h/config_repl.c) gates the offload. Defaulting this flag to ON (1) was tried first and reproduced a real hang: test_convergence_verification.c's pre-existing ACK-driven catch-up fixtures (which directly exercise this exact code path via ha_walack_continuation_for_test) never call slowlane_init, so offloaded work was silently queued and never ran, hanging controller_unit_test at ~90s+ (confirmed reproducible both via meson and a direct standalone binary run with SIGQUIT timeout). Reverted the default to OFF (0), matching the same rollout discipline as bug-1450's demotion-convergence fix and the three original raft-drive-async reference offloads. Mechanism covered by dedicated tests (test_catchup_offload_probe_runs_promptly / test_catchup_inline_probe_blocks_RED in test_config_repl.c) using a REAL, started, single-worker delaymgr (delaymgr_init+delaymgr_start, not just an un-started fixture) plus a probe task queued behind the continuation to directly prove the delaymgr thread is/isn't blocked. Flipping the cluster-wide default to ON is a follow-up compatibility audit, out of scope for this fix. Verified: standalone controller_unit_test binary run directly reports 976 Tests 0 Failures 0 Ignored, no hang/timeout.",
"tags": [
"async-io",
"thread-role-violation",
"delaymgr",
"slowlane",
"config_repl.c",
"catchup-continuation",
"raft-drive-async-pattern",
"xce-audit-invariants",
"controller-tests",
"spurious-election-risk"
],
"related_bugs": [
"bug-1429",
"bug-1450"
],
"occurrences": 1,
"last_seen": "2026-07-18"
},
{
"id": "bug-1452",
"timestamp": "2026-07-18T00:00:00Z",
"error_message": "AddressSanitizer: SEGV on unknown address 0x000000000038 in commit_advance_locked (controller/commit.c:208), called from commit_on_durable_advance (commit.c:341) <- fsync_worker (persist.c:399), a background ASAN thread started by persist_start (persist.c:723) inside test_persist_truncate.c's make_fresh_persist / test_truncate_above_resolves_pending_continuations_as_failed. Reproduced by temporarily flipping BOTH g_drive_async_demotion_offload and g_drive_async_catchup_offload (bug-1450/1451) from their default 0 to 1 and running the standalone controller_unit_test binary directly (976-test suite) -- crashed immediately, before completing the suite. Confirmed absent with both flags at their committed default of 0: the same binary/build (just reverted) completed cleanly, 976 Tests 0 Failures 0 Ignored, twice.",
"file": "src/controller/commit.c (crash site, commit_advance_locked); src/controller/persist.c (fsync_worker background thread, apply_committed_entry/restore_job/makeJob chain also appears in a related non-crashing stack from the same investigation); src/controller/ha_convergence.c + config_repl.c (g_drive_async_demotion_offload / g_drive_async_catchup_offload -- the flags whose flip triggers this)",
"root_cause": "NOT ROOT-CAUSED -- discovered as a side effect of an xce-audit-invariants Step 4 live-regression attempt (2026-07-18), not investigated further, deliberately out of scope for that session (see bug-1450/1451: flipping either offload flag's default to ON is explicitly scoped to a LATER, dedicated compatibility-audit pass, matching the original rollout discipline of the snapshot/config-push/reconnect offloads). bug-1450/1451 already documented a HANG from flipping these flags ON against pre-existing fixtures that never call slowlane_init (uninitialized c->slowlane silently queues work that's never run). This is a DIFFERENT and more severe symptom -- a hard SEGV, not a hang -- in an apparently-unrelated code path (commit.c's durability-advance callback, invoked from the fsync background thread against a WAL-truncation test fixture), suggesting the two offload flags' interaction with the broader pre-existing test-fixture surface is not limited to the single hang mechanism already documented. Not confirmed whether this reproduces with only ONE of the two flags flipped (both were flipped together when this was found) or whether it is fixture-specific (test_persist_truncate.c) vs a broader live-relevant race -- the crash is inside commit.c/persist.c machinery that IS also exercised by the real live daemon, so this should NOT be assumed to be test-only until someone actually checks whether a live cluster with both flags forced on hits the same SEGV.",
"fix": "NOT FIXED -- out of scope for the xce-audit-invariants session that found it (which correctly kept both flags at their default of 0, matching bug-1450/1451's scoping decision; this finding is additional evidence supporting that decision, discovered incidentally while verifying it). Whoever picks up the deferred 'flip demotion/catchup offload to default ON' follow-up audit (referenced in bug-1450/1451) MUST reproduce and root-cause this SEGV as part of that work, in addition to the already-documented hang -- do not treat the hang as the only pre-flip blocker. Suggested next steps: (1) reproduce with each flag flipped individually to isolate which one (or the combination) triggers it; (2) reproduce running only test_persist_truncate.c in isolation (narrow binary via meson test args or a --run filter) to rule out cross-test global-state pollution as a confound; (3) once isolated, read commit_advance_locked (commit.c:208) against what's at offset 0x38 in whatever struct is being dereferenced -- a near-null read at a small fixed offset strongly suggests an uninitialized or already-freed context pointer, consistent with the same 'offload path assumes c->slowlane/commit-engine machinery is live-initialized' class of issue as the documented hang, just manifesting as a crash instead of a stall in this particular fixture.",
"tags": [
"ha_convergence.c",
"config_repl.c",
"commit.c",
"demotion-offload",
"catchup-offload",
"raft-drive-async-pattern",
"xce-audit-invariants",
"controller-tests",
"ASAN",
"SEGV",
"NOT-FIXED",
"not-root-caused"
],
"related_bugs": [
"bug-1429",
"bug-1450",
"bug-1451"
],
"occurrences": 1,
"last_seen": "2026-07-18"
}
]

View file

@ -503,3 +503,8 @@ tasks.md showed 18/18 (12 leaf + 6 group-header) checked since 2026-06-16, and r
## Key Learning (2026-07-18, bug-1449 fixed) — run-ha-soak.sh had TWO independent convergence signals conflated into one; round 1 was only guarded by the wrong one
`wait_for_full_convergence`/`cluster_committed_converged` (gates every round AFTER round 1) checks role-count (1 primary + 2 standby) + committed-LSN catch-up — it says nothing about `configVersion` (the config "tag" a standby needs before `ha_on_primary_ctrl_fd_rdhup`'s ha-config-bootstrap Req 2.8 voter-only guard will let it self-elect). Round 1 had NO wait for that second, independent signal before its first kill, so an unlucky-but-legitimate slow cold-start election (observed 10-12s across 3 verification runs, vs. the ~1.7s job-submit budget the old code implicitly relied on) could let the kill land before config-push finished, correctly triggering the tag=0 guard and producing a false "no primary elected" round-1 failure. Fix mirrors `wait_for_full_convergence`'s exact polling primitive (alive-count via `pgrep -c -f "$(run_pattern)"`, STABLE_TICKS=2, sleep 2, ~90s bound) but gates on a NEW `cluster_config_converged` (all 3 nodes' `configVersion` via ctrlstatus non-zero, probed by a new `node_config_tag()` mirroring `node_probe`'s single-node client-probe technique) — added right after `wait_for_full_convergence` (~line 556) and called once, right before the round loop (~line 1897), so it applies ONLY to sequential mode's round-1 kill (chaotic mode intentionally tolerates this exact race elsewhere — a stalled election there is a COUNTED outcome, never a hard fail, so it was correctly left untouched). **Do-not-repeat**: when a soak/test driver has more than one "is the cluster ready" signal server-side (committed-prefix vs. config-tag here), a wait function gating on ONE does not imply the other is also satisfied — check what the actual guard/precondition being raced depends on before assuming an existing convergence wait already covers it. Verified 3/3 back-to-back clean `--sequential 6` runs on genuine ext4 (`/home/claude/tmp`, confirmed via `mount`/`df -T`, not overlayfs); cross-checked cold-start timing from controller logs across all 3 runs (election 10.3-11.7s after process start, config-push landing only 40-46ms after election every time) to confirm the new gate's ~4s minimum dwell / ~90s ceiling has ~2 orders of magnitude of margin, not just enough for that day's luck.
## Do-Not-Repeat (2026-07-18, bugs 1450/1451) — a new slow-lane offload's own non-blocking-submit contract is not enough; defaulting the flag ON can STILL hang controller_unit_test via a DIFFERENT mechanism than the vtask-pool deadlock it replaces
Fixed two `xce-audit-invariants controller` thread-role violations (`ha_demotion_convergence_fn` ha_convergence.c, `ha_catchup_continuation_fn` config_repl.c — both ran blocking work directly on the delaymgr thread) by adding a 4th/5th instance of the established slow-lane pattern (`slowlane_submit`/`*_lane_fn`, mirroring `ha_snapshot_send_dispatch`/`ha_config_push_offloadable`/`ha_connect_to_peer` in ha.c) — NOT the VTASK_LIVE_BLOCK vtask-pool approach a prior session tried and reverted (bug-1429) after it deadlocked `controller_unit_test`. Confirmed via `git log`/memory why that attempt failed: `c->ha.pool` was never initialized in the lightweight unit-test fixtures it ran against, so `vtask_enq_classified(..., VTASK_LIVE_BLOCK)` blocked forever waiting for a worker that didn't exist. `slowlane_submit` is architecturally immune to THAT failure mode (it is unconditionally non-blocking — with zero live workers it just returns `-EAGAIN`/`0` and leaves the work queued-but-never-run, never blocks the caller). BUT defaulting the new offload flags to ON (`g_drive_async_demotion_offload`/`g_drive_async_catchup_offload = 1`) reproduced a REAL hang anyway, via a DIFFERENT mechanism: the production callback being offloaded (`raft_cb_become_follower` for demotion, `ha_walack_on_ack`/`ha_catchup_continuation_fn` for catch-up) is exercised by a VERY BROAD set of PRE-EXISTING `controller_unit_test` fixtures across many files (confirmed hit: `test_convergence_verification.c`'s ACK-driven catch-up suite) that never call `slowlane_init` — so the offloaded work is silently queued on an uninitialized lane and NEVER RUNS, and any downstream code (in THOSE pre-existing tests, not the new ones) that waits — even boundedly — for the now-never-happening completion stalls/times out. Reproduced with a direct standalone binary run (`timeout -s QUIT 90 ./controller_unit_test`, bypassing meson) BOTH before (hang, SIGQUIT after 90s) and after (clean, "976 Tests 0 Failures 0 Ignored") the flag-default flip to OFF. **General rule**: when adding a NEW slow-lane offload flag for an existing, widely-exercised callback, default it OFF and add your own dedicated tests that explicitly flip it ON against a fixture you control (with `slowlane_init` called) — do NOT default a new offload flag ON just because the lane mechanism itself is provably non-blocking; the blast radius of "which pre-existing fixtures reach this code path without lane infra" is the real risk, and it can only be ruled out by a full test-suite compatibility audit (exactly how the ORIGINAL three raft-drive-async offloads were rolled out: shipped OFF, flipped ON only in a later dedicated task-3.4-style pass). Always verify a "fixed hang" claim by running the STANDALONE `controller_unit_test` binary directly with an external `timeout -s QUIT`, not just `meson test` (whose `Ok:N/Fail:0` aggregate can hide a TIMEOUT line entirely if you only grep for `Fail:`) — grep for `Timeout` explicitly too.
## Key Learning (2026-07-18) — `test_convergence_verification.c` is the direct, pre-existing test surface for `ha_walack_on_ack`/`ha_catchup_continuation_fn` (the ACK-driven catch-up decision, ha-convergence-recovery task 4.x); it drives them via `ha_walack_continuation_for_test` against a real-but-un-started `delaymgr` and has NO `slowlane_init` — any future change that makes catch-up continuation offload-capable must either default OFF or update this file's fixture.

View file

@ -8,13 +8,13 @@
"first_read": "2026-07-18T11:55:14.604Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
"count": 3,
"count": 32,
"tokens": 111605,
"first_read": "2026-07-18T11:56:51.745Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/netio.c": {
"count": 2,
"tokens": 9351,
"count": 4,
"tokens": 9456,
"first_read": "2026-07-18T12:15:26.076Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_netio_on_connected.c": {
@ -66,6 +66,191 @@
"count": 1,
"tokens": 2311,
"first_read": "2026-07-18T13:28:24.046Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/kiro-steering/rules/steering-principles.md": {
"count": 1,
"tokens": 593,
"first_read": "2026-07-18T14:22:54.964Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/ha-async-io.md": {
"count": 4,
"tokens": 4180,
"first_read": "2026-07-18T14:22:55.308Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/product.md": {
"count": 1,
"tokens": 1246,
"first_read": "2026-07-18T14:22:55.688Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/persistence.md": {
"count": 2,
"tokens": 2600,
"first_read": "2026-07-18T14:22:55.981Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/testing.md": {
"count": 1,
"tokens": 741,
"first_read": "2026-07-18T14:23:24.458Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-audit-invariants/SKILL.md": {
"count": 1,
"tokens": 2126,
"first_read": "2026-07-18T14:29:44.986Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/CLAUDE.md": {
"count": 4,
"tokens": 1400,
"first_read": "2026-07-18T14:31:37.947Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/error-handling.md": {
"count": 2,
"tokens": 1026,
"first_read": "2026-07-18T14:31:38.079Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c": {
"count": 22,
"tokens": 72815,
"first_read": "2026-07-18T14:33:42.209Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/vtaskmgr.c": {
"count": 1,
"tokens": 3822,
"first_read": "2026-07-18T14:34:02.440Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha_convergence.c": {
"count": 5,
"tokens": 10800,
"first_read": "2026-07-18T14:34:23.891Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/commit.c": {
"count": 2,
"tokens": 12825,
"first_read": "2026-07-18T14:35:13.971Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft.c": {
"count": 3,
"tokens": 9701,
"first_read": "2026-07-18T14:35:14.000Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/repl.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-18T14:35:26.952Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/job.c": {
"count": 4,
"tokens": 24947,
"first_read": "2026-07-18T14:35:35.666Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/worker.c": {
"count": 3,
"tokens": 0,
"first_read": "2026-07-18T14:35:54.361Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/xce_select.c": {
"count": 2,
"tokens": 0,
"first_read": "2026-07-18T14:36:12.052Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/request.c": {
"count": 1,
"tokens": 17093,
"first_read": "2026-07-18T14:36:12.266Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/cluster.c": {
"count": 3,
"tokens": 18669,
"first_read": "2026-07-18T14:36:36.941Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/sched.c": {
"count": 2,
"tokens": 6155,
"first_read": "2026-07-18T14:36:41.650Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/delay.c": {
"count": 1,
"tokens": 1520,
"first_read": "2026-07-18T14:36:44.302Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/main.c": {
"count": 1,
"tokens": 3035,
"first_read": "2026-07-18T14:36:56.309Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/apiserver.c": {
"count": 1,
"tokens": 5962,
"first_read": "2026-07-18T14:37:04.426Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/config_repl.c": {
"count": 6,
"tokens": 35500,
"first_read": "2026-07-18T14:37:24.158Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/join.c": {
"count": 1,
"tokens": 2795,
"first_read": "2026-07-18T14:37:26.025Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/resreq.c": {
"count": 2,
"tokens": 0,
"first_read": "2026-07-18T14:37:33.012Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h": {
"count": 1,
"tokens": 24030,
"first_read": "2026-07-18T14:38:57.581Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_ha_dispatch_integration.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-18T14:48:52.975Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_slowlane.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-18T14:49:01.741Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_ha_liveness.c": {
"count": 2,
"tokens": 24748,
"first_read": "2026-07-18T14:49:13.555Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c": {
"count": 6,
"tokens": 15318,
"first_read": "2026-07-18T14:49:26.389Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h": {
"count": 9,
"tokens": 29374,
"first_read": "2026-07-18T14:51:33.237Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_commit_durability.c": {
"count": 1,
"tokens": 32879,
"first_read": "2026-07-18T14:54:48.795Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/delay.h": {
"count": 1,
"tokens": 414,
"first_read": "2026-07-18T14:56:03.241Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_config_repl.c": {
"count": 3,
"tokens": 42756,
"first_read": "2026-07-18T14:56:56.098Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/slowlane.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-18T15:11:52.433Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_convergence_verification.c": {
"count": 1,
"tokens": 18715,
"first_read": "2026-07-18T15:16:32.304Z"
}
},
"files_written": [
@ -152,6 +337,264 @@
"action": "create",
"tokens": 4478,
"at": "2026-07-18T13:30:08.576Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/product.md",
"action": "edit",
"tokens": 412,
"at": "2026-07-18T14:24:39.241Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/ha-async-io.md",
"action": "edit",
"tokens": 412,
"at": "2026-07-18T14:24:51.395Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/roadmap.md",
"action": "edit",
"tokens": 598,
"at": "2026-07-18T14:25:09.086Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/roadmap.md",
"action": "edit",
"tokens": 299,
"at": "2026-07-18T14:25:21.554Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/roadmap.md",
"action": "edit",
"tokens": 140,
"at": "2026-07-18T14:25:29.605Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/roadmap.md",
"action": "edit",
"tokens": 198,
"at": "2026-07-18T14:25:38.634Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/roadmap.md",
"action": "edit",
"tokens": 162,
"at": "2026-07-18T14:25:48.331Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/testing.md",
"action": "edit",
"tokens": 230,
"at": "2026-07-18T14:26:01.571Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-audit-invariants/SKILL.md",
"action": "edit",
"tokens": 253,
"at": "2026-07-18T14:30:19.188Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/CLAUDE.md",
"action": "edit",
"tokens": 191,
"at": "2026-07-18T14:43:11.203Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 775,
"at": "2026-07-18T14:58:10.410Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 882,
"at": "2026-07-18T14:58:26.105Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 422,
"at": "2026-07-18T14:58:38.697Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 270,
"at": "2026-07-18T14:58:48.634Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha_convergence.c",
"action": "edit",
"tokens": 2655,
"at": "2026-07-18T14:59:30.511Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/config_repl.c",
"action": "edit",
"tokens": 2138,
"at": "2026-07-18T15:00:07.188Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 250,
"at": "2026-07-18T15:00:36.307Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 320,
"at": "2026-07-18T15:00:46.737Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 304,
"at": "2026-07-18T15:00:56.406Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 494,
"at": "2026-07-18T15:01:12.624Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 236,
"at": "2026-07-18T15:01:22.509Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 1809,
"at": "2026-07-18T15:02:25.287Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 107,
"at": "2026-07-18T15:02:34.124Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_config_repl.c",
"action": "edit",
"tokens": 2444,
"at": "2026-07-18T15:03:11.995Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_config_repl.c",
"action": "edit",
"tokens": 110,
"at": "2026-07-18T15:03:18.926Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha_convergence.c",
"action": "edit",
"tokens": 724,
"at": "2026-07-18T15:17:20.955Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/config_repl.c",
"action": "edit",
"tokens": 558,
"at": "2026-07-18T15:17:50.872Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 439,
"at": "2026-07-18T15:18:18.097Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 323,
"at": "2026-07-18T15:18:30.804Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 275,
"at": "2026-07-18T15:18:55.056Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.h",
"action": "edit",
"tokens": 152,
"at": "2026-07-18T15:19:01.445Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 251,
"at": "2026-07-18T15:19:58.064Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 142,
"at": "2026-07-18T15:20:06.833Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 109,
"at": "2026-07-18T15:20:20.504Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 81,
"at": "2026-07-18T15:20:32.752Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 42,
"at": "2026-07-18T15:21:08.357Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_demotion_convergence.c",
"action": "edit",
"tokens": 22,
"at": "2026-07-18T15:21:18.258Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_config_repl.c",
"action": "edit",
"tokens": 22,
"at": "2026-07-18T15:21:24.798Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/config_repl.c",
"action": "edit",
"tokens": 169,
"at": "2026-07-18T15:21:43.973Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha_convergence.c",
"action": "edit",
"tokens": 43,
"at": "2026-07-18T15:35:27.573Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/config_repl.c",
"action": "edit",
"tokens": 43,
"at": "2026-07-18T15:35:28.674Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha_convergence.c",
"action": "edit",
"tokens": 16,
"at": "2026-07-18T15:39:24.718Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/config_repl.c",
"action": "edit",
"tokens": 16,
"at": "2026-07-18T15:39:26.410Z"
}
],
"edit_counts": {
@ -161,11 +604,22 @@
"src/lib/testsuite/test_netio_batch_dispatch_uaf.c": 2,
"src/lib/meson.build": 1,
"src/lib/testsuite/test.c": 2,
"reports/jul-13-jul-18.md": 1
"reports/jul-13-jul-18.md": 1,
".kiro/steering/product.md": 1,
".kiro/steering/ha-async-io.md": 1,
".kiro/steering/roadmap.md": 5,
".kiro/steering/testing.md": 1,
".claude/skills/xce-audit-invariants/SKILL.md": 1,
"src/controller/CLAUDE.md": 1,
"src/controller/ha.h": 8,
"src/controller/ha_convergence.c": 4,
"src/controller/config_repl.c": 5,
"src/controller/testsuite/test_demotion_convergence.c": 13,
"src/controller/testsuite/test_config_repl.c": 3
},
"anatomy_hits": 11,
"anatomy_misses": 2,
"repeated_reads_warned": 10,
"anatomy_hits": 41,
"anatomy_misses": 9,
"repeated_reads_warned": 108,
"cerebrum_warnings": 0,
"stop_count": 9
"stop_count": 22
}

View file

@ -5139,3 +5139,61 @@
| 06:24 | Session end: 13 writes across 6 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 11 reads | ~190470 tok |
| 06:30 | Created reports/jul-13-jul-18.md | — | ~4180 |
| 06:30 | Session end: 14 writes across 7 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 13 reads | ~197259 tok |
| 07:24 | Edited .kiro/steering/product.md | modified backlog() | ~385 |
| 07:24 | Edited .kiro/steering/ha-async-io.md | 1→3 lines | ~384 |
| 07:25 | Edited .kiro/steering/roadmap.md | 3→3 lines | ~559 |
| 07:25 | Edited .kiro/steering/roadmap.md | inline fix | ~279 |
| 07:25 | Edited .kiro/steering/roadmap.md | inline fix | ~131 |
| 07:25 | Edited .kiro/steering/roadmap.md | inline fix | ~185 |
| 07:25 | Edited .kiro/steering/roadmap.md | "/kiro-discovery" → "s boot-replay fence gap, " | ~151 |
| 07:26 | Edited .kiro/steering/testing.md | 1→5 lines | ~214 |
| 07:26 | Session end: 22 writes across 11 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 18 reads | ~208754 tok |
| 07:30 | Edited .claude/skills/xce-audit-invariants/SKILL.md | expanded (+10 lines) | ~236 |
| 07:30 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 19 reads | ~211133 tok |
| 07:33 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 22 reads | ~286690 tok |
| 07:38 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 39 reads | ~429741 tok |
| 07:39 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 40 reads | ~453771 tok |
| 07:39 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 40 reads | ~453771 tok |
| 07:40 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 40 reads | ~453771 tok |
| 07:41 | Session end: 23 writes across 12 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 40 reads | ~453771 tok |
| 07:43 | Edited src/controller/CLAUDE.md | modified XCE_TESTING() | ~178 |
| 07:43 | Session end: 24 writes across 13 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 40 reads | ~453962 tok |
| 07:58 | Edited src/controller/ha.h | expanded (+20 lines) | ~723 |
| 07:58 | Edited src/controller/ha.h | modified is() | ~823 |
| 07:58 | Edited src/controller/ha.h | modified XCE_TESTING() | ~394 |
| 07:58 | Edited src/controller/ha.h | 7→12 lines | ~252 |
| 07:59 | Edited src/controller/ha_convergence.c | added 5 condition(s) | ~2478 |
| 08:00 | Edited src/controller/config_repl.c | added 6 condition(s) | ~1995 |
| 08:00 | Edited src/controller/testsuite/test_demotion_convergence.c | 10→12 lines | ~233 |
| 08:00 | Edited src/controller/testsuite/test_demotion_convergence.c | modified hands() | ~299 |
| 08:00 | Edited src/controller/testsuite/test_demotion_convergence.c | added 1 condition(s) | ~284 |
| 08:01 | Edited src/controller/testsuite/test_demotion_convergence.c | modified Assert() | ~461 |
| 08:01 | Edited src/controller/testsuite/test_demotion_convergence.c | expanded (+7 lines) | ~221 |
| 08:02 | Edited src/controller/testsuite/test_demotion_convergence.c | added 4 condition(s) | ~1688 |
| 08:02 | Edited src/controller/testsuite/test_demotion_convergence.c | expanded (+6 lines) | ~100 |
| 08:03 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~2281 |
| 08:03 | Edited src/controller/testsuite/test_config_repl.c | 3→7 lines | ~103 |
| 08:17 | Edited src/controller/ha_convergence.c | modified hands() | ~675 |
| 08:17 | Edited src/controller/config_repl.c | modified hands() | ~520 |
| 08:18 | Edited src/controller/ha.h | modified OFF() | ~410 |
| 08:18 | Edited src/controller/ha.h | modified itself() | ~302 |
| 08:18 | Edited src/controller/ha.h | modified XCE_TESTING() | ~257 |
| 08:19 | Edited src/controller/ha.h | 5→7 lines | ~142 |
| 08:19 | Edited src/controller/testsuite/test_demotion_convergence.c | 8→12 lines | ~234 |
| 08:20 | Edited src/controller/testsuite/test_demotion_convergence.c | 5→7 lines | ~132 |
| 08:20 | Edited src/controller/testsuite/test_demotion_convergence.c | 5→6 lines | ~102 |
| 08:20 | Edited src/controller/testsuite/test_demotion_convergence.c | 4→4 lines | ~75 |
| 08:21 | Edited src/controller/testsuite/test_demotion_convergence.c | modified dcb_gated_restart_hook() | ~39 |
| 08:21 | Edited src/controller/testsuite/test_demotion_convergence.c | 1 → 0 | ~20 |
| 08:21 | Edited src/controller/testsuite/test_config_repl.c | 1 → 0 | ~20 |
| 08:21 | Edited src/controller/config_repl.c | modified does() | ~158 |
| 15:31 | Fixed 2 async-io thread-role violations (bug-1450 demotion, bug-1451 catchup) — slow-lane offload pattern; found+fixed a NEW hang from defaulting offload flags ON (broke test_convergence_verification.c); flags now OFF by default | ha_convergence.c, config_repl.c, ha.h, test_demotion_convergence.c, test_config_repl.c | 976 Tests 0 Failures 0 Ignored (direct binary run, no hang) | ~180000 |
| 08:35 | Edited src/controller/ha_convergence.c | inline fix | ~40 |
| 08:35 | Edited src/controller/config_repl.c | inline fix | ~40 |
| 08:39 | Edited src/controller/ha_convergence.c | inline fix | ~15 |
| 08:39 | Edited src/controller/config_repl.c | inline fix | ~15 |
| 08:45 | Session end: 57 writes across 18 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 50 reads | ~644687 tok |
| 21:42 | Session end: 57 writes across 18 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 50 reads | ~644687 tok |
| 22:31 | Session end: 57 writes across 18 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 50 reads | ~644687 tok |
| 08:50 | /xce-audit-invariants controller 10 complete: 6-dim parallel audit (4 dims clean, dim1 found+fixed bug-1450/1451, dim6=new lifetime-batch dimension added post-bug-1447, 0 findings), Step 4 live regression chaotic+sequential 10-round both PASS (33/36 jobs, zero loss). Found bug-1452 (real SEGV, not just hang) while temporarily flipping offload flags ON for live validation — reverted to committed OFF default, logged for future flip-to-ON audit. | ha-convergence.c, config_repl.c, buglog.json, SKILL.md, 4 steering files | GO — 2 real violations fixed, 1 new bug discovered+deferred, live regression clean | ~large |
| 22:36 | Session end: 57 writes across 18 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 50 reads | ~644687 tok |

File diff suppressed because it is too large Load diff