chore(wolf): session state trail

This commit is contained in:
Rongsong Shen 2026-07-12 04:25:53 +00:00
parent 3ef05da8a3
commit 976a6c956b
5 changed files with 1529 additions and 23 deletions

View file

@ -1,7 +1,7 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-07-10T13:48:28.439Z
> Files: 630 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-07-11T13:58:37.442Z
> Files: 637 tracked | Anatomy hits: 0 | Misses: 0
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/5e55cf26-d05d-4a2e-8196-d2c70e0c063c/scratchpad/
@ -399,9 +399,11 @@
## .kiro/specs/ha-elect-ctx-lifecycle/
- `brief.md` — Discovery Brief: ha-elect-ctx-lifecycle (~774 tok)
- `design.md` — Design Document: ha-elect-ctx-lifecycle (~5128 tok)
- `requirements.md` — Requirements Document (~1634 tok)
- `research.md` — Research / Gap Analysis — ha-elect-ctx-lifecycle (~1092 tok)
- `spec.json` (~133 tok)
- `research.md` — Research / Gap Analysis — ha-elect-ctx-lifecycle (~2095 tok)
- `spec.json` (~130 tok)
- `tasks.md` — Implementation Plan: ha-elect-ctx-lifecycle (~1696 tok)
## .kiro/specs/ha-entity-snapshots/
@ -664,6 +666,7 @@
## reports/
- `jul-06-jul-11.md` — Jul 6 → Jul 11 Progress Report (~2311 tok)
- `jun29-jul04-2026.md` — Work Summary: June 30 – July 4, 2026 (~857 tok)
## src/
@ -945,12 +948,12 @@
- `config_repl.c` — define _GNU_SOURCE (~30643 tok)
- `config_repl.h` — pragma once (~6155 tok)
- `ha_convergence.c` — Declares char (~5779 tok)
- `ha.c` — define _GNU_SOURCE (~108102 tok)
- `ha.h` — ifndef HA_H (~26783 tok)
- `ha.c` — define _GNU_SOURCE (~109219 tok)
- `ha.h` — ifndef HA_H (~27080 tok)
- `job.c` — include <errno.h> (~18619 tok)
- `join.c` — include "config.h" (~2795 tok)
- `main.c` — include <argp.h> (~3035 tok)
- `meson.build` (~1580 tok)
- `meson.build` (~1592 tok)
- `myapp.c` — define _GNU_SOURCE (~5215 tok)
- `persist.c` — define _GNU_SOURCE (~65371 tok)
@ -961,7 +964,7 @@
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1986 tok)
- `controller_test.c` — include <argp.h> (~2009 tok)
- `snapdiff.c` — define _GNU_SOURCE (~833 tok)
- `snapdiff.h` — pragma once (~569 tok)
- `snapverify.c` — define _GNU_SOURCE (~2505 tok)
@ -984,6 +987,7 @@
- `test_e2e_hooks.c` — define _GNU_SOURCE (~2184 tok)
- `test_egress_filter.c` — Declares xce_job_t (~5072 tok)
- `test_ha_core_safety.c` — Declares char (~36309 tok)
- `test_ha_elect_ctx_lifecycle.c` — Declares char (~2900 tok)
- `test_ha_liveness.c` — Declares directly (~24748 tok)
- `test_ha_partition.c` (~5459 tok)
- `test_ha_rdhup_integration.c` — Declares char (~7341 tok)
@ -1010,7 +1014,7 @@
- `config.h` — if !defined(_CONFIG_H_) (~1257 tok)
- `delay.h` — if !defined(_DELAY_H_) (~414 tok)
- `myconf.h` — if !defined(_MYCONF_H_) (~560 tok)
- `netio.h` — if !defined(_NETIO_H_) (~2313 tok)
- `netio.h` — if !defined(_NETIO_H_) (~2469 tok)
- `persist.h` — pragma once (~22084 tok)
- `typemap.h` — if !defined(_TYPEMAP_H_) (~5116 tok)
- `wal_types.h` — pragma once (~1996 tok)
@ -1019,7 +1023,8 @@
- `delay.c` — include <errno.h> (~1520 tok)
- `jobj.c` — include <errno.h> (~4867 tok)
- `netio.c` — include <arpa/inet.h> (~8554 tok)
- `meson.build` (~1643 tok)
- `netio.c` — include <arpa/inet.h> (~8709 tok)
- `request.c` — include <errno.h> (~16804 tok)
- `vtaskmgr.c` — include <errno.h> (~3822 tok)
@ -1032,8 +1037,10 @@
## src/lib/testsuite/
- `test_ctrlstatus.c` (~4821 tok)
- `test_netio_on_fini.c` (~1328 tok)
- `test_raft.c` — include <errno.h> (~98108 tok)
- `test_wal_wire.c` — include <errno.h> (~4204 tok)
- `test.c` — include <argp.h> (~1130 tok)
## src/schema/

View file

@ -12,6 +12,7 @@
- Keep test logs for analysis, don't skip intermediate output even if a test hangs
## Key Learnings
- (2026-07-11, controller_unit_test isolated-group + leak-detection runs) `controller_unit_test` supports `--run STR` (repeatable, arg key `-t`) to select one or more TEST()-registered groups by exact function name (matches `argp`-parsed `-t`/`--run`, dispatch via `check_test()`/`TEST()` macro in `controller_test.c`) — use it to isolate a single suite instead of parsing the full-suite log. Full-suite direct runs interleave two PASS-print conventions: older suites only print via Unity's own per-assertion output (no `<file>:<line>:<name>:PASS` banner), while newer TDD suites (apply/HA-prevote family) add a custom `printf("%s: PASS\n", ...)` + banner — so grepping for `../controller/testsuite/<file>.c` banners under-counts groups that don't use the newer convention; don't infer suite pass/fail or crash-truncation from banner presence alone, always check the process exit code. Confirmed via meson (`Ok:1/Fail:0`, one lumped "controller test") AND a clean direct exit(0) with zero grep hits for AddressSanitizer/heap-buffer-overflow/UAF/double-free/FAIL across a 15.7k-line full-suite log — the log's abrupt-looking tail (last group's own internal UNITY_BEGIN/END subtotal printed twice) is normal, not truncation; verify with `echo $?` instead. The historical "test_ha_prevote 18/18" figure from an earlier session was the COMBINED total of `test_ha_prevote` (12 tests, confirmed via `grep -c RUN_TEST` in the .c file) + `test_ha_failover_prevote` (6 tests) — the two are separate TEST() groups in `controller_test.c`, not one suite of 18; report them separately (12/12 + 6/6) to avoid a false "regression" reading.
- (2026-07-08, HA soak test sequential vs chaotic modes) Sequential mode soak validates clean failover lifecycle: 10 rounds, 36 jobs, zero loss, all 3 nodes led primary — deterministic validation of promotion + catch-up + reapply paths working as designed. Chaotic mode deliberately overlaps kills across election/catch-up windows to surface edge cases: rounds 1-4 pass verification, round 5 hits primary-discovery lag (bug-854 root-caused earlier, excluded from verdict), rounds 6-7 stall from election convergence lag + duplicate-node-guard (bug-1418, pre-existing guard). The election stall in chaotic round 6 is likely compound: (a) tight overlapping kill cadence outpaces election+restart timing (heartbeat-timeout bounds ~500ms but nix shell spinup adds latency), (b) a node restarted but not fully converged rejoins before the prior kill round's primary finishes its term tenure. Sequential mode PASS verdict is solid baseline. Chaotic mode needs either slower kill spacing OR a real-fs host deploy (current sandbox has overlayfs fdatasync EIO bug-924) to distinguish fault-tolerance (still holds) from environment-induced timing collapse.
- (2026-07-04, raft-apply-decoupling task 6.1 round 2 — job-state merge-staleness fix) Round 1's `apply_committed_entry` wired the live pump but left a real corruption: `record_job_fail_fact`/`record_job_update_fact` (job.c) precomputed a FINAL absolute state at RECEIPT time (a scratch-copy merge for FAIL, an inline backstop+PDONE merge for postdone) and baked it into `wal_jobstate_t.state`; under deferred apply, a second fact for the same job (e.g. FINISH) could apply BETWEEN receipt and this fact's own apply, making the baked snapshot stale — a POST_EXEC-stage FAIL received while a job is still `RUN` (FINISH not yet applied) bakes `RUN|EXIT|PERR`, and when FINISH applies first (`DONE`) then FAIL's blind absolute overwrite clobbers it back to `RUN|EXIT|PERR` — DONE silently lost. FIX PATTERN: `wal_jobstate_t._pad` (previously always-zero, unused) is repurposed as a discriminator — `_pad==0` still means "legacy absolute state, safe" (job_ctrl's mark_job_fail/mark_job_update paths, which mutate+capture state synchronously under one lock, no staleness window); a nonzero `_pad` (`WAL_JOB_FAIL_STAGE_FLAG|stage` for FAIL, `WAL_JOB_UPDATE_POSTDONE_MARK` for postdone) marks "recompute at apply time against ACTUAL current state" — `apply_committed_entry` (persist.c) dispatches on this bit per WAL type, calling the REAL `apply_job_fail_transition`/new `apply_job_postdone_transition` (job.c, mirrors the existing guard-core family) instead of `apply_job_update_transition`'s blind overwrite. New low-level `persist_submit_job_fact_ex(p, type, jobid, state, pad)` added as the primitive; `persist_submit_job_fact` becomes a pad=0 thin wrapper — ZERO existing call sites (including ~20 in test_commit_barrier_reply.c) needed touching. GENERAL PATTERN for "wire format has 2 producers, one safe one not": don't change the safe producer's shape — add a discriminator field/flag so the consumer can tell them apart, and make "0"/"absent" mean exactly the old/safe behavior so every existing writer keeps working unchanged.
- (2026-07-04, same task) Fixing a wire-format bug INSIDE a task's own scope can break pre-existing tests that assert the wire format directly (not just behavior) — 2 tests in test_commit_barrier_reply.c (`test_guard_forward_update_on_terminal_preserved`, `test_guard_fail_after_finish_preserves_perr_merge`) read the raw on-disk `wal_jobstate_t.state` via a helper (`cbrp_count_jobstate_entries`) and asserted DONE/PDONE/PERR bits directly on it — exactly the field this fix stops populating for FAIL/postdone. These are NOT the "old apply-before-commit ordering" tests task 7.3 owns (a different axis — ordering vs wire-format) — they are a DIRECT, in-scope, mechanical consequence of the wire-format fix itself, so I fixed them here rather than deferring: extended the helper to a `_ex` variant returning `_pad` too (existing helper becomes a thin wrapper, zero other call sites touched — same pattern as persist_submit_job_fact_ex), changed the wire-level assertions to check the new `_pad` marker, and added a direct call to the guard core (`apply_job_fail_transition`/`apply_job_postdone_transition`) on a scratch `job_t` to keep pinning the actual merge semantics the test was written to protect. A 3rd test (`test_fact_state_unchanged_throughout_quorum_loss_outage`) had the identical `st_upd` bit-check for postdone; fixed the same way (already had a `(void)st_fail` precedent in the same file for exactly this "order-dependent, can't check at receipt time" reasoning — extended it to `st_upd`/`pad_upd`). Lesson: when a task's fix legitimately changes what a durable record contains, grep for every DIRECT byte/field-level test assertion on that record (not just behavioral tests) before declaring the regression sweep clean — `grep cbrp_count_jobstate_entries` found them, `--run test_apply_pump` alone would not have.

View file

@ -1,12 +1,21 @@
{
"session_id": "session-2026-07-11-0400",
"started": "2026-07-11T11:00:18.263Z",
"session_id": "session-2026-07-11-0654",
"started": "2026-07-11T13:54:32.676Z",
"files_read": {},
"files_written": [],
"edit_counts": {},
"files_written": [
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-elect-ctx-lifecycle/tasks.md",
"action": "edit",
"tokens": 117,
"at": "2026-07-11T13:58:37.453Z"
}
],
"edit_counts": {
".kiro/specs/ha-elect-ctx-lifecycle/tasks.md": 1
},
"anatomy_hits": 0,
"anatomy_misses": 0,
"repeated_reads_warned": 0,
"cerebrum_warnings": 0,
"stop_count": 1
"stop_count": 3
}

View file

@ -4371,3 +4371,72 @@
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 05:25 | Created .kiro/specs/ha-elect-ctx-lifecycle/design.md | — | ~5498 |
| 05:25 | Edited .kiro/specs/ha-elect-ctx-lifecycle/research.md | expanded (+17 lines) | ~1105 |
| 05:25 | Edited .kiro/specs/ha-elect-ctx-lifecycle/spec.json | 12→12 lines | ~74 |
| 05:27 | Session end: 3 writes across 3 files (design.md, research.md, spec.json) | 3 reads | ~10007 tok |
| 05:30 | Edited .kiro/specs/ha-elect-ctx-lifecycle/design.md | 18→18 lines | ~223 |
| 05:30 | Edited .kiro/specs/ha-elect-ctx-lifecycle/design.md | inline fix | ~29 |
| 05:31 | Edited .kiro/specs/ha-elect-ctx-lifecycle/spec.json | 4→4 lines | ~20 |
| 05:32 | Session end: 6 writes across 3 files (design.md, research.md, spec.json) | 3 reads | ~10295 tok |
| 05:33 | Created reports/jul-06-jul-11.md | — | ~2465 |
| 05:33 | Session end: 7 writes across 4 files (design.md, research.md, spec.json, jul-06-jul-11.md) | 4 reads | ~18064 tok |
| 05:38 | Created .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | — | ~1810 |
| 05:38 | Edited .kiro/specs/ha-elect-ctx-lifecycle/spec.json | 17→17 lines | ~94 |
| 06:00 | Edited .kiro/specs/ha-elect-ctx-lifecycle/spec.json | 6→6 lines | ~31 |
| 06:02 | Created src/lib/testsuite/test_netio_on_fini.c | — | ~1328 |
| 06:02 | Edited src/lib/meson.build | 1→2 lines | ~30 |
| 06:03 | Edited src/include/netio.h | expanded (+7 lines) | ~166 |
| 06:03 | Edited src/include/netio.h | 1→2 lines | ~45 |
| 06:03 | Edited src/lib/netio.c | added 1 condition(s) | ~122 |
| 06:03 | Edited src/lib/netio.c | added 1 condition(s) | ~120 |
| 06:04 | Session end: 16 writes across 9 files (design.md, research.md, spec.json, jul-06-jul-11.md, tasks.md) | 9 reads | ~32934 tok |
| 06:07 | Edited src/lib/testsuite/test.c | 1→2 lines | ~22 |
| 06:07 | Edited src/lib/testsuite/test.c | 1→2 lines | ~18 |
| 06:07 | Session end: 18 writes across 10 files (design.md, research.md, spec.json, jul-06-jul-11.md, tasks.md) | 10 reads | ~34098 tok |
| 06:10 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | 2→2 lines | ~52 |
| 06:13 | Created src/controller/testsuite/test_ha_elect_ctx_lifecycle.c | — | ~995 |
| 06:13 | Edited src/controller/meson.build | 1→2 lines | ~21 |
| 06:13 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~22 |
| 06:13 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~18 |
| 06:13 | Edited src/controller/ha.h | expanded (+12 lines) | ~288 |
| 06:14 | Edited src/controller/ha.c | expanded (+10 lines) | ~208 |
| 06:14 | Edited src/controller/ha.c | added 3 condition(s) | ~498 |
| 06:15 | Edited src/controller/ha.c | removed 12 lines | ~33 |
| 06:15 | Edited src/controller/ha.h | modified ha_elect_ctx_release_for_test() | ~160 |
| 06:16 | Session end: 28 writes across 14 files (design.md, research.md, spec.json, jul-06-jul-11.md, tasks.md) | 14 reads | ~175863 tok |
| 06:23 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | 2→2 lines | ~39 |
| 06:24 | Edited src/controller/ha.c | modified if() | ~333 |
| 06:24 | Edited src/controller/ha.c | modified if() | ~254 |
| 06:25 | Edited src/controller/ha.c | modified if() | ~308 |
| 06:25 | Edited src/controller/ha.c | modified self_tag() | ~75 |
| 06:25 | Edited src/controller/ha.c | 2→7 lines | ~123 |
| 06:25 | Edited src/controller/ha.c | modified if() | ~301 |
| 06:26 | Session end: 35 writes across 14 files (design.md, research.md, spec.json, jul-06-jul-11.md, tasks.md) | 15 reads | ~177570 tok |
| 06:34 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | inline fix | ~32 |
| 06:34 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | inline fix | ~31 |
| 06:34 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | inline fix | ~26 |
| 06:39 | Edited src/controller/ha.c | modified XCE_TESTING() | ~184 |
| 06:39 | Edited src/controller/testsuite/test_ha_elect_ctx_lifecycle.c | expanded (+19 lines) | ~303 |
| 06:41 | Edited src/controller/testsuite/test_ha_elect_ctx_lifecycle.c | added 3 condition(s) | ~1822 |
| 06:41 | Edited src/controller/testsuite/test_ha_elect_ctx_lifecycle.c | below() → itself() | ~253 |
| 06:42 | Edited src/controller/testsuite/test_ha_elect_ctx_lifecycle.c | removed 5 lines | ~16 |
| 06:42 | Session end: 43 writes across 14 files (design.md, research.md, spec.json, jul-06-jul-11.md, tasks.md) | 19 reads | ~183582 tok |
| 13:52 | Verified controller ASAN build + tests: 5/5 test_ha_elect_ctx_lifecycle pass, LSan-isolated run clean (0 leaks) | src/controller/ha.c, src/controller/testsuite/test_ha_elect_ctx_lifecycle.c, src/xce.build/controller/controller_unit_test | PASS, no regressions | ~4200 |
| 06:51 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | 2→2 lines | ~48 |
| 06:51 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | inline fix | ~30 |
| 06:51 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | inline fix | ~35 |
| 06:51 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | inline fix | ~23 |
## Session: 2026-07-11 06:53
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-07-11 06:54
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 06:58 | Edited .kiro/specs/ha-elect-ctx-lifecycle/tasks.md | 5→5 lines | ~109 |
| 06:58 | Session end: 1 writes across 1 files (tasks.md) | 0 reads | ~117 tok |
| 21:23 | Session end: 1 writes across 1 files (tasks.md) | 0 reads | ~117 tok |

File diff suppressed because it is too large Load diff