chore(wolf): sync anatomy/cerebrum/memory/hooks/token-ledger bookkeeping
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
parent
ac46bfa649
commit
a16776bdbd
5 changed files with 3228 additions and 18 deletions
|
|
@ -1,7 +1,7 @@
|
|||
# anatomy.md
|
||||
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-08-25T14:45:35.964Z
|
||||
> Files: 741 tracked | Anatomy hits: 0 | Misses: 0
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-08-25T16:05:07.063Z
|
||||
> Files: 742 tracked | Anatomy hits: 0 | Misses: 0
|
||||
|
||||
## ../../../../../../tmp/
|
||||
|
||||
|
|
@ -962,7 +962,7 @@
|
|||
- `main.c` — build-time-versioning task 3.2 (argp_program_version runtime-composed into static `version_buf[128]`, "My name is..." startup INFO reads that buffer) PLUS dirty-content-fingerprint task 3.2 (2026-08-21): new static `version_display_buf[128]` composed via `xce_compose_version_ex()` (adds `STR(VERSION_DIRTY_FINGERPRINT)`) feeds `argp_program_version` instead; `version_buf`'s composition + the startup INFO log are byte-identical/untouched (Req 3.2). Live-verified: `--version` -> `7ebcba6d-cdb7cc7d`, startup log -> `7ebcba6d-dirty` (~5346 tok)
|
||||
- `meson.build` (~3546 tok)
|
||||
- `misc.c` — include <errno.h> (~1359 tok)
|
||||
- `myapp.c` — define _GNU_SOURCE (~6287 tok)
|
||||
- `myapp.c` — define _GNU_SOURCE (~6277 tok)
|
||||
- `myconf.c` — define _GNU_SOURCE (~1180 tok)
|
||||
- `opget.c` — include "config.h" (~1730 tok)
|
||||
- `opupd.c` — include "config.h" (~714 tok)
|
||||
|
|
@ -973,7 +973,7 @@
|
|||
- `resources.c` — include <errno.h> (~50 tok)
|
||||
- `sched.c` — include <errno.h>; bug-1693 (2026-08-25): `handle_scheduler_leave()` now calls `schedadmin_signal_disconnect()` (cluster.c) BEFORE acquiring `c->sched.lock` (calling after would deadlock against the OPCode_schedAdmin dispatch case, which holds that lock for its whole wait) so a mid-wait scheduler disconnect wakes the pending waiter immediately instead of only at the 5s deadline (~6405 tok)
|
||||
- `slowlane.c` — define _GNU_SOURCE (~2392 tok)
|
||||
- `startup_validation.c` — define _GNU_SOURCE (~1524 tok)
|
||||
- `startup_validation.c` — validate_vcpus/peer_count/cert_files/startup_config/heartbeat_config; bug-1695: also hosts myapp_leader_already_known() (extracted from myapp.c so it links into controller_unit_test, which excludes myapp.c) (~3050 tok)
|
||||
- `task.c` — include <errno.h> (~1376 tok)
|
||||
- `worker.c` — include <errno.h>; bug-1693 (2026-08-25): `handle_worker_leave()` now calls `fetchfile_signal_worker_disconnect(w)` (job.c) right after nulling `w->ep`, so a mid-wait worker disconnect wakes any `on_fetchfile()` waiter proxied to it immediately instead of only at the 10s deadline (~8298 tok)
|
||||
|
||||
|
|
@ -1023,6 +1023,7 @@
|
|||
- `test_fetchfile_disconnect.c` — bug-1693 regression suite (NEW, 2026-08-25): drives job.c's real `fetchfile_signal_worker_disconnect()`/`fetchfile_wait_for_test()` (XCE_TESTING seam) directly with a plain stack `worker_t` (not the full `on_fetchfile()`/real jobmgr+workermgr chain, out of this fix's scope). Proves a worker disconnect wakes a pending fetchfile wait in <1s with `-ECONNRESET` (not the ~5-10s deadline's `-ETIMEDOUT`), and that a disconnect on a DIFFERENT worker never cross-wakes an unrelated waiter (~2197 tok)
|
||||
- `test_ha_durability.c` — Declares char (~5643 tok)
|
||||
- `test_ha_rdhup_integration.c` — Declares char (~8362 tok)
|
||||
- `test_ha_startup.c` — startup validation + dispatch gate + myapp_run building-block tests; bug-1695: 7 new tests for myapp_leader_already_known() (stale-restored-term regression, term-advance/primary-fd/first-boot/voted-for-name/self-vote cases) (~10750 tok)
|
||||
- `test_job_array_sched_push.c` — job-array-single-wal-record task 5.1: standalone binary (`sched_push_unit_test`) linking real `sched.c`+`dataex.c`+`egress.c` to test `sched_job_new_fun`/`xcesched_push_jobarray`; bug-1693 (2026-08-25): added a one-line link-satisfying `schedadmin_signal_disconnect()` stub (mirrors the file's existing `check_worker_()` stub) since `sched.c`'s `handle_scheduler_leave()` now calls the real one, defined in cluster.c, which this binary doesn't link (~2863 tok)
|
||||
- `test_job_dispatch_fail_count.c` (~1002 tok)
|
||||
- `test_job_dispatch_fail_count.c` — scheduler-dispatch-commit task 1.3: proves job_t.dispatch_fail_count (xce/types.h wrapper struct, new uint32_t next to submit_lsn) is a plain, directly settable/incrementable/resettable in-memory-only field; bare stack-local job_t, no jobmgr/RTTI, mirrors test_persist_types.c's test_slot_key_t_fields precedent. Data-model only -- no dispatch_a_job_() wiring (~1600 tok)
|
||||
|
|
|
|||
|
|
@ -128,6 +128,11 @@
|
|||
- **WAL field-parity fix family (bug-1678/1687/1689/1690/1691/1692, controller persist.c/job.c)**: when a hook-driven WAL fact (payload built inside `persist_job_hook_fun`, not a direct `persist_submit_job_*_fact()` call site) needs a new field added to survive replay: (1) check whether the field's own release function uses `FREEUP()`/`free()` (plain malloc/strdup/calloc — safe to free-and-replace, e.g. `JobAlloc`/`JobPIDInfo`/`JobPendReason`) vs `dereference()` (xstring/RTTI-backed, e.g. `ckptinfo`) vs "can be the SAME live object the apply pump re-applies moments later" (needs a MERGE, not free-and-replace — only `WAL_JOB_RUSAGE`'s `run_usage` needed this, because `update_host_runusage()` mutates it in place before the WAL fact is even appended); (2) check whether adoption must be gated on `JOB_TRANSITION_APPLIED` or unconditional — if the setter never changes `job.state` (true for CKPT/RUSAGE/PREASON, false for START/FINISH/DISPATCH), gating makes the fix dead code since every real replay hits `JOB_TRANSITION_NOOP_DUPLICATE`; (3) **check whether the field has a genuine CLEAR-to-NULL operation** (only `pend_reasons`/`do_clr_preason` in this family so far) — if so, "trailing bytes present or absent" (bug-1690/1691's convention) cannot distinguish a legitimate clear from an old-format payload; needs an explicit presence byte instead. Always re-verify (2) and (3) from the real setter code, never assume the newest precedent's shape applies unchanged.
|
||||
- **capnp-c's generated decode path does not validate content, only bounds via `capn_getp`/`capn_resolve`**: passing well-formed-length-but-semantically-garbage bytes to a `decode_X()`/`read_X()` pair (e.g. `decode_JobPendReason`) does NOT return NULL or error — it silently produces a zero-valued struct. A "decode failure returns NULL" defensive branch (`decode_X() == NULL`) is only reachable via allocation failure or an upstream length/bounds guard (e.g. `payload_len` too short to even attempt the read) — never via corrupt-but-present bytes. Don't write a regression test assuming garbadge bytes will fail decode; it will silently pass through and the test either false-passes or fails for a confusing, misleading reason (`Expected 4 Was 0` — the "corrupted" value decoded as an all-zero struct, not a NULL).
|
||||
|
||||
- **(2026-08-25, bug-1695) `meson.build`'s `controller_unit_test_src` deliberately does NOT link `myapp.c` (or `main.c`) — only `startup_validation.c`, `ha.c`, `cluster.c`, `persist.c`, etc.** `myapp_run()` is documented as an "untestable integration entry point" (test_ha_startup.c's own Task 5.1 comment) requiring a live network socket, so it's excluded from the unit-test binary entirely. If you need to unit-test logic that currently lives inline inside `myapp_run()`, do NOT just add a non-static function to `myapp.c` and forward-declare it in a test file — it will compile but fail to LINK (`undefined reference`) because `myapp.c` itself never gets compiled into `controller_unit_test`. Instead, extract the logic into `startup_validation.c` (already linked into both `xce_controller`/`main.c` AND `controller_unit_test`, and explicitly documented as existing "for use in main.c and the controller unit test suite") and have `myapp.c` forward-declare + call it — exactly mirroring the pre-existing `validate_startup_config()` pattern (defined in `startup_validation.c`, forward-declared at the top of `myapp.c`, called from `myapp_run()`). Confirmed via a real RED/GREEN round-trip: extracting into `myapp.c` first produced a clean compile + link FAILURE across all of `controller_unit_test`'s ~90 test files; moving the same function body to `startup_validation.c` fixed it with zero other changes.
|
||||
- **(2026-08-25, bug-1695) `ha_state_t` has TWO separate term fields that look similar but are written by different code paths and must not be conflated**: `ha->term` (top-level, `_Atomic uint64_t`, `ha.h:157`) is the adapter's own copy, written by `ha_restore_term()` at boot and by `raft_cb_persist()` on every `on_persist` callback; `ha->raft.term` (nested inside the embedded `raft_state_t`, `ha.h:198`/`raft.h:29`) is the raft LIBRARY's own internal term, seeded ONCE at `ha_init_raft()` time from `ha->term` (`rcfg.initial_term = atomic_load(&ha->term)`) and thereafter updated only by the library itself (`raft_start_election`/`raft_on_vote_request`/`raft_on_append_entries_request`, all via the SAME `on_persist` callback that also updates `ha->term`) — so they stay in sync after boot, but the ONLY correct place to read "the term as of right after this boot's restore" is `ha->raft.term` captured immediately after `ha_init_raft()` returns (not `ha->term`, and not before `ha_init_raft()` runs). myapp_run's bootstrap check reads `ha->raft.term` specifically (not `ha->term`) — a fix that snapshotted the wrong field would silently do nothing.
|
||||
- **(2026-08-25, bug-1695) A `--chaotic` HA soak's per-round "wait_for_new_primary window" log line is a WEAK signal on its own — always cross-check the raw controller logs (`grep -n "CTRL_PRIMARY\|CTRL_STANDBY\|PreVote round" logs/ctrl-*.log`) before trusting it.** The chaotic driver's own per-round polling window (~90s, 30×3s probes) can time out ("NO new primary within window") even when a genuinely NEW node DID win a real election shortly after — the log line only reflects whether the driver's OWN bounded poll happened to catch it, not whether the underlying HA mechanism worked. In this session's soak, the fixed run showed "NO new primary within window" on 5/6 rounds (superficially similar to the unfixed baseline's 6/6) while the RAW logs proved something structurally different was happening: term genuinely progressing 1→2→3→4→7 and a different node (`ctrl-2`) achieving a real, log-verified `CTRL_PRIMARY` transition via a won PreVote — vs. the baseline where term was frozen at a single value for the entire run and NEITHER peer ever showed any PreVote/election activity at all, anywhere. **The A/B technique that resolved this ambiguity**: `git stash push -- <the exact files you changed>` (reverts just those files, keeps everything else including the same freshly-built binaries' surrounding tree state), rebuild, re-run the IDENTICAL soak invocation against a fresh scratch dir, compare raw logs side by side, then `git stash pop` to restore the fix. Far more conclusive than trusting either run's summary line alone.
|
||||
- **(2026-08-25, bug-1695) A chaotic-soak `expect-job`/`verify_classify` "COMMITTED JOB LOSS" verdict is NOT self-verifying — cross-check the raw WAL with `xce_test`'s `waldump <data-dir>` REPL command (built into `xce_test`, not a separate binary) before treating it as a real controller data-loss bug.** `verify_classify` reads through the apply-pump/materialized-jobmgr layer (an RPC-level `expect-job` query), which per already-documented bug-1487 can legitimately lag behind the raw committed WAL for several seconds after a genuinely-new election (the new leader's OWN-term commit-barrier gate, `ha_arm_own_term_gate`, requires a fresh majority ack for its own-term no-op before folding in prior-term entries) — a RETRY-EXHAUSTED "not found" there does not by itself prove the entry is actually missing from the durable log. `walcompare`'s `committed_divergence` count (also built into `xce_test`) is the AUTHORITATIVE data-safety signal — it stayed 0 in this session's soak even while `verify_classify` counted 6 losses, and a direct `waldump` of all 3 nodes' `data/wal/seg-*.wal` files confirmed the ONE specific jobid in question genuinely does not appear on ANY node's WAL at all (not divergent between nodes — universally absent), tracing it to the cluster's very first cold-boot config-bootstrap window rather than anything this specific fix touched.
|
||||
|
||||
## Do-Not-Repeat
|
||||
- **When translating a `fp_ok`-tracked, per-step-gated failure-detection pattern (like `xce-build.sh`'s dirty-fingerprint computation) into a Make `$(shell sh -c '...')` one-liner, do NOT collapse two commands into `{ cmd1; cmd2; } >"$$fp_tmp"` and gate the whole group with one `if`** — a POSIX sequential group's exit status reflects only its LAST command, so `cmd1`'s failure is silently discarded whenever `cmd2` still succeeds (2026-08-21, bug-1612: `git diff HEAD` failing was swallowed because the trailing `git ls-files | sort | xargs cat` pipe still exited 0, producing a fabricated fingerprint instead of degrading to empty). Give each step its own explicit `if ! step; then fp_ok=0; fi` (or `&&`-chain), exactly mirroring the reference script's discipline, even inside a cramped one-liner. Caught by adversarial code review with a targeted failure-injection test (wrapper `git` on `PATH` that fails only the targeted subcommand) — that test pattern is the reusable way to verify this class of fix.
|
||||
- Don't run `clang-format -i` on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via `git checkout HEAD -- <file>` on a file with other uncommitted work destroys that work irrecoverably.
|
||||
|
|
@ -191,3 +196,4 @@
|
|||
- **When offloading a newly-discovered delaymgr/iomgr blocking-I/O violation, "vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK)" is NOT automatically the right destination even though it's this codebase's most commonly cited fix-pattern name — check whether the SAME pool already carries a lock the new work also needs, via a DIFFERENT already-offloaded caller** (bug-1694, checkpoint_task_fn offload, 2026-08-25). This codebase actually has TWO distinct offload destinations, easy to conflate: the vtaskmgr N-thread pool (`vtask_enq_classified`, VTASK_LIVE_BLOCK/VTASK_REQ_BLOCK — used for *bounded* handler work, e.g. `persist_apply_pump_kick`'s apply-pump drain and `commit.c`'s `dispatch_continuation`) vs. the dedicated SlowLane (`slowlane_submit`, `c->slowlane`, `n_workers` separate pthreads — used for genuinely *long/unbounded* blocking I/O: snapshot send, config-push, reconnect, catch-up-streaming, demotion-truncate). `ha-async-io.md`'s own table already states this split ("vtaskmgr must NOT do: long/unbounded blocking I/O ... starves the pool") but it's easy to miss which category new work falls into. Concretely: `persist_apply_pump_run()` (the live WAL-apply walk) holds `apply_pump_lock` across its ENTIRE walk and is ITSELF dispatched onto the vtask pool via `persist_apply_pump_kick()`'s `vtask_enq_classified(..., VTASK_LIVE_BLOCK)` — so any OTHER unbounded, lock-holding work (checkpoint's `write_snapshot()`, also under `apply_pump_lock`) put on that SAME pool would risk starving it (a worker parked on the lock for the whole snapshot-write duration, stealing capacity from commit-continuation/apply-pump liveness work) — not a classic deadlock, but the exact "starves the pool" failure mode the split exists to prevent. The fix used the slow lane instead, exactly mirroring `ha_snapshot_do_send()`'s pre-existing (already-in-production, default-ON) call to the SAME function (`persist_checkpoint()`) from a slow-lane thread — i.e. the safest new offload destination is often "wherever an already-proven-safe sibling call already runs the identical function from," not a fresh pattern match on a fix-recipe's literal wording. **General technique: before picking an offload destination, grep every existing caller of the SAME blocking function/lock for its current thread context — if a sibling caller already runs it safely from candidate destination X, reuse X; if a DIFFERENT already-offloaded consumer of the same lock lives on candidate destination Y, that's evidence AGAINST Y.**
|
||||
- **A buglog entry's own "root cause: NOT YET ROOT-CAUSED" (or similarly hedged) status is a real signal to re-derive the causal claim from primary evidence before citing it as settled, especially when later re-occurrences of the SAME symptom show it firing on completely unrelated diffs.** bug-1429 (cited by `config_repl.c`/`ha_convergence.c`'s catchup/demotion offload-flag comments as "the vtask pool that deadlocked controller_unit_test on 2026-07-10") is itself logged as never root-caused, and its OWN later occurrences (2026-08-20, 2026-08-23) reproduced the identical TIMEOUT/hang signature on diffs that touched neither the vtask offload code nor persist.c's checkpoint path at all — "strongly corroborates this is a pre-existing, systemic, timing/order-dependent hang in the controller_unit_test harness," per the buglog's own later entries, not proven caused by VTASK_LIVE_BLOCK dispatch specifically. Don't repeat a downstream comment's causal claim as fact in a new design doc/comment without checking the buglog entry it cites — the comment can be more confident than the evidence it's citing.
|
||||
- **`controller_unit_test` (the main shared meson.build `controller_unit_test_src` binary) does NOT link the real `sched.c`/`dataex.c` at all** (bug-1693, 2026-08-25) — `sched.c`'s own `on_scheduler_join()`/`collect_confvars()` and `dataex.c`'s `notify()` collide at link time with fake local definitions several OTHER files in that same binary already provide (`test_commit_barrier_reply.c`, `test_worker_join.c`), so `test_config_repl.c` instead ships a trivial no-op `handle_scheduler_leave(cluster_t *c) { (void)c; return 0; }` stub purely to satisfy the linker. **Any new/changed behavior added to a real `sched.c` function is therefore SILENTLY UNTESTED by any `controller_unit_test`-based test that calls that function by name** — the call resolves to the stub, not the real implementation, with no compile/link error to flag the mismatch (confirmed empirically: a new regression test asserting sub-second wake timing after calling `handle_scheduler_leave()` passed compilation and linking fine, then failed by taking the full 5s deadline — the stub had silently swallowed the call). `nm '<obj>.o' | grep ' T <symbol>$'` across every `.o` in the target's build dir is the fast way to confirm which TU actually provides a given symbol in a given binary before trusting a test's result. The two standalone binaries that DO link the real `sched.c` (`sched_push_unit_test`, `sched_worker_change_unit_test` — see their own `meson.build` comments for the collision rationale) don't link `cluster.c`, so neither binary alone can exercise a fix that spans both files for real; bug-1693's fix split verification across both (a direct-call test against `cluster.c`'s real new function in `controller_unit_test`, plus a small counting-spy test in `test_sched_worker_change.c` proving `sched.c`'s real caller invokes it) rather than inventing a third binary that links both (would need to duplicate most of `controller_unit_test_src`'s already-large dependency list to satisfy `cluster.c`'s own huge fan-out). General lesson: when a fix spans a function in `sched.c`/`dataex.c` AND a function in any of the ~15 other files real-linked into `controller_unit_test_src`, check with `nm`/grep for a same-name fake stub in a sibling test file in that binary BEFORE trusting a test that calls the `sched.c`/`dataex.c` side by name — it may be exercising a stub, not your change.
|
||||
- **(2026-08-25, bug-1695) A bootstrap "is a leader already known" fallback check must compare a term against a snapshot taken at THIS boot's start, never against a bare constant like `> 0`, whenever the compared value is restored from durable on-disk state that only ever increases (Raft terms, monotonic counters, etc.).** `myapp_run()`'s `raft.term > 0` fallback was trivially true forever after the cluster's first-ever election because `ha_restore_term()` restores a persisted, never-decreasing term at every boot. Fixed by capturing `term_at_boot = ha->raft.term` immediately after `ha_init_raft()` seeds it (right after restore, strictly before any new activity could legitimately advance it) and comparing `> term_at_boot` instead of `> 0` — the general pattern: "is X known/true" checks against monotonic restored state should ask "did X change SINCE I booted," not "is X nonzero," whenever a nonzero-forever steady state is a normal, expected outcome of prior successful operation (not an error state).
|
||||
|
|
|
|||
|
|
@ -328,8 +328,8 @@
|
|||
"first_read": "2026-08-25T09:29:39.634Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/meson.build": {
|
||||
"count": 7,
|
||||
"tokens": 0,
|
||||
"count": 9,
|
||||
"tokens": 3546,
|
||||
"first_read": "2026-08-25T09:31:05.167Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/testsuite/controller_test.c": {
|
||||
|
|
@ -446,6 +446,21 @@
|
|||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-08-25T14:31:21.226Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c": {
|
||||
"count": 6,
|
||||
"tokens": 6000,
|
||||
"first_read": "2026-08-25T15:03:41.642Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/testsuite/test_ha_startup.c": {
|
||||
"count": 3,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-08-25T15:12:09.549Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/startup_validation.c": {
|
||||
"count": 2,
|
||||
"tokens": 3132,
|
||||
"first_read": "2026-08-25T15:19:53.323Z"
|
||||
}
|
||||
},
|
||||
"files_written": [
|
||||
|
|
@ -1642,6 +1657,78 @@
|
|||
"action": "edit",
|
||||
"tokens": 61,
|
||||
"at": "2026-08-25T14:45:35.971Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 1724,
|
||||
"at": "2026-08-25T15:15:11.594Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 332,
|
||||
"at": "2026-08-25T15:15:22.305Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 222,
|
||||
"at": "2026-08-25T15:15:32.738Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/testsuite/test_ha_startup.c",
|
||||
"action": "edit",
|
||||
"tokens": 188,
|
||||
"at": "2026-08-25T15:15:52.138Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/testsuite/test_ha_startup.c",
|
||||
"action": "edit",
|
||||
"tokens": 2353,
|
||||
"at": "2026-08-25T15:17:20.315Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/testsuite/test_ha_startup.c",
|
||||
"action": "edit",
|
||||
"tokens": 237,
|
||||
"at": "2026-08-25T15:17:31.707Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 72,
|
||||
"at": "2026-08-25T15:20:54.544Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 194,
|
||||
"at": "2026-08-25T15:21:06.913Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/startup_validation.c",
|
||||
"action": "edit",
|
||||
"tokens": 276,
|
||||
"at": "2026-08-25T15:21:21.675Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/startup_validation.c",
|
||||
"action": "edit",
|
||||
"tokens": 1732,
|
||||
"at": "2026-08-25T15:21:48.364Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/startup_validation.c",
|
||||
"action": "edit",
|
||||
"tokens": 92,
|
||||
"at": "2026-08-25T15:26:07.728Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/scheduler-framework/src/controller/myapp.c",
|
||||
"action": "edit",
|
||||
"tokens": 275,
|
||||
"at": "2026-08-25T16:05:07.071Z"
|
||||
}
|
||||
],
|
||||
"edit_counts": {
|
||||
|
|
@ -1689,11 +1776,14 @@
|
|||
"src/controller/testsuite/test_fetchfile_disconnect.c": 1,
|
||||
"src/controller/testsuite/test_job_array_sched_push.c": 1,
|
||||
"src/controller/testsuite/test_sched_worker_change.c": 6,
|
||||
"src/controller/testsuite/test_persist_checkpoint_offload.c": 1
|
||||
"src/controller/testsuite/test_persist_checkpoint_offload.c": 1,
|
||||
"src/controller/myapp.c": 6,
|
||||
"src/controller/testsuite/test_ha_startup.c": 3,
|
||||
"src/controller/startup_validation.c": 3
|
||||
},
|
||||
"anatomy_hits": 54,
|
||||
"anatomy_misses": 35,
|
||||
"repeated_reads_warned": 263,
|
||||
"anatomy_hits": 56,
|
||||
"anatomy_misses": 36,
|
||||
"repeated_reads_warned": 273,
|
||||
"cerebrum_warnings": 0,
|
||||
"stop_count": 43
|
||||
"stop_count": 45
|
||||
}
|
||||
|
|
@ -4113,3 +4113,18 @@
|
|||
| 07:39 | Edited src/controller/persist.c | 4→5 lines | ~78 |
|
||||
| 07:45 | Edited src/controller/persist.c | 5→4 lines | ~57 |
|
||||
| 14:51 | xce-audit-invariants dim1 fix: checkpoint_task_fn (delaymgr thread) blocked on real persist_checkpoint() I/O -- no offload existed at all, unlike every sibling (snapshot-send/config-push/reconnect). Investigated apply_pump_lock lock-order risk first: ruled OUT vtask pool (persist_apply_pump_run holds the same lock and is ITSELF dispatched onto that pool via VTASK_LIVE_BLOCK -- would starve it), used the SLOW LANE instead (slowlane_submit), matching snapshot-send's already-proven-safe precedent (persist_checkpoint already runs there today via ha_snapshot_do_send). New flag g_persist_checkpoint_offload defaults ON (not flagged off like catchup/demotion -- verified no test-fixture-hang precedent applies, checkpoint's trigger is fire-and-forget). bug-1694 logged. RED (probe forced pre-fix inline path, GREEN test hung, kill -9'd, confirmed real detection) then GREEN (reverted, rebuilt) both verified via full synchronous `/xce-build controller asan build-asan --incremental` runs | src/controller/persist.c, src/include/persist.h, src/controller/testsuite/test_persist_checkpoint_offload.c (new), src/controller/testsuite/controller_test.c, src/controller/meson.build, .wolf/buglog.json | 1184 Tests 2 pre-existing Failures (bug-1561, unrelated) 0 Ignored, 0 NEW failures, 3/3 new tests PASS | ~145k |
|
||||
| 07:56 | Session end: 199 writes across 44 files (persistence.md, SKILL.md, plugin.h, plugin.c, driver.c) | 89 reads | ~901908 tok |
|
||||
| 08:06 | Session end: 199 writes across 44 files (persistence.md, SKILL.md, plugin.h, plugin.c, driver.c) | 90 reads | ~908195 tok |
|
||||
| 08:15 | Edited src/controller/myapp.c | added 3 condition(s) | ~1610 |
|
||||
| 08:15 | Edited src/controller/myapp.c | expanded (+13 lines) | ~310 |
|
||||
| 08:15 | Edited src/controller/myapp.c | modified if() | ~208 |
|
||||
| 08:15 | Edited src/controller/testsuite/test_ha_startup.c | expanded (+6 lines) | ~175 |
|
||||
| 08:17 | Edited src/controller/testsuite/test_ha_startup.c | modified test_bug1695_stale_restored_term_does_not_skip_election() | ~2196 |
|
||||
| 08:17 | Edited src/controller/testsuite/test_ha_startup.c | expanded (+11 lines) | ~222 |
|
||||
| 08:20 | Edited src/controller/myapp.c | removed 116 lines | ~68 |
|
||||
| 08:21 | Edited src/controller/myapp.c | expanded (+7 lines) | ~181 |
|
||||
| 08:21 | Edited src/controller/startup_validation.c | expanded (+8 lines) | ~258 |
|
||||
| 08:21 | Edited src/controller/startup_validation.c | added 3 condition(s) | ~1616 |
|
||||
| 08:26 | Edited src/controller/startup_validation.c | CURRENT() → term() | ~86 |
|
||||
| 16:05 | bug-1695 FIXED: myapp_run's "leader already known" bootstrap check now requires raft.term to advance past a term_at_boot snapshot (not merely >0) — extracted to myapp_leader_already_known() in startup_validation.c (links into controller_unit_test, myapp.c doesn't) so it's unit-testable; RED/GREEN with 7 new tests in test_ha_startup.c; full suite 1191 tests, 0 new failures (2 pre-existing bug-1561); live A/B 6-round --chaotic soak (fixed vs git-stashed baseline) proved the fix: baseline term frozen at 4 forever, 6/6 "NO new primary", ctrl-2/ctrl-3 show zero election activity ever; fixed run shows term progressing 1->2->3->4->7 and ctrl-2 winning a real logged CTRL_PRIMARY via PreVote — direct proof a new node can now take over. Residual, honestly-flagged finding: fixed run's chaotic verdict showed committed_job_loss=6 (baseline=0) but committed_divergence=0 in both; waldump-traced the one lost jobid to the cluster's very first cold-boot config-bootstrap window (pre-round-1), not present in ANY node's WAL — resembles already-documented bug-1487's apply-pump-lag false-positive class, newly exposed because this fix is the first thing that lets the chaotic driver's overlapping kills exercise genuine repeated elections at all. Updated buglog.json bug-1695 (fixed, occurrences=2), cerebrum.md (3 Key Learnings, 1 Do-Not-Repeat, 1 Decision Log entry) | src/controller/myapp.c, src/controller/startup_validation.c, src/controller/testsuite/test_ha_startup.c, .wolf/buglog.json | fixed+verified, 1 residual follow-up flagged | ~large |
|
||||
| 09:05 | Edited src/controller/myapp.c | expanded (+8 lines) | ~256 |
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue