myxce/.wolf/cerebrum.md

# Cerebrum

> OpenWolf's learning memory. General, reusable knowledge only — bug-specific history lives in .wolf/buglog.json, not here.
> Do not edit manually unless correcting an error.
> Last updated: 2026-08-16 (condensed from a ~40k-token file; see .wolf/buglog.json for full bug history)

## User Preferences
- Run soak tests with specific round counts and deploy directories for precise regression tracking.
- Prefer parallel agent runs for independent analysis work; sequential for dependent tasks.
- Keep test logs for analysis, don't skip intermediate output even if a test hangs.

## Key Learnings
- **Build entry point**: `flake.nix` lives in `src/`, not the repo root. Always `nix develop /abs/path/to/src --command bash -c '...'` with an explicit path — Bash cwd isn't stable across calls, and a bare `cd src && nix develop` can silently short-circuit.
- **Test coverage gaps**: documented full-suite commands (`xce-build.sh <component>`, `make test`) each cover only a subset — e.g. a component's default meson run misses newly-added standalone test binaries unless wired into its `RUN_TESTS` case, and the CGO/apiserver backend suite needs `make test-backend` with CGO_CFLAGS/LDFLAGS reconstructed via pkg-config (package is `c-capnproto`, not `capnp_c`). A green default build is not proof of full coverage — verify every new test binary actually runs.
- **Unity harness**: use `UnitySetTestFile(__FILE__)` (not nested `UNITY_BEGIN()/UNITY_END()`) when multiple test-group files share one binary — nested BEGIN/END resets global counters and can mask most of the real suite exit code. Authoritative green = grep raw testlog for `:FAIL:` / word-boundary `\bTIMEOUT\b`, never trust an aggregate `Fail:0` alone. Also rule out a concurrent competing build/soak against the same build dir before treating a timeout/crash as a regression.
- **capnp/typemap conventions**: every schema struct gets `$C.mapname(...)`; for a wire-only concept with no live C struct, hand-define a small matching struct in `typemap.h`. A `Data`-typed field's generated composite codec is a silent no-op in this backend AND the generated `_ptr` wrapper doesn't zero-init — an unset field is written as literal uninitialized stack garbage (real crash risk). Prefer `List(UInt8)` + a `$C.maplistcount` field, which this backend does generate real code for. A nested-struct getter can return an unresolved far pointer after a segment round-trip — call `capn_resolve()` before reading its scalar fields (Text/Data getters resolve internally and mask this).
- **C comments**: never write a bare `type*/` (pointer type immediately followed by `/`) inside a `/* */` block comment — it silently closes the comment early, producing a cascade of unrelated syntax errors elsewhere. Recurred repeatedly; grep new/edited comments for `[A-Za-z0-9_]\*/` before building.
- **Thread/lock hazards on async-ified single-threaded code**: a library/struct built for serialized calling (Raft state, SSL* objects, an epoll batch-dispatch loop) that later gets a "drive it on a thread pool" feature needs its own lock/refcount around the *shared* state — per-endpoint or per-callback serialization does not imply serialization across different endpoints. A same-thread same-batch use-after-free (one event's handler freeing something a later event in the identical batch still references) can look identical to a genuine cross-thread race.
- **makeString() (lib/xstring.c)** is rtti-backed, not malloc/strdup — only free it via its own module's rtti-aware teardown, or don't free it at all; a generic `free()`/capnp `free_X()` on it is heap corruption. Use `strdup()` for fields a generated `free_X()` will free directly.
- **Threaded test fixtures**: teardown for a fixture with real background threads must run *unconditionally* before any `TEST_ASSERT_*` — a failed assertion longjmps past later code, orphaning the thread and corrupting the next test's fixture memory.
- **Live-soak validation discipline**: a unit-test-green fix can still be unreachable/dead code in production — always re-run the exact live scenario end-to-end, not just the mechanism fixed. A later, differently-shaped soak failure after a fix isn't automatically a regression from that fix — check whether that failure point was ever exercised before.
- **Sandbox quirks**: `/tmp` can be overlayfs with intermittent fdatasync EIO — durability-sensitive tests need a real fs (e.g. `~/tmp` on ext4). This sandbox's `date` may resolve to BusyBox (no `%N`, silently passes it through instead of erroring) — use bash's `$EPOCHREALTIME`/`SECONDS` builtins for timing instead.
- **HA election timer discipline (bug-1562)**: every early-return branch in a function driven by a one-shot timer (`ha_on_primary_ctrl_fd_rdhup`'s liveness-watchdog callback) must either re-arm the next timer itself or verify some OTHER live timer is actually still scheduled before skipping the re-arm — checking only elapsed-time-since-last-event (the 5.10b debounce branch) is not the same as checking an election is still in flight (the 5.6 guard's actual role/pre_vote_in_progress check). A silently-dropped re-arm freezes the node with zero CPU/log activity forever (looks identical to a clean idle state, not a crash) — diagnose via `/proc/<pid>/task/*/wchan` (all threads in `futex_do_wait`/`do_epoll_wait` with no fds beyond listen+epoll = dropped timer, not deadlock) before reaching for gdb (ptrace is blocked in this sandbox, `yama ptrace_scope=1` + no ancestor relationship to the target).
- **Reproducing HA bootstrap bugs**: match the user's actual inter-node start stagger exactly — a tight ~0.5s stagger (the `/xce-deploy-ha` skill's default) can mask a bug that only triggers when a node sits solo for tens of seconds (real peers answering fast enough prevents the trap window from ever closing). Always ask/check the real timestamps before concluding "could not reproduce."
- **A soak's own "excluded"/PASS verdict is not a license to stop reading (bug-1563)**: `/xce-test-ha --chaotic`'s verdict deliberately excludes `bug924_fdatasync` (overlayfs EIO) from pass/fail — but a huge excluded count (300K+ in one run) or a different errno than the documented one (EINVAL, not EIO) is worth a manual look anyway. Found: `fsync_worker`'s retry loop only backs off when NOT pending, and a sustained fdatasync failure makes "pending" permanently true (durable_lsn never advances on failure) — a real busy-spin/livelock, invisible to the soak's own job-loss/divergence checks since the other 2-of-3 standbys still satisfy majority. Lesson: skim excluded/noise counts for magnitude and errno before trusting a documented exclusion covers them.

## Do-Not-Repeat
- Don't run `clang-format -i` on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via `git checkout HEAD -- <file>` on a file with other uncommitted work destroys that work irrecoverably.
- Before citing a bug as "still open" or a spec as "blocked," cross-check `.wolf/buglog.json`'s actual fix status and the spec's own tasks.md/commit history — blocked_on notes and prior narrative claims (including this file's own past entries) can be stale.
- Before trusting an "all tests pass, GO" claim on a stale/uncommitted tree, re-run the suite yourself — a buglog "FIXED" note can describe the *intended* fix while the code on disk is mid-revert from RED-phase construction.
- A generic retry/failover classifier that judges retriable-vs-terminal by a raw return value's magnitude (comparing against known errno values) is unsafe unless every wrapped call site shares the same success convention — a call that legitimately returns a variable count on success can numerically collide with an error code.
- When two independent "is it ready" signals exist server-side, a wait/convergence helper gating on only one does not imply the other is satisfied — check what the actual raced precondition depends on.
- Nix fixed-output-derivation hash mismatches from an upstream input pinning a *mutable* branch: re-locking flake.lock is a dead end once already at the latest commit; fix via a local `overrideAttrs` pinning `src` to an explicit immutable commit in the consuming repo.
- `pkill`/`pgrep` by bare process name (unscoped) can kill or match a *different* concurrent invocation's processes, including a sibling session's build/soak — always scope process matching to something unique to the current run (e.g. a workdir path).

## Decision Log
- HA/consistency model follows Raft + CP strictly: majority ACK is the hard commit barrier for all state-changing writes (reject on timeout, never local-apply-and-degrade); the own-term commit rule (not a 4th "COMMITTING" state) is what gives a new leader safety.
- A controller process must NEVER self-exit/self-abort as a designed response to an operational condition (only a genuine crash or admin/supervisor kill) — an unrecoverable/fenced member logs loudly, stays permanently non-promotable, stays alive, and keeps retrying.
- Commit-first-then-reply: job/queue/host control ops validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply; no success reply before commit. Worker-reported fact entries instead follow pure Raft semantics — retained/retried indefinitely, no commit timeout.
- No client reply-wait timeout as a fix for a hung request — fail loud at the real transport/source instead of masking with a deadline backstop.
- Product is pre-release: no backward/mixed-version wire compatibility required; wire format changes may be made freely.