myxce/.wolf/cerebrum.md

Cerebrum

OpenWolf's learning memory. General, reusable knowledge only — bug-specific history lives in .wolf/buglog.json, not here. Do not edit manually unless correcting an error. Last updated: 2026-08-16 (condensed from a ~40k-token file; see .wolf/buglog.json for full bug history)

User Preferences

• Run soak tests with specific round counts and deploy directories for precise regression tracking.
• Prefer parallel agent runs for independent analysis work; sequential for dependent tasks.
• Keep test logs for analysis, don't skip intermediate output even if a test hangs.

Key Learnings

• Build entry point: flake.nix lives in src/, not the repo root. Always nix develop /abs/path/to/src --command bash -c '...' with an explicit path — Bash cwd isn't stable across calls, and a bare cd src && nix develop can silently short-circuit.
• Test coverage gaps: documented full-suite commands (xce-build.sh <component>, make test) each cover only a subset — e.g. a component's default meson run misses newly-added standalone test binaries unless wired into its RUN_TESTS case, and the CGO/apiserver backend suite needs make test-backend with CGO_CFLAGS/LDFLAGS reconstructed via pkg-config (package is c-capnproto, not capnp_c). A green default build is not proof of full coverage — verify every new test binary actually runs.
• Unity harness: use UnitySetTestFile(__FILE__) (not nested UNITY_BEGIN()/UNITY_END()) when multiple test-group files share one binary — nested BEGIN/END resets global counters and can mask most of the real suite exit code. Authoritative green = grep raw testlog for :FAIL: / word-boundary \bTIMEOUT\b, never trust an aggregate Fail:0 alone. Also rule out a concurrent competing build/soak against the same build dir before treating a timeout/crash as a regression.
• capnp/typemap conventions: every schema struct gets $C.mapname(...); for a wire-only concept with no live C struct, hand-define a small matching struct in typemap.h. A Data-typed field's generated composite codec is a silent no-op in this backend AND the generated _ptr wrapper doesn't zero-init — an unset field is written as literal uninitialized stack garbage (real crash risk). Prefer List(UInt8) + a $C.maplistcount field, which this backend does generate real code for. A nested-struct getter can return an unresolved far pointer after a segment round-trip — call capn_resolve() before reading its scalar fields (Text/Data getters resolve internally and mask this).
• C comments: never write a bare type*/ (pointer type immediately followed by /) inside a /* */ block comment — it silently closes the comment early, producing a cascade of unrelated syntax errors elsewhere. Recurred repeatedly; grep new/edited comments for [A-Za-z0-9_]\*/ before building.
• Thread/lock hazards on async-ified single-threaded code: a library/struct built for serialized calling (Raft state, SSL* objects, an epoll batch-dispatch loop) that later gets a "drive it on a thread pool" feature needs its own lock/refcount around the shared state — per-endpoint or per-callback serialization does not imply serialization across different endpoints. A same-thread same-batch use-after-free (one event's handler freeing something a later event in the identical batch still references) can look identical to a genuine cross-thread race.
• makeString() (lib/xstring.c) is rtti-backed, not malloc/strdup — only free it via its own module's rtti-aware teardown, or don't free it at all; a generic free()/capnp free_X() on it is heap corruption. Use strdup() for fields a generated free_X() will free directly.
• Threaded test fixtures: teardown for a fixture with real background threads must run unconditionally before any TEST_ASSERT_* — a failed assertion longjmps past later code, orphaning the thread and corrupting the next test's fixture memory.
• Live-soak validation discipline: a unit-test-green fix can still be unreachable/dead code in production — always re-run the exact live scenario end-to-end, not just the mechanism fixed. A later, differently-shaped soak failure after a fix isn't automatically a regression from that fix — check whether that failure point was ever exercised before.
• Sandbox quirks: /tmp can be overlayfs with intermittent fdatasync EIO — durability-sensitive tests need a real fs (e.g. ~/tmp on ext4). This sandbox's date may resolve to BusyBox (no %N, silently passes it through instead of erroring) — use bash's $EPOCHREALTIME/SECONDS builtins for timing instead.
• HA election timer discipline (bug-1562): every early-return branch in a function driven by a one-shot timer (ha_on_primary_ctrl_fd_rdhup's liveness-watchdog callback) must either re-arm the next timer itself or verify some OTHER live timer is actually still scheduled before skipping the re-arm — checking only elapsed-time-since-last-event (the 5.10b debounce branch) is not the same as checking an election is still in flight (the 5.6 guard's actual role/pre_vote_in_progress check). A silently-dropped re-arm freezes the node with zero CPU/log activity forever (looks identical to a clean idle state, not a crash) — diagnose via /proc/<pid>/task/*/wchan (all threads in futex_do_wait/do_epoll_wait with no fds beyond listen+epoll = dropped timer, not deadlock) before reaching for gdb (ptrace is blocked in this sandbox, yama ptrace_scope=1 + no ancestor relationship to the target).
• Reproducing HA bootstrap bugs: match the user's actual inter-node start stagger exactly — a tight ~0.5s stagger (the /xce-deploy-ha skill's default) can mask a bug that only triggers when a node sits solo for tens of seconds (real peers answering fast enough prevents the trap window from ever closing). Always ask/check the real timestamps before concluding "could not reproduce."
• A soak's own "excluded"/PASS verdict is not a license to stop reading (bug-1563): /xce-test-ha --chaotic's verdict deliberately excludes bug924_fdatasync (overlayfs EIO) from pass/fail — but a huge excluded count (300K+ in one run) or a different errno than the documented one (EINVAL, not EIO) is worth a manual look anyway. Found: fsync_worker's retry loop only backs off when NOT pending, and a sustained fdatasync failure makes "pending" permanently true (durable_lsn never advances on failure) — a real busy-spin/livelock, invisible to the soak's own job-loss/divergence checks since the other 2-of-3 standbys still satisfy majority. Lesson: skim excluded/noise counts for magnitude and errno before trusting a documented exclusion covers them.

Do-Not-Repeat

• Don't run clang-format -i on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via git checkout HEAD -- <file> on a file with other uncommitted work destroys that work irrecoverably.
• Before citing a bug as "still open" or a spec as "blocked," cross-check .wolf/buglog.json's actual fix status and the spec's own tasks.md/commit history — blocked_on notes and prior narrative claims (including this file's own past entries) can be stale.
• Before trusting an "all tests pass, GO" claim on a stale/uncommitted tree, re-run the suite yourself — a buglog "FIXED" note can describe the intended fix while the code on disk is mid-revert from RED-phase construction.
• A generic retry/failover classifier that judges retriable-vs-terminal by a raw return value's magnitude (comparing against known errno values) is unsafe unless every wrapped call site shares the same success convention — a call that legitimately returns a variable count on success can numerically collide with an error code.
• When two independent "is it ready" signals exist server-side, a wait/convergence helper gating on only one does not imply the other is satisfied — check what the actual raced precondition depends on.
• Nix fixed-output-derivation hash mismatches from an upstream input pinning a mutable branch: re-locking flake.lock is a dead end once already at the latest commit; fix via a local overrideAttrs pinning src to an explicit immutable commit in the consuming repo.
• pkill/pgrep by bare process name (unscoped) can kill or match a different concurrent invocation's processes, including a sibling session's build/soak — always scope process matching to something unique to the current run (e.g. a workdir path).

Decision Log

• HA/consistency model follows Raft + CP strictly: majority ACK is the hard commit barrier for all state-changing writes (reject on timeout, never local-apply-and-degrade); the own-term commit rule (not a 4th "COMMITTING" state) is what gives a new leader safety.
• A controller process must NEVER self-exit/self-abort as a designed response to an operational condition (only a genuine crash or admin/supervisor kill) — an unrecoverable/fenced member logs loudly, stays permanently non-promotable, stays alive, and keeps retrying.
• Commit-first-then-reply: job/queue/host control ops validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply; no success reply before commit. Worker-reported fact entries instead follow pure Raft semantics — retained/retried indefinitely, no commit timeout.
• No client reply-wait timeout as a fix for a hung request — fail loud at the real transport/source instead of masking with a deadline backstop.
• Product is pre-release: no backward/mixed-version wire compatibility required; wire format changes may be made freely.