myxce/.wolf/cerebrum.md

# Cerebrum

> OpenWolf's learning memory. Updated automatically as the AI learns from interactions.
> Do not edit manually unless correcting an error.
> Last updated: 2026-05-24

## User Preferences

<!-- How the user likes things done. Code style, tools, patterns, communication. -->

## Key Learnings

- Use `clang-format` for C/C++ formatting; shared config lives at `xce/.clang-format`.
- `msg_handler_t` in netio.h is `int (*)(void*)` — NOT `int (*)(endpoint_t*)`. Functions assigned to `ep->msg_handler` must use `void *ep` parameter. (Task 2.2, 2026-05-30)
- `xce_request()` is defined in request.c but not declared in request.h; files that need it must add `extern int xce_request(endpoint_t *ep, xce_request_t *req);` locally (pattern from config_repl.c).
- Test seam pattern for ha.c XCE_TESTING: declare `int (*hook)(ep, req) = NULL` under `#ifdef XCE_TESTING`, expose as extern in test file, test sets it before calling function under test. Mirrors `xce_retry_send_hook` in request.c.
- capnp UAF fix pattern (Task 2.3): INVOKE_MSG_HANDLER fires ep->msg_handler(ep) directly from iomgr thread; packet is in ep->fsm_in.pkt. Do raw capn_init_mem decode, read_RequestVoteReply into local struct BEFORE capn_free, then capn_free + free_packet + free(pkt). Use local scalars (not capnp pointers) after free. (2026-05-30)
- capnp serialization for tests: use new_RequestVoteReply(seg) / new_RequestReply(seg) / new_Request(seg) — NOT raw capn_new_struct. Follow pattern from xce_encode / xce_replconfigpush_req_raw in request.c.
- To avoid C99 "goto jumps over declaration" error: declare all local vars at top of function (or in a sub-block), not after the first goto target label.
- hook seam forward-declaration order: declare the hook variable BEFORE the function that references it (even within the same XCE_TESTING block). Use a dispatch helper static function to decouple seam from production path.
- term fencing must live in the handler, not the dispatch path (2026-05-31): vote_reply_msg_handler delegates to vote_reply_dispatch which calls either xce_vote_reply_hook (XCE_TESTING) or raft_on_vote_reply (production). If term fencing only happens inside raft_cb_become_follower (production), it is silently skipped in the test hook path. Pattern: do explicit CAS loop (atomic_compare_exchange_weak) on ha->term before calling vote_reply_dispatch, so the update applies regardless of which dispatch path runs.

- ha_start_election_async ownership rule (Task 2.4, 2026-05-30): in the production path (c != NULL), ha_elect_ctx_t is owned by election_timeout_fn which frees it on fire. In the test-mode path (c == NULL, peers == NULL), no delaymgr task owns ctx — must free ctx inline after ha_elect_evaluate_and_signal fires. ASAN will catch the leak if forgotten.
- ha_start_election_async single-node shortcircuit (Task 2.4): n_peers==1 means n_other=0 — allocate ha_elect_ctx_t with 0 flexible entries (no votes[] slots needed), set votes_granted=1, call evaluate directly, then free ctx immediately. No iomgr or delaymgr involvement for single-node.
- election_timeout_fn must lock ctx->lock before calling ha_elect_evaluate_and_signal (Task 2.4): the evaluate function has a done_posted guard but its callers in the reply path hold ctx->lock; timeout fn must also hold the lock to avoid races with concurrent reply processing.
- single-node shortcircuit enters CTRL_PRIMARY_COMMITTING, NOT CTRL_PRIMARY (Task 2.4 review fix): task 4.4 committing timeout promotes to PRIMARY for degraded/single-node mode (Req 8.4). Tests must assert CTRL_PRIMARY_COMMITTING.
- ha_elect_evaluate_and_signal out-of-bounds guard (Task 2.4 review fix): ctx->votes[] is a C99 flexible array with n_peers entries. When n_peers==0 (single-node), votes[0] is out-of-bounds. Must check n_peers==0 FIRST and short-circuit to PRIMARY (quorum=1, self-vote=1 always wins). Guard added before any votes[0] access.
- client_ssl_ctx for outgoing vote connections (Task 2.4 review fix): endpoint_connect() creates SSL from iomgr->ssl_ctx (server context). After connect returns, replace ep->ssl with SSL_new(client_ssl_ctx), SSL_set_fd to the same fd, SSL_set_connect_state. If SSL_new fails, fall through (handshake fails → vote denied via err_handler).
- vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.
- timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.

- Heartbeat freshness source of truth (2026-06-09): `ha_heartbeat_is_fresh` derives from `ha->last_activity` (whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The old `last_heartbeat` timespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate is `role==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha)`; RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh.
- raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so `raft_init` must set `voted_for_term = initial_term` when `initial_voted_for != -1`. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6.

## Do-Not-Repeat

<!-- Mistakes made and corrected. Each entry prevents the same mistake recurring. -->
<!-- Format: [YYYY-MM-DD] Description of what went wrong and what to do instead. -->

- [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.
- [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).

- [2026-06-04] capnpc-c 1.0.1 has a bug with `:group` inside a `union` when `$C.codecgen` is active: the encode/decode generator emits `d->FIELDNAME` (flat) instead of `d->data.GROUPNAME.FIELDNAME`, AND places the case as dead code after the prior `break`. Do NOT use inline `:group` in a union that has `$C.codecgen`. Instead, define a named outer struct (`struct Foo $C.mapname("foo_t")`) and reference it as a pointer union member. The named struct generates correct `encode_Foo_ptr`/`decode_Foo_ptr` calls. The `read_RequestReply`/`write_RequestReply` wire layer works correctly with groups but the codec layer does not.

## Decision Log

<!-- Significant technical decisions with rationale. Why X was chosen over Y. -->

- Replication payload wire format (2026-06-09, ha-repl-hardening 7.1): chose the established `_raw` sender convention (manual capnp build via capn_new_list8/capn_setv8 + xce_decode per-opcode raw extraction into raw_payload/raw_payload_len struct fields) over extending the typemap codegen to map capnp Data ↔ raw buffers. Rationale: replConfigPush/upgradeConfig already prove the pattern in production; the codegen route risks regressions across all opcodes and fights the known capnpc-c 1.0.1 codecgen bugs. Schema change: Request.body union gained walcatchupentrymsg @28 / snapshottransfermsg @29 — these messages previously had NO union arm at all (sent as body_which=nomsg, i.e. nothing crossed the wire).
- Convergence threshold (2026-06-09, ha-repl-hardening 7.5): chose requirements.md Req 2.3 formula — ceil(n_peers/2)−1 converged standbys (integer form n_peers/2) — over the code's previous all-standbys rule. Rationale: primary + converged standbys form a cluster majority, matching Raft quorum; committed entries are already on a majority per-entry, and safe_committed_lsn derives from a majority of voters, so promotion with one dead minority standby is safe and avoids a 60s degraded-mode timeout. Lagging standbys keep catching up via the heartbeat decision loop after promotion.

- Election liveness fix (2026-06-09): `skip_election` path in myapp.c had two bugs: (1) `voted_for_name != ""` triggered even for self-votes and stale terms — fixed by adding `strcmp(name,"self")!=0 && voted_for_term==ha->term`; (2) `ha_arm_heartbeat_timeout_vtask` was never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.c `handle_request_vote_req` also simplified to term-only deny.
  - CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.

- Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.