myxce/.wolf/cerebrum.md

Cerebrum

OpenWolf's learning memory. Updated automatically as the AI learns from interactions. Do not edit manually unless correcting an error. Last updated: 2026-05-24

User Preferences

Key Learnings

• Use clang-format for C/C++ formatting; shared config lives at xce/.clang-format.

• msg_handler_t in netio.h is int (*)(void*) — NOT int (*)(endpoint_t*). Functions assigned to ep->msg_handler must use void *ep parameter. (Task 2.2, 2026-05-30)

• xce_request() is defined in request.c but not declared in request.h; files that need it must add extern int xce_request(endpoint_t *ep, xce_request_t *req); locally (pattern from config_repl.c).

• Test seam pattern for ha.c XCE_TESTING: declare int (*hook)(ep, req) = NULL under #ifdef XCE_TESTING, expose as extern in test file, test sets it before calling function under test. Mirrors xce_retry_send_hook in request.c.

• capnp UAF fix pattern (Task 2.3): INVOKE_MSG_HANDLER fires ep->msg_handler(ep) directly from iomgr thread; packet is in ep->fsm_in.pkt. Do raw capn_init_mem decode, read_RequestVoteReply into local struct BEFORE capn_free, then capn_free + free_packet + free(pkt). Use local scalars (not capnp pointers) after free. (2026-05-30)

• capnp serialization for tests: use new_RequestVoteReply(seg) / new_RequestReply(seg) / new_Request(seg) — NOT raw capn_new_struct. Follow pattern from xce_encode / xce_replconfigpush_req_raw in request.c.

• To avoid C99 "goto jumps over declaration" error: declare all local vars at top of function (or in a sub-block), not after the first goto target label.

• hook seam forward-declaration order: declare the hook variable BEFORE the function that references it (even within the same XCE_TESTING block). Use a dispatch helper static function to decouple seam from production path.

• term fencing must live in the handler, not the dispatch path (2026-05-31): vote_reply_msg_handler delegates to vote_reply_dispatch which calls either xce_vote_reply_hook (XCE_TESTING) or raft_on_vote_reply (production). If term fencing only happens inside raft_cb_become_follower (production), it is silently skipped in the test hook path. Pattern: do explicit CAS loop (atomic_compare_exchange_weak) on ha->term before calling vote_reply_dispatch, so the update applies regardless of which dispatch path runs.

• ha_start_election_async ownership rule (Task 2.4, 2026-05-30): in the production path (c != NULL), ha_elect_ctx_t is owned by election_timeout_fn which frees it on fire. In the test-mode path (c == NULL, peers == NULL), no delaymgr task owns ctx — must free ctx inline after ha_elect_evaluate_and_signal fires. ASAN will catch the leak if forgotten.

• ha_start_election_async single-node shortcircuit (Task 2.4): n_peers==1 means n_other=0 — allocate ha_elect_ctx_t with 0 flexible entries (no votes[] slots needed), set votes_granted=1, call evaluate directly, then free ctx immediately. No iomgr or delaymgr involvement for single-node.

• election_timeout_fn must lock ctx->lock before calling ha_elect_evaluate_and_signal (Task 2.4): the evaluate function has a done_posted guard but its callers in the reply path hold ctx->lock; timeout fn must also hold the lock to avoid races with concurrent reply processing.

• single-node shortcircuit enters CTRL_PRIMARY_COMMITTING, NOT CTRL_PRIMARY (Task 2.4 review fix): task 4.4 committing timeout promotes to PRIMARY for degraded/single-node mode (Req 8.4). Tests must assert CTRL_PRIMARY_COMMITTING.

• ha_elect_evaluate_and_signal out-of-bounds guard (Task 2.4 review fix): ctx->votes[] is a C99 flexible array with n_peers entries. When n_peers==0 (single-node), votes[0] is out-of-bounds. Must check n_peers==0 FIRST and short-circuit to PRIMARY (quorum=1, self-vote=1 always wins). Guard added before any votes[0] access.

• client_ssl_ctx for outgoing vote connections (Task 2.4 review fix): endpoint_connect() creates SSL from iomgr->ssl_ctx (server context). After connect returns, replace ep->ssl with SSL_new(client_ssl_ctx), SSL_set_fd to the same fd, SSL_set_connect_state. If SSL_new fails, fall through (handshake fails → vote denied via err_handler).

• vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.

• timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.

• Heartbeat freshness source of truth (2026-06-09): ha_heartbeat_is_fresh derives from ha->last_activity (whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The old last_heartbeat timespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate is role==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha); RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh.

• raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so raft_init must set voted_for_term = initial_term when initial_voted_for != -1. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6.

Do-Not-Repeat

• [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.

• [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).

• [2026-06-04] capnpc-c 1.0.1 has a bug with :group inside a union when $C.codecgen is active: the encode/decode generator emits d->FIELDNAME (flat) instead of d->data.GROUPNAME.FIELDNAME, AND places the case as dead code after the prior break. Do NOT use inline :group in a union that has $C.codecgen. Instead, define a named outer struct (struct Foo $C.mapname("foo_t")) and reference it as a pointer union member. The named struct generates correct encode_Foo_ptr/decode_Foo_ptr calls. The read_RequestReply/write_RequestReply wire layer works correctly with groups but the codec layer does not.

Decision Log

• Replication payload wire format (2026-06-09, ha-repl-hardening 7.1): chose the established _raw sender convention (manual capnp build via capn_new_list8/capn_setv8 + xce_decode per-opcode raw extraction into raw_payload/raw_payload_len struct fields) over extending the typemap codegen to map capnp Data ↔ raw buffers. Rationale: replConfigPush/upgradeConfig already prove the pattern in production; the codegen route risks regressions across all opcodes and fights the known capnpc-c 1.0.1 codecgen bugs. Schema change: Request.body union gained walcatchupentrymsg @28 / snapshottransfermsg @29 — these messages previously had NO union arm at all (sent as body_which=nomsg, i.e. nothing crossed the wire).

• Convergence threshold (2026-06-09, ha-repl-hardening 7.5): chose requirements.md Req 2.3 formula — ceil(n_peers/2)−1 converged standbys (integer form n_peers/2) — over the code's previous all-standbys rule. Rationale: primary + converged standbys form a cluster majority, matching Raft quorum; committed entries are already on a majority per-entry, and safe_committed_lsn derives from a majority of voters, so promotion with one dead minority standby is safe and avoids a 60s degraded-mode timeout. Lagging standbys keep catching up via the heartbeat decision loop after promotion.

• Election liveness fix (2026-06-09): skip_election path in myapp.c had two bugs: (1) voted_for_name != "" triggered even for self-votes and stale terms — fixed by adding strcmp(name,"self")!=0 && voted_for_term==ha->term; (2) ha_arm_heartbeat_timeout_vtask was never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.c handle_request_vote_req also simplified to term-only deny.

  • CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.

• Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.