132 lines
50 KiB
Markdown
132 lines
50 KiB
Markdown
# Cerebrum
|
||
|
||
> OpenWolf's learning memory. Updated automatically as the AI learns from interactions.
|
||
> Do not edit manually unless correcting an error.
|
||
> Last updated: 2026-05-24
|
||
|
||
## User Preferences
|
||
|
||
<!-- How the user likes things done. Code style, tools, patterns, communication. -->
|
||
|
||
## Key Learnings
|
||
- (2026-06-15, ha-rejoin bug-340) A killed+restarted controller loops elections forever because the PRIMARY never evicts the dead/wedged outbound standby endpoint. primary_heartbeat_fn (ha.c) has two steps: (1) NOP each ep in ha->standby_endpoints, (2) reconnect to any peer NOT present in standby_endpoints. A wedged ep (peer died → handle_error did EPOLL_CTL_DEL but left it in the array; epoll MOD then returns ENOENT → the `WARN endpoint_update_epoll rc1=-1` spam) makes step 2 think the peer is still connected → reconnect SUPPRESSED → restarted node never gets a ctrlhello → re-elects every heartbeat_timeout (term climbs unbounded). xce_nop_req returns >=0 on enqueue even when the epoll re-arm fails, so the heartbeat layer can't see the wedge. FIX: evict in step 1 using endpoint_t.ts_last_in (existing wall-clock field, stamped by handle_input) — if ts_last_in>0 && now-ts_last_in>heartbeat_timeout, zarray_del from standby_endpoints + repl_remove_standby; reconnect then re-establishes the channel and ctrl-1 settles via ha_discover_primary (NO term gate — accepts any primary msg while ELECTING regardless of its own high term). Gotchas: (a) ha_arm_heartbeat_timeout_vtask ALREADY resets last_activity=now, so 'become_follower doesn't reset liveness' is NOT a bug — the ~timeout loop period proves the watchdog is correct; the sole cause is the broken channel. (b) standby_endpoints holds TWO ep types (promoted vote_ep_ctx_t with ctx!=NULL, and heartbeat standby_ep_ctx_t with ctx==NULL) sharing first 4 fields; don't free vote-ep data (UAF — live vctx pointers). (c) Run one suite: controller_unit_test -t test_ha_liveness, ASAN_OPTIONS=detect_leaks=0.
|
||
- (2026-06-15, ha-election bug-339) Candidate-advertised RequestVote LSN MUST equal the voter-side freshness floor. Voter floor = ha_vote_self_freshness(ha, config_tag, applied_lsn) = max(applied_lsn, last_known_committed_lsn) (cluster.c OPCode_requestVote). On a STANDBY, persist.committed_lsn NEVER advances (only the primary commit engine writes it at persist.c:1192); standbys advance applied_lsn (ha_apply_wal_catchup_entry) + last_known_committed_lsn (heartbeat replies). So a candidate that advertises persist.committed_lsn under-advertises vs its own floor → after a primary dies, every surviving standby advertises stale-low LSN while each peer's floor is high → freshness gate (cand_lsn<my_lsn) denies on BOTH sides → no quorum → infinite RDHUP→ELECTING→STANDBY loop, terms climb in lockstep, nobody ever become_leader. Boot path (myapp.c:217) already uses applied_lsn; only the RDHUP failover path (ha_on_primary_ctrl_fd_rdhup) was inconsistent. Fix: compute cand_tag/cand_lsn via ha_vote_self_freshness once and feed BOTH the elect_ctx (votes[0]+self_*) AND the ha_rdhup_election_hook args. Symmetric advertised==floor guarantees the freshest node is granted by all peers. Diagnostic signature in logs: both nodes only raft_cb_become_follower (never become_leader) + RDHUP every ~heartbeat_timeout.
|
||
- (2026-06-14, election-safety) ha_on_primary_ctrl_fd_rdhup MUST mirror the ha_start_election_async no-config voter-only guard: a config-less node (myconf->config_version_tag==0) with candidates_count>1 must NOT start an RDHUP re-election (it would tie-grant with another tag=0 node — freshness gate only denies a LOWER tag — and elect a config-less primary that pushes an empty config). Guard sits AFTER the WAL-truncate block, BEFORE elect_ctx alloc/hook/raft_on_leader_lost; sets role=CTRL_STANDBY and returns. RDHUP integration/failover tests that assert election-after-RDHUP use make_cluster/memset which leaves config_version_tag=0 + candidates_count=2 — they were incidentally testing the buggy condition; give them a nonzero tag (0x1234) so they exercise the legitimate CONFIGURED-node path. RDHUP handlers called with c==NULL early-return before the guard (unit-mode tests assert only the CTRL_ELECTING transition). CTRL_PRIMARY enum==0, CTRL_STANDBY==1, CTRL_ELECTING==3 (test printf "role=0==CTRL_PRIMARY" is expected). Full controller suite: meson driver SIGTERMs at 300s — run ./xce.build/controller/controller_unit_test directly with ASAN_OPTIONS=detect_leaks=0.
|
||
- (2026-06-13, barrier-rollout 5.2) push_config_to_standbys now RETURNS the configAck barrier result (0/-ETIMEDOUT/-EINVAL), push count via acked_out; the configAck branch of repl_wait_majority_ack reads entry_lsn as the push tag and r->ce->n_peers (set-once, lock-free) as the denominator, ce==NULL -> -EINVAL fail-loud. ha.c/join.c streaming pushes discard the return BY DESIGN (log-only); only do_upgrade_config_with_bytes treats it as authoritative.
|
||
|
||
- [2026-06-12] Census parsing caveat: controller_unit_test logs (tinylog) and Unity result lines BOTH go to stdout and can splice mid-line under the multi-threaded config-push tests — a distinct-Unity-line count silently drops spliced results. Robust census = union of Unity result names AND the `test_x: PASS` printf lines, cross-checked against the per-group `N Tests M Failures` summary blocks (which are splice-immune).
|
||
- [2026-06-12] Config-barrier test trick: when a test needs a covering config ACK for a tag only known AFTER conf_save_config hashes the re-encoded blob (upgradeConfig path), pre-ack with acked_config_lsn = UINT64_MAX — covers any push tag under both legacy (>0) and per-push (>= tag) counting.
|
||
- [2026-06-12] A zeroed stack cbuf_t DEADLOCKS on cbuf_put (ncap==0 → len==ncap → cond_wait forever); any fixture driving do_upgrade_config_with_bytes' restart path must cbuf_init(&c->wbuf, …) first.
|
||
- [2026-06-13, single-leader-role 1.1] A WAL type with value 31..99 (below WAL_ENTRY_TYPE_MAX=100) is observationally IDENTICAL to a generic entry in the current tree for append (persist_wal_async writes any type), apply_wal_entry (common-path LSN advance via `default:` arm), and replay_wal (its OWN type filter only skips `type > WAL_CHECKPOINT(100)`). So a behavioral RED test for "WAL_NOOP append/apply works" PASSES even before WAL_NOOP exists. The ONLY genuinely-broken behavior pre-impl is wal_classify (totality `assert(0)` aborts on the unclassified value). RED carrier for append/apply cases = couple them to `wal_classify(WAL_NOOP)→RETAIN` via the fork helper (the enum value + classify row land in the SAME task 1.2 change per design).
|
||
- [2026-06-13] persist_t starts current_lsn==0 (calloc-zeroed; persist_init does NOT reset it). The FIRST persist_wal_async gets lsn==0; in replay_wal the `hdr.lsn <= from_lsn` (0<=0) skip means current_lsn never advances for a lone first entry. Replay fixtures asserting an LSN advance must write a preceding entry (or otherwise land the entry-under-test at lsn>0). replay_wal (persist.c:783) is SEPARATE from apply_wal_entry and has its own zero-region detector (`lsn==0 && type==0 && payload_len==0 → break`) and type filter.
|
||
- Census EXACT method (2026-06-12): count DISTINCT Unity `file:line:test:PASS` triples (`grep -aoE` + sort -u), not raw `2mPASS` occurrences — tinylog interleaving garbles ~9 verdict lines per run and duplicates others; distinct-triple count matched the 494-baseline arithmetic exactly (495 after task 4.3's +1).
|
||
- repl registry reconnect semantics: repl_remove_standby vacates the slot (ep=NULL, keeps last_disconnected_lsn guard); repl_add_standby reuses the first empty slot and memsets it (acked_lsn=0). With both standbys removed, majority_candidate_locked zero-fills to n_peers-1 → candidate 0 → committed_lsn frozen (RETAIN entries safely pending); after re-add, ONE covering ack = 2-of-3 majority.
|
||
- repl_wait_majority_ack walAck track is engine-only since barrier-rollout 2.2: entry_lsn==0 returns 0 immediately (documented once-per-lifetime LSN-0 first entry), everything else delegates to commit_submit unconditionally — ce==NULL surfaces as -EINVAL from commit_submit (unreachable in production; cluster_init_phase2 always wires the engine). The counting loop that remains in the function is configAck-ONLY (no self-count, n_active==0 shortcut is Req 8.5 config behavior). persist_wal_sync_append's residual non-engine branch can only store committed_lsn for the LSN-0 no-op case. (2026-06-12, task 2.2)
|
||
- `commit_submit(COMMIT_MODE_RETAIN)` is fully lock-free: it returns 0 BEFORE `pthread_mutex_lock(&ce->lock)` (commit.c:233-239), so it is safe to call from hook functions even while entity write locks are held — only WAIT mode takes ce->lock. (barrier-rollout task 1.2, 2026-06-12)
|
||
- ha-commit-engine: once repl.c references commit_* symbols (task 5.2), commit.c MUST be in controller_src (meson.build) or the production xce_controller link fails — the "linkage lands with 5.3" note in tasks.md only held while repl.c had no engine references. Lock order ce->lock → r->lock: repl.c calls engine hooks only AFTER releasing r->lock.
|
||
|
||
- Use `clang-format` for C/C++ formatting; shared config lives at `xce/.clang-format`.
|
||
- Logging macros in utils.h: TRACE (= tlog_debug3, verbose), INFO, WARN. There is NO DEBUG3 macro — use TRACE for debug-level logging in repl.c and other controller files. (2026-06-10)
|
||
- `msg_handler_t` in netio.h is `int (*)(void*)` — NOT `int (*)(endpoint_t*)`. Functions assigned to `ep->msg_handler` must use `void *ep` parameter. (Task 2.2, 2026-05-30)
|
||
- `xce_request()` is defined in request.c but not declared in request.h; files that need it must add `extern int xce_request(endpoint_t *ep, xce_request_t *req);` locally (pattern from config_repl.c).
|
||
- Test seam pattern for ha.c XCE_TESTING: declare `int (*hook)(ep, req) = NULL` under `#ifdef XCE_TESTING`, expose as extern in test file, test sets it before calling function under test. Mirrors `xce_retry_send_hook` in request.c.
|
||
- capnp UAF fix pattern (Task 2.3): INVOKE_MSG_HANDLER fires ep->msg_handler(ep) directly from iomgr thread; packet is in ep->fsm_in.pkt. Do raw capn_init_mem decode, read_RequestVoteReply into local struct BEFORE capn_free, then capn_free + free_packet + free(pkt). Use local scalars (not capnp pointers) after free. (2026-05-30)
|
||
- capnp serialization for tests: use new_RequestVoteReply(seg) / new_RequestReply(seg) / new_Request(seg) — NOT raw capn_new_struct. Follow pattern from xce_encode / xce_replconfigpush_req_raw in request.c.
|
||
- To avoid C99 "goto jumps over declaration" error: declare all local vars at top of function (or in a sub-block), not after the first goto target label.
|
||
- hook seam forward-declaration order: declare the hook variable BEFORE the function that references it (even within the same XCE_TESTING block). Use a dispatch helper static function to decouple seam from production path.
|
||
- term fencing must live in the handler, not the dispatch path (2026-05-31): vote_reply_msg_handler delegates to vote_reply_dispatch which calls either xce_vote_reply_hook (XCE_TESTING) or raft_on_vote_reply (production). If term fencing only happens inside raft_cb_become_follower (production), it is silently skipped in the test hook path. Pattern: do explicit CAS loop (atomic_compare_exchange_weak) on ha->term before calling vote_reply_dispatch, so the update applies regardless of which dispatch path runs.
|
||
|
||
- ha_start_election_async ownership rule (Task 2.4, 2026-05-30): in the production path (c != NULL), ha_elect_ctx_t is owned by election_timeout_fn which frees it on fire. In the test-mode path (c == NULL, peers == NULL), no delaymgr task owns ctx — must free ctx inline after ha_elect_evaluate_and_signal fires. ASAN will catch the leak if forgotten.
|
||
- ha_start_election_async single-node shortcircuit (Task 2.4): n_peers==1 means n_other=0 — allocate ha_elect_ctx_t with 0 flexible entries (no votes[] slots needed), set votes_granted=1, call evaluate directly, then free ctx immediately. No iomgr or delaymgr involvement for single-node.
|
||
- election_timeout_fn must lock ctx->lock before calling ha_elect_evaluate_and_signal (Task 2.4): the evaluate function has a done_posted guard but its callers in the reply path hold ctx->lock; timeout fn must also hold the lock to avoid races with concurrent reply processing.
|
||
- single-node shortcircuit enters CTRL_PRIMARY_COMMITTING, NOT CTRL_PRIMARY (Task 2.4 review fix): task 4.4 committing timeout promotes to PRIMARY for degraded/single-node mode (Req 8.4). Tests must assert CTRL_PRIMARY_COMMITTING.
|
||
- ha_elect_evaluate_and_signal out-of-bounds guard (Task 2.4 review fix): ctx->votes[] is a C99 flexible array with n_peers entries. When n_peers==0 (single-node), votes[0] is out-of-bounds. Must check n_peers==0 FIRST and short-circuit to PRIMARY (quorum=1, self-vote=1 always wins). Guard added before any votes[0] access.
|
||
- client_ssl_ctx for outgoing vote connections (Task 2.4 review fix): endpoint_connect() creates SSL from iomgr->ssl_ctx (server context). After connect returns, replace ep->ssl with SSL_new(client_ssl_ctx), SSL_set_fd to the same fd, SSL_set_connect_state. If SSL_new fails, fall through (handshake fails → vote denied via err_handler).
|
||
- vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.
|
||
- timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.
|
||
|
||
- repl_add_standby idempotency (2026-06-10): scan all MAX_STANDBYS slots for ep match BEFORE the nstandbys guard and empty-slot search. Return the existing slot index immediately on match (no state change). This prevents double-registration when raft_cb_become_leader promotes vote eps and on_standby_connected fires for a reconnecting late-joiner. Covered by test_repl_add_standby_idempotent.
|
||
- Heartbeat freshness source of truth (2026-06-09): `ha_heartbeat_is_fresh` derives from `ha->last_activity` (whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The old `last_heartbeat` timespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate is `role==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha)`; RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh.
|
||
- raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so `raft_init` must set `voted_for_term = initial_term` when `initial_voted_for != -1`. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6.
|
||
|
||
- repl_add_standby returns the SLOT INDEX (>=0) on success, not 0 (2026-06-11, ha-commit-engine task 1.1): asserting `== 0` breaks on the second standby. Assert `>= 0` or the expected index.
|
||
- Unity harness quirks in controller_unit_test (2026-06-11): (a) failure lines misattribute the FILE — Unity.TestFile comes from whichever group's UNITY_BEGIN ran, so a failure printed as test_repl_unit.c:294 may live in test_p2_integration.c; locate the actual test by grepping for the test NAME. (b) Each group's UNITY_BEGIN resets counters, so the process exit code only reflects the LAST group — failures in earlier groups are masked (exit=0); meson reports green despite real failures. Grep output for FAIL with `grep FAIL` on the raw log (ANSI codes sit between ':' and 'FAIL', so `grep ":FAIL"` misses them).
|
||
- 3 pre-existing controller suite failures on persistent_ha baseline (2026-06-11, verified by stash/rebuild without ha-commit-engine changes, see bug-411): test_committed_lsn_does_not_advance_on_timeout (rc -110 vs expected 0 — persist_wal_sync_append now returns -ETIMEDOUT, test expects old rc==0+sync_rc contract), test_repl_ack_after_promote_gate_allows_jobsub and test_rdhup_primary_dispatch_gate_opens (both -10013: ha_has_write_quorum false, standby_endpoints empty with n_peers=2).
|
||
- ha-commit-engine: task 1.2 shipped commit_on_membership_change as a FULL re-evaluation (commit_advance_locked + cond broadcast), and majority_candidate_locked already excludes in_catch_up + ep==NULL slots. Task 3.1 membership tests are green-on-arrival; task 3.2 is a no-op (verified 2026-06-11, tests 17-20 in test_commit_engine.c).
|
||
- key_find (xkvmgr_find) returns 1 when found, NOT 0 — queue_ctrl/worker_ctrl/job_ctrl leak this into rc: `if (rc == 0)` hook gates never fire on the found path (bug-447); any code reusing key_find's rc for success checks must normalise it.
|
||
- Controller test binary reply observation: no xce_reply hook exists in src/lib; observe replies on a zeroed no-iomgr endpoint_t — xce_reply endpoint_enq()s onto ep->outq; drain with endpoint_deq + raw capnp read_Request/read_RequestReply (test_commit_barrier_reply.c).
|
||
- makeJob needs job_obj_init() once per process AND xce_job_t.submit != NULL; makeString needs xstring_table_init (test_worker_join.c inits it unguarded at group entry — probe makeString for NULL before initialising to stay -t-selection safe).
|
||
- (2026-06-13, barrier-rollout 6.1) mark_job_start/finish/fail/update are STATIC in job.c — testsuite reaches them ONLY via the non-static wrappers on_job_start/on_job_finish/on_job_fail (and on_job_postdone → mark_job_update). Those wrappers `return 0` unconditionally regardless of mark_job_* rc, so the guard's "success-no-op rc" observable is asserted on the wrapper rc (== 0). No public caller produces a terminal→non-terminal UPDATE (on_job_postdone OR's PDONE = forward; job_ctrl SIGCONT only clears suspend bits) — mark_job_start (state := JOB_STAT_RUN) is the faithful publicly-reachable terminal-clearing witness for the shared backward rule. current_lsn is the "did a hook fire / WAL append" oracle: a guarded-out no-op runs no hooks → no persist_wal_async → current_lsn frozen; combine with cbrp_count_jobstate_entries (on-disk scan) + a key_find live-state read. Controller suite baseline 2026-06-13: 502 distinct PASS triples pre-6.1; +3 with 6.1's 2 preservation pins landing green (forward-update, PERR-merge) = 505, 3 FAIL = the 3 new RED guard tests.
|
||
|
||
- (2026-06-13, single-leader-role 2.2) The own-term promotion noop uses `persist_wal_async(&c->persist, WAL_NOOP, NULL, 0)` which ALWAYS calls repl_broadcast → xce_wal_batch_req_raw(s->ep) for every registered, non-catch-up standby. Any test fixture that registers fake `int` endpoints (the `g_*_eps[]` pattern) via repl_add_standby MUST install `repl_broadcast_send_hook` (drop hook) before driving a real append, or ASAN aborts with global-buffer-overflow reading ep->peer (offset 36 past the 4-byte int). Precedent: test_commit_engine.c ce_drop_send_hook, test_p2_integration.c p2_drop_send_hook. The task-2.1 fixture lifecycle_cluster_setup (test_ha_role_lifecycle.c) OMITS this hook AND off-by-ones the current_lsn prime (sets appended_lsn+1; persist_wal_async takes current_lsn then ++, so to land the noop at `appended_lsn` the prime must be `appended_lsn`). Both are fixture bugs (bug-477/478), out of ha.c boundary. Production noop_lsn = current_lsn-1 read under p->lock AFTER the append is correct.
|
||
- (2026-06-13, single-leader-role 2.2) raft_cb_become_leader does NOT call ha_do_promote — the raft FSM already synced ha->term at entry (~ha.c:463 atomic_store of the raft-reported term); calling ha_do_promote would double-increment (research §4 finding 2). The CTRL_PRIMARY store sits OUTSIDE the `if(c)` block so the c==NULL test shim reaches it. The non-raft road (ha_promote_to_primary) DOES call ha_do_promote (it owns its term). Both roads share ha_register_persist_hooks (BEFORE noop append) + ha_arm_own_term_gate (noop + commit_set_writable_lsn).
|
||
|
||
- (2026-06-13, single-leader-role COMPLETE 11/11, validate-impl GO) `CTRL_PRIMARY_COMMITTING` REMOVED — `ctrl_role_t` now 4 values (CTRL_PRIMARY=0, CTRL_STANDBY=1, CTRL_INIT=2, CTRL_ELECTING=3). Deleted without replacement: `committing_timeout_fn` (60s watchdog), `ha_check_majority_convergence`, `ha_repl_ack_received`/`repl_ack_count`, `committing_timeout_ms`/`HA_DEFAULT_COMMITTING_TIMEOUT_MS`. `OPCode_replCatchUpDone` is now a gate-allowed INFO no-op. Dispatch-gate PRIMARY branch + barrier-rollout fact allowlist preserved byte-for-byte; commit.c consumed with ZERO diff. Repo-wide completion grep (`CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence`, excl .git/.kiro/specs) = ZERO.
|
||
- (2026-06-13) OPERATIONAL: controller meson test wrapper ROUTINELY hits its 300s timeout under load (barrier commit-deadline waits) — `Timeout: 1` is NOT a failure. Authoritative green = direct binary `src/xce.build/controller/controller_unit_test` (EXIT 0, every group "N Tests 0 Failures"). A pre-existing ~153KB suite-wide LSan leak is SUPPRESSED by `ASAN_OPTIONS=detect_leaks=0` at controller/meson.build:163 — LSan exit=1 only appears on a RAW direct-binary run without that env; baseline, reproduces at pre-spec commits, not a per-change regression.
|
||
- (2026-06-14, reserve-lsn-0 / bug-543 FIX) LSN 0 is now RESERVED as the none/empty sentinel — `persist_init()` sets `current_lsn = 1` on a fresh store (persist.c, the ONLY fresh-init site; recovery overwrites from snapshot/WAL tip). First-ever WAL entry (promotion NOOP) is lsn 1. This was a ONE-LINE production change + a comment block; everything downstream ALREADY treated 0 as "none/empty" and lsn>=1 as real, so NO change was needed in commit.c (highest_appended_lsn `current_lsn>0?-1:0`, commit_submit rejects lsn 0, wait_done writable_lsn==0=gate-off), the streamer (`hdr.lsn>standby_lsn`: 1>0=true now streams the first entry → no hole), ha.c own-term gate (noop_lsn=current_lsn-1=1, writable_lsn=1 effective), or the wire/capnp format. The pre-alloc zero-region sentinel `hdr.lsn==0 && type==0 && payload_len==0` (persist.c:874, config_repl.c:1580, truncate:1666) is STRENGTHENED — no real entry is ever lsn 0. Segment naming follows current_lsn at open time, so the first segment is `seg-...001.wal` (not 000); recovery/truncate scan by sscanf, no hard-coded seg-000 assumption.
|
||
- (2026-06-14, reserve-lsn-0 TEST FALLOUT) Base-shift broke tests that did a "priming append so the entry lands at LSN >= 1" — that priming was ONLY to escape the old lsn-0 special-case and is now redundant. Fix = REMOVE the priming append so the real entry lands at lsn 1 (test_commit_engine tests 32/34, test_commit_barrier_rollout fact-hook, test_commit_barrier_reply cbrp_setup, test_p2_integration single_node) OR bump a hard-coded `current_lsn==1` assertion to `==2` if the priming must stay (test_egress_filter ef_setup). The ENGINE-LESS walAck path only "worked" via `repl_wait_majority_ack`'s `entry_lsn==0 → return 0` hatch (repl.c:437); with the first entry now lsn 1 it routes to `commit_submit(ce=NULL)` → -EINVAL, so engine-less commit tests MUST wire a commit engine (test_p2_integration test 9: added commit_engine_init(n_peers=3) + repl_set_commit_engine + commit_engine_fini). Tests asserting RELATIVE lsn (lsn_before+1, current_lsn>0, h1.lsn>h0.lsn) or byte-offset truncation math are base-shift-immune.
|
||
- (2026-06-13, kiro-impl process) A RED-only test pin is only valid if its OWN impl task can turn it green via the SAME seam the test drives. Misfires caught in review: boot-replay pin RED on an lsn-0 `replay_wal` skip artifact; hooks pin via the `c==NULL` raft shim where `ha_register_persist_hooks(&c->hooksys)` can never run (re-point to the real-cluster `ha_promote_to_primary` seam). Deleting machinery → DELETE the machinery-subject tests (design "no shims, rewritten or deleted not stubbed"), never add file-local no-op stubs. Unity runner prints the PREVIOUS passing test's file:line as a stale prefix on FAIL lines → attribute failures by test NAME.
|
||
|
||
## Do-Not-Repeat
|
||
|
||
<!-- Mistakes made and corrected. Each entry prevents the same mistake recurring. -->
|
||
<!-- Format: [YYYY-MM-DD] Description of what went wrong and what to do instead. -->
|
||
|
||
- [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.
|
||
- [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).
|
||
- [2026-06-15] SOLVED (bug-563): "jobsub ok but joblist/workerlist empty on the HA primary; reslist works". ROOT CAUSE: `cluster_start()` is the ONLY setter of `c->running = 1` and it is ORPHANED — zero callers (grep `cluster_start(`). The HA `myapp_run` path starts iomgr/vtaskmgr/delaymgr/persist itself but never calls `cluster_start`, so `c->running` stays 0 on the primary. `check_controller()` (the streaming gate `xce_reply_get` uses for joblist/workerlist) returns `-EINVAL` when `!c->running`; `xce_reply_get` then sets the chunk count `nc = 0`, shipping an EMPTY list even though `collect_jobs()` found the rows. `reslist` passes `checkf=NULL` so it bypasses the gate — that asymmetry is the tell. Single-node "works" because the legacy `cluster_init` path called `cluster_start`. FIX: `c.running = 1;` in `myapp_run()` after `persist_start()` (src/controller/myapp.c). RULE: when only SOME list ops are empty but the data is present server-side, check the `xce_reply_get` `check_*` predicate — a `<0` return silently zeroes the chunk (`nc=0`), not an error. To prove it: log `nc`/`checkstatus` in the `xce_reply_get` send loop. Reproduce HA bugs locally with `/xce-build` + a 3-node deploy-ha under ASAN rather than theorizing. (My earlier UAF/stale-binary guess here was WRONG — disproven by "crash gone, binary current".) Still-latent: `cluster_start` also armed `check_workers_timeout` (delaymgr) — that side-effect remains unrestored on the HA path.
|
||
- [2026-06-16] When adding a NEW opcode to the HA protocol, it MUST be wired into `ha_dispatch_gate()` (src/controller/ha.c) — the role-based admission gate runs in `myreq_handler` (cluster.c:251) BEFORE the per-opcode handler, and an unlisted opcode is rejected (`-XCE_ERR_STANDBY`/`-XCE_ERR_NOT_READY`) so the handler never runs. bug-605: ha-prevote added `OPCode_preVoteRequest` (opcode 38) + handler (cluster.c:440) but not the gate; STANDBY voters silently dropped every incoming PreVote, the candidate counted the failed exchange as a denial, and a cold cluster looped PreVote forever with NO primary. INVARIANT: a PreVote is strictly non-mutating, so it must be admitted in EXACTLY the roles a real `requestVote` is (CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY). Diagnostic tell: voters log zero "PreVote DENIED" yet the candidate keeps "losing" — means the request never reached the handler (gate drop), not a policy denial. Confirm with gdb breakpoint on `ha_dispatch_gate` rc.
|
||
- [2026-06-16] HA election timeouts must be RANDOMIZED/STAGGERED on the FAILOVER path, not just at startup. bug-606: when the primary dies, the surviving standbys lose it at the same instant and (before the fix) campaigned in perfect lock-step — both win PreVote (neither sees a primary), both vote self at the same term, split the vote (quorum unreachable with 2 survivors), time out on a FIXED watchdog (raft.c:105) and retry on the FIXED heartbeat_timeout liveness cadence → term climbs unbounded, no leader. The jitter machinery (election_priority_step_ms=500, election_jitter_max_ms=200, xorshift64 on jitter_seed) existed but was wired ONLY into the startup heartbeat-wait (ha_wait_for_heartbeat_or_timeout / myapp.c stagger); the PreVote failover path (liveness_poll_fn→ha_on_primary_ctrl_fd_rdhup→ha_begin_failover→ha_start_prevote_async) bypassed it, and raft's own staggered-start (raft_on_leader_lost→on_start_election_after_delay, raft.c:421) is also bypassed by the PreVote path. FIX: ha_election_stagger_ms() + a delaymgr-scheduled, guarded PreVote start in ha_begin_failover (skip if a primary appears during the stagger). RULE: any new re-election/failover trigger must apply per-node stagger (priority*step + fresh random jitter, re-randomized each attempt) AND a "primary alive?" guard before campaigning. Diagnostic tell: identical ms-aligned election timestamps across nodes + term climbing by 1 each cycle = lock-step split vote, not a vote-policy bug. step MUST exceed jitter_max so adjacent priorities never overlap.
|
||
- [2026-06-14] A partially-initialized `xce_reply_t` is a UAF hazard. The Cap'n-C generated `encode_RequestReply` (controller.capnp.c) reads the COMMON TRAILER fields (`redirectHost`, `redirectPort`) on EVERY reply regardless of `data_which` — `redirect_host` is `strlen`'d whenever `!= NULL`. Handlers declare `xce_reply_t reply;` on the stack and only set status/data, so those trailer fields were stack garbage; a garbage `redirect_host` aliasing a freed capnp segment = heap-use-after-free (seen on the OPCode_nop/state-report path, cluster.c). FIX (bug-562): `init_reply_by_req()` now `memset`s the whole struct first. RULE: any xce_reply_t built WITHOUT init_reply_by_req (e.g. ha_send_wal_ack_reply) MUST memset it. Generated union encoders are not safe to feed a partially-init struct — common/trailer fields are always serialized.
|
||
|
||
- [2026-06-04] capnpc-c 1.0.1 has a bug with `:group` inside a `union` when `$C.codecgen` is active: the encode/decode generator emits `d->FIELDNAME` (flat) instead of `d->data.GROUPNAME.FIELDNAME`, AND places the case as dead code after the prior `break`. Do NOT use inline `:group` in a union that has `$C.codecgen`. Instead, define a named outer struct (`struct Foo $C.mapname("foo_t")`) and reference it as a pointer union member. The named struct generates correct `encode_Foo_ptr`/`decode_Foo_ptr` calls. The `read_RequestReply`/`write_RequestReply` wire layer works correctly with groups but the codec layer does not.
|
||
|
||
- ha-consistency-invariants decomposed into 4 sub-specs (2026-06-11, user decision): umbrella spec keeps requirements/design/research as the authoritative contract (generates NO tasks); implementation via ha-commit-engine → ha-commit-barrier-rollout → ha-single-leader-role (HARD ordering: barrier before COMMITTING removal), with ha-convergence-recovery parallel after engine. Briefs reference umbrella docs with per-spec requirement ID subsets. Dependency order lives in steering/roadmap.md "## Specs (dependency order)" (kiro-spec-batch parseable). Rationale: ~30-38 tasks in one spec exceeds the ~20-task decomposition threshold; phases are independently shippable (each leaves the system no less consistent).
|
||
- Commit-first-then-reply contract (2026-06-11, user, ha-consistency-invariants): job/queue/host(worker) control operations MUST follow validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply. No success reply before commit. Known violators to restructure: do_jobcontrol_op (job.c:921-934, replies with only async WAL), handle_jobdispatch (job.c:459-471, replies before processing — classification pending). handle_jobsubmit is the reference implementation.
|
||
- Four design decisions locked (2026-06-11, user, ha-consistency-invariants): (1) fact entries follow pure Raft — log-is-the-queue, append to WAL + retry replication forever, NO commit timeout for facts; (2) crash-loss of uncommitted facts accepted until worker-rejoin-resync lands (strictly better than today); (3) demotion eviction = process SELF-RESTART after truncation (rebuild-in-place only if restart proves disruptive); (4) own-term commit via explicit NO-OP WAL entry at promotion. Rationale: "follow Raft as it has been well verified."
|
||
- Fact-vs-request commit semantics (2026-06-11, user decision, ha-consistency-invariants Req 2.2/2.8): job state transitions are REPORTED FACTS, not requests. Request-shaped ops (submit, queue ctrl, config) reject to requester on commit timeout; fact-shaped ops (worker-reported transitions) are retained and retried until committed — including across quorum loss; primary never discards an uncommitted fact while running+leading. Backstop for facts never received / lost across failover: worker-rejoin-resync (roadmap backlog, designated official backstop).
|
||
- vcpus is operator tuning, not a spec constant (2026-06-11, user): `--vcpus` should be sized per deployment by host capability + cluster size + workload; 8 is only the fallback default. Specs/designs must express thread-pool-relative mechanisms as functions of n_vcpus, never absolute counts, and put sizing advice in operator docs.
|
||
- vtask pool sizing facts (verified 2026-06-11): default is 8 (NOT 4 — ha-async-io.md was stale, now fixed) via `--vcpus` CLI override, cluster.c:1082. WAL ACK processing runs on the IOMGR thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, ha.c:2770-2792) — so blocking commit-waits on vtask threads can never deadlock their own wakeups; all-workers-blocked is a throughput problem only. Pool size should scale with write concurrency/cores, NOT with n_peers (ACKs gather in parallel; Little's law). Mitigation pattern: commit-wait admission cap (n_vcpus - K), async continuation as durable fix.
|
||
- xzarray COW shadow is DORMANT infrastructure (discovered 2026-06-11): `xzarray_freeze`/`xzarray_thaw` (zarray.c:695-757) are built, in the varray vtable, unit-tested (test_zarray_cow.c) — but have ZERO production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). It is slot-level STRUCTURAL COW only: slots hold item pointers, so in-place field mutations are visible through both shadow and active views; freeze is O(1) pointer swap, reads merge active-over-shadow, thaw merges back with a `should_retain` GC filter, single generation (-EALREADY on double freeze). Intended consumer: the never-implemented entity-bearing checkpoint (Task 11 / sentinel-only snapshots).
|
||
|
||
## Decision Log
|
||
|
||
<!-- Significant technical decisions with rationale. Why X was chosen over Y. -->
|
||
|
||
- Recovery-oriented reframing supersedes ARIES/undo direction (2026-06-10, user insight, ha-consistency-invariants): replication here serves FAILOVER/RECOVERY, not transactions. Structural fact: a single-writer leader never truncates its own log while leader, so provisional in-memory state on a live primary either commits later or the node is demoted — discard coincides exactly with role/lifecycle events. Therefore NO per-entry undo log; discard handled by recovery mechanics (truncate divergent suffix + coarse rebuild via existing persist_recover/replay_wal at demotion/rejoin). Only the watermark egress filter survives (entity WAL-LSN tag, committed-only export to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch), not isolation. Also: Raft commitIndex is volatile — prefer leader-driven log reconciliation at rejoin over persisting committed_lsn.
|
||
- STM evaluated and rejected for provisional-state mechanism (2026-06-10, ha-consistency-invariants): user asked if STM could help; analysis concluded full STM is a mismatch (commit point = majority network ACK up to seconds — STM transactions can't span syscalls; libitm/TSX dead in C; single-writer + RTTI per-object locks already cover the concurrency). Adopted the useful subset instead: "committed-view + undo log" (ARIES/MVCC-lite) — entities tagged with WAL LSN, committed-only egress filter for dataex/scheduler, per-in-flight undo records for prompt rollback on abort. Dual-zarray COW checkpoint mechanism is the in-house precedent for maintaining a second consistent view.
|
||
- Provisional-state model for Req 2.4 (2026-06-10, user decision, ha-consistency-invariants): in-memory controller state is a PROVISIONAL materialized view; the persisted majority-committed WAL (+snapshot) is the authoritative cluster state. Chosen over hook inversion (apply-after-commit). Mutation order stays mutate-first; instead, commit gates (1) requester confirmation and (2) irrevocable-action egress — dataex/scheduler export must filter uncommitted entities (closes the dispatch-of-uncommitted-job race). Status queries MAY observe provisional state (documented deviation; read consistency out of scope). Consequence: §C convergence/reconciliation work (truncate-on-demotion, phantom eviction, persisted committed_lsn) is mandatory core, not optional — "provisional" is only true if something discards it.
|
||
- Raft + CP adopted as governing HA model (2026-06-10, user decision, supersedes "No Raft" in product.md): consistency requirements are DEDUCED from the Raft paper + CP, not chosen per-feature. Concretely: (1) no CTRL_PRIMARY_COMMITTING state — Raft has only Follower/Candidate/Leader; new-leader safety comes from the own-term commit rule (§5.4.2), not a fourth state; (2) majority ACK is the hard commit barrier for ALL state-changing writes — reject on timeout/quorum loss, never local-apply-and-degrade (committed_lsn ≈ commitIndex); (3) replication fans out to all standbys with leader-driven continuous catch-up; majority is the commit threshold, not the fan-out target. Single-writer log retained as XCE's simplification. Spec: .kiro/specs/ha-consistency-invariants/brief.md. Implementation ordering constraint: universal commit barrier MUST land before removing COMMITTING.
|
||
|
||
- Replication payload wire format (2026-06-09, ha-repl-hardening 7.1): chose the established `_raw` sender convention (manual capnp build via capn_new_list8/capn_setv8 + xce_decode per-opcode raw extraction into raw_payload/raw_payload_len struct fields) over extending the typemap codegen to map capnp Data ↔ raw buffers. Rationale: replConfigPush/upgradeConfig already prove the pattern in production; the codegen route risks regressions across all opcodes and fights the known capnpc-c 1.0.1 codecgen bugs. Schema change: Request.body union gained walcatchupentrymsg @28 / snapshottransfermsg @29 — these messages previously had NO union arm at all (sent as body_which=nomsg, i.e. nothing crossed the wire).
|
||
- Convergence threshold (2026-06-09, ha-repl-hardening 7.5): chose requirements.md Req 2.3 formula — ceil(n_peers/2)−1 converged standbys (integer form n_peers/2) — over the code's previous all-standbys rule. Rationale: primary + converged standbys form a cluster majority, matching Raft quorum; committed entries are already on a majority per-entry, and safe_committed_lsn derives from a majority of voters, so promotion with one dead minority standby is safe and avoids a 60s degraded-mode timeout. Lagging standbys keep catching up via the heartbeat decision loop after promotion.
|
||
|
||
- Election liveness fix (2026-06-09): `skip_election` path in myapp.c had two bugs: (1) `voted_for_name != ""` triggered even for self-votes and stale terms — fixed by adding `strcmp(name,"self")!=0 && voted_for_term==ha->term`; (2) `ha_arm_heartbeat_timeout_vtask` was never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.c `handle_request_vote_req` also simplified to term-only deny.
|
||
- CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.
|
||
|
||
- Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.
|
||
- (2026-06-11, ha-convergence-recovery spec) `repl_complete_catchup` (repl.c:521) has NO production caller — `in_catch_up` is set by broadcast-overflow eviction (repl.c:240) and never cleared; re-inclusion trigger is wired by the ACK-driven catch-up continuation.
|
||
- (2026-06-11) Scheduler-bound job egress has TWO seams, not one: full push `xcesched_push_jobs` (sched.c:169, on scheduler join) AND incremental JOB_NEW push `sched_job_new_fun`→`xcesched_push_joblist` (sched.c:236-248). Status reads (apiserver.c:83, opget.c:173) share `collect_jobs` (dataex.c:294) and must stay unfiltered per umbrella deviation.
|
||
- (2026-06-11) No saved-argv/execv machinery exists anywhere in src/controller or src/lib — demotion self-restart is net-new; `job_t` is a thin wrapper `{ xce_job_t job; }` at src/include/xce/types.h:483-485 (controller-local fields go on the wrapper, wire type untouched).
|
||
- (2026-06-16, ha-prevote cleanup) The HA election carrier `ha_elect_ctx_t` (`ha->elect_ctx`) is deliberately leaked one-per-round on BOTH the real-election and PreVote paths — vote endpoints hold a bare `vctx->ctx` pointer freed async by endpoint_fini, so freeing the carrier risks UAF; there is NO real `ha_election_cleanup()`. Pre-existing + bounded (rare failover events). Deferred by user decision to follow-up spec `.kiro/specs/ha-elect-ctx-lifecycle/` (refcount-the-carrier or teardown-ordered free, both paths). Do NOT attempt a drive-by free — UAF on failover is worse than the bounded leak. The ha-prevote 3.1 log-reason nit is RESOLVED (commit 47f7338): `ha_heartbeat_is_fresh` un-static'd + declared in ha.h; cluster.c denial-reason now mirrors `raft_cb_vote_eligible` order exactly.
|
||
- (2026-06-16, ha-prevote impl) The controller `"controller test"` 300s TIMEOUT under `meson test` is a WALL-CLOCK WRAPPER ARTIFACT, not a hang/failure: running the binary directly (`ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test`) completes all 564 tests incl. the slow `test_commit_barrier_reply` (61 tests, real 500ms commit barriers) with 0 failures. For a true controller pass/fail, run the binary directly (no cap) or filter with `-t <suite>`; do NOT treat the meson timeout as a regression. ha-prevote feature COMPLETE: 11 tasks, raft suite 123/0, controller 564/0, real-vote path byte-for-byte unchanged (0 raft.c deletions), commits 0d66026..e6b234a.
|
||
- (2026-06-15, ha-prevote design review) PRODUCTION vote-reply tally path = `raft_on_vote_reply` in the raft library, NOT `ha_elect_ctx_t`. `vote_reply_dispatch` (ha.c ~2630-2650) calls `raft_on_vote_reply` directly in production; the `ha_elect_ctx_t` / `handle_request_vote_reply` / `ha_elect_evaluate_and_signal` path runs ONLY behind `#ifdef XCE_TESTING` via `xce_vote_reply_hook` (plus the startup/first-boot `ha_elect_ctx_t.done` semaphore). The raft library owns quorum tracking and fires `on_become_leader`/`on_become_follower`. Do-Not-Repeat: when mirroring the election reply path, mirror `raft_on_vote_reply` (raft lib), not `ha_elect_ctx_t`. (An earlier exploration mislabeled ha_elect_ctx_t as the operative tally; caught in design validation.)
|
||
- (2026-06-15, user decision, ha-prevote RN-2) Leader-stickiness window for PreVote = reuse `ha_heartbeat_is_fresh()` / `heartbeat_timeout_ms` (10s), gated on role==CTRL_STANDBY. Rationale: the stickiness window MUST equal XCE's failure-detection window or you get either a disruption gap (window too short) or a liveness stall (window too long). XCE splits textbook Raft's overloaded "election timeout" — `HEARTBEAT_TIMEOUT_MS` (10s) is the leader-failure detector; `ELECTION_TIMEOUT_MS` (5s) is only the candidate vote-collection deadline. Raft's "minimum election timeout" disruption rule maps to HEARTBEAT_TIMEOUT_MS, NOT ELECTION_TIMEOUT_MS. Reusing it keeps stickiness == failure-detect == existing real-vote gate (`raft_cb_vote_eligible`), one predicate; role-gate handles RDHUP/connection-loss for free. A shorter window buys no failover speedup (nobody campaigns before failure-detect anyway; real crashes RDHUP immediately) and only opens a disruption gap on flaky links. Rejected a separate `prevote_stickiness_ms` constant (lets the two timeouts drift). Full comparison table in .kiro/specs/ha-prevote/research.md §6.
|
||
- (2026-06-15, user decision, ha-prevote) Product is PRE-RELEASE: no backward/rolling-upgrade compatibility is required for any HA/wire change. All controllers in a cluster deploy from the same build, so there is no mixed-version (old-binary ↔ new-binary) cluster. Wire-format changes (new capnp opcodes/fields, vote-RPC changes) may be made freely without negotiation/fallback paths. Do NOT raise inter-version compat as a requirement or research item. (Dropped RN-5 from ha-prevote/research.md on this basis.)
|
||
|
||
- commit_submit(COMMIT_MODE_RETAIN) is REGISTRATION-FREE (commit.c:233-239, verified 2026-06-11): returns 0 immediately, records NOTHING per entry — commit state derives entirely from registry watermarks at ACK time. RETAIN routing at the hook seam therefore has NO engine-side observable distinct from the append itself; tests of RETAIN routing must assert classification + non-blocking return + lsn > committed_lsn pending + ACK-driven advancement, never a fake "registered" counter. (ha-commit-barrier-rollout 1.1)
|
||
- XCE_TESTING fail-fast (assert/abort) paths are testable via fork(): child silences stderr, calls the guarded fn, _exit(0); parent asserts WIFSIGNALED && WTERMSIG == SIGABRT. Precedent: lib testsuite test.c:1783; ASAN-compatible with detect_leaks=0. (ha-commit-barrier-rollout 1.1)
|
||
- ha_has_write_quorum (ha.c:1507) is STATIC — testsuite reaches it only through ha_dispatch_gate(ha, opcode) (extern-declared in test files, pattern: test_ha_repl_ack.c). Gate fixtures: ha_init + n_peers + atomic role store + heap zarray standby_endpoints (test_ha_connection_reuse.c pattern). Catch-up exclusion at the gate is untestable until the ha_state_t.repl back-pointer lands (ha-commit-engine 4.2) — assert via repl_active_standby_count instead. (2026-06-11, task 4.1)
|
||
- persist_commit_barrier watermark = current_lsn - 1 (matching persist.c:1021 persist_wal_sync_append read); covered ⇔ current_lsn == 0 OR committed_lsn >= current_lsn - 1. Design §Commit Barrier Helper "lsn = p->current_lsn" is shorthand — the parenthetical persist.c:1019-1021 reference pins it to current_lsn - 1 (engine clamps committed to highest appended, so waiting on current_lsn itself would never complete). (2026-06-12, task 2.1)
|
||
|
||
- (2026-06-13, single-leader-role 4.2) ha_dispatch_gate's OUTER `default: return 0` ADMITS — so after deleting the `case CTRL_PRIMARY_COMMITTING:` block, a role manually set to COMMITTING falls through to admit (not NOT_READY). That is exactly why test_ha_startup.c:test_dispatch_gate_committing_denies_job_ops flips from PASS→FAIL (Expected -10013 Was 0) — category-(a), 5.1-owned. The 6 PRIMARY||COMMITTING compound sites in ha.c: raft_cb_vote_eligible(~215), handle_request_vote term-stepdown(~1984)+role-deny(~2004), peer_ctrl_msg_handler case label(~2820), vote_ep_err_handler(~3088), primary_heartbeat_fn re-arm(~3883). Collapsing all to PRIMARY-only makes a manually-COMMITTING test role a NON-leader. Census after 4.2: exactly 5 FAIL, ALL COMMITTING-state category-(a) (test_ha_startup ×2, test_ha_rdhup_integration, test_ha_repl_push, test_ha_connection_reuse), zero category-(b). enum value CTRL_PRIMARY_COMMITTING=4 stays in ha.h (5.2 deletes it); no ha.c CODE branches on it post-4.2 (only comments in 2.2/3.1-owned functions remain). Unity attributes vote-gate cases to test_ha_failover.c by group UNITY_BEGIN — locate by NAME. FAIL bytes: `name:\x1b[41mFAIL\x1b[0m` (color BETWEEN ':' and FAIL) — grep raw `FAIL`, count was exactly 5.
|
||
|
||
### libxce async failover (2026-06-16)
|
||
- `xce_request(ep,req)` is fire-and-forget enqueue; reply status is only available at the `cbuf_get` round trip in xceapi.c. `xce_request_with_retry()` production branch is a no-op stub (retries only under the test hook). Any client-side retry/failover MUST wrap the send+wait round trip at the `api_ctx_t` level, not `xce_request`. Controller redirect (`E_XCE_REDIRECT`, join.c) sends status-only; redirect_host/port schema fields exist (controller.capnp:568) but are never populated. See spec libxce-failover, bug-608.
|
||
- xceapi.c per-op guard rejects ctx->state != XCE_CTX_RUN with -EINVAL; the transport-error path (xcectx.c:148) drives ctx to XCE_CTX_ERR -> this is the exact source of the failover "Invalid argument" symptom. Any failover must restore XCE_CTX_RUN after reconnect.
|
||
- ctx->wbuf is a single-slot status-only channel (cbuf_init ...,1,sizeof(int), xcectx.c:185): no seq correlation, so client is single-in-flight per ctx. Retries must drain/seq-match to avoid consuming a stale reply.
|
||
- Job submit uses a CLIENT-generated jobid (job.c:56) + server EEXIST/E_XCE_DUP_JOB replay guard (job.c:95) -> writes are idempotent on re-send; DUP on a retried submit means success.
|
||
- libxce layering: xcecli_* (client.c) is the UNIVERSAL client surface — worker (controller_connector.c), scheduler (scheduler.c), apiserver cgo (clib.go), xce_test all call xcecli_*, never xceapi xce_* directly. Only api_ctx_t (client.c) holds the master list; xce_ctx_t (xceapi/xcectx.h) does not. => client-side cross-cutting concerns (failover/retry) belong in client.c wrapping xce_*, keeping xce_* one-shot/fail-fast and xceapi unchanged.
|
||
|
||
### Decision: full-state rejoin on controller failover (2026-06-16)
|
||
- On failover, stateful daemons re-join the new primary and apiserver/scheduler receive FULL state (full job list), not a delta. Accepted because failover is rare, some handshake is unavoidable (new primary has no live connection to the daemon until rejoin), and full-state-on-rejoin is simpler/correct and still better than IBM LSF (full-history replace, no snapshot). Lightweight re-attach / delta transfer deferred (YAGNI) — roadmap backlog. Revisit only if measured as a scale problem.
|