wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity
myxce/.wolf/cerebrum.md

50 KiB
Raw Blame History

Cerebrum

OpenWolf's learning memory. Updated automatically as the AI learns from interactions. Do not edit manually unless correcting an error. Last updated: 2026-05-24

User Preferences

Key Learnings

  • (2026-06-15, ha-rejoin bug-340) A killed+restarted controller loops elections forever because the PRIMARY never evicts the dead/wedged outbound standby endpoint. primary_heartbeat_fn (ha.c) has two steps: (1) NOP each ep in ha->standby_endpoints, (2) reconnect to any peer NOT present in standby_endpoints. A wedged ep (peer died → handle_error did EPOLL_CTL_DEL but left it in the array; epoll MOD then returns ENOENT → the WARN endpoint_update_epoll rc1=-1 spam) makes step 2 think the peer is still connected → reconnect SUPPRESSED → restarted node never gets a ctrlhello → re-elects every heartbeat_timeout (term climbs unbounded). xce_nop_req returns >=0 on enqueue even when the epoll re-arm fails, so the heartbeat layer can't see the wedge. FIX: evict in step 1 using endpoint_t.ts_last_in (existing wall-clock field, stamped by handle_input) — if ts_last_in>0 && now-ts_last_in>heartbeat_timeout, zarray_del from standby_endpoints + repl_remove_standby; reconnect then re-establishes the channel and ctrl-1 settles via ha_discover_primary (NO term gate — accepts any primary msg while ELECTING regardless of its own high term). Gotchas: (a) ha_arm_heartbeat_timeout_vtask ALREADY resets last_activity=now, so 'become_follower doesn't reset liveness' is NOT a bug — the ~timeout loop period proves the watchdog is correct; the sole cause is the broken channel. (b) standby_endpoints holds TWO ep types (promoted vote_ep_ctx_t with ctx!=NULL, and heartbeat standby_ep_ctx_t with ctx==NULL) sharing first 4 fields; don't free vote-ep data (UAF — live vctx pointers). (c) Run one suite: controller_unit_test -t test_ha_liveness, ASAN_OPTIONS=detect_leaks=0.

  • (2026-06-15, ha-election bug-339) Candidate-advertised RequestVote LSN MUST equal the voter-side freshness floor. Voter floor = ha_vote_self_freshness(ha, config_tag, applied_lsn) = max(applied_lsn, last_known_committed_lsn) (cluster.c OPCode_requestVote). On a STANDBY, persist.committed_lsn NEVER advances (only the primary commit engine writes it at persist.c:1192); standbys advance applied_lsn (ha_apply_wal_catchup_entry) + last_known_committed_lsn (heartbeat replies). So a candidate that advertises persist.committed_lsn under-advertises vs its own floor → after a primary dies, every surviving standby advertises stale-low LSN while each peer's floor is high → freshness gate (cand_lsn<my_lsn) denies on BOTH sides → no quorum → infinite RDHUP→ELECTING→STANDBY loop, terms climb in lockstep, nobody ever become_leader. Boot path (myapp.c:217) already uses applied_lsn; only the RDHUP failover path (ha_on_primary_ctrl_fd_rdhup) was inconsistent. Fix: compute cand_tag/cand_lsn via ha_vote_self_freshness once and feed BOTH the elect_ctx (votes[0]+self_*) AND the ha_rdhup_election_hook args. Symmetric advertised==floor guarantees the freshest node is granted by all peers. Diagnostic signature in logs: both nodes only raft_cb_become_follower (never become_leader) + RDHUP every ~heartbeat_timeout.

  • (2026-06-14, election-safety) ha_on_primary_ctrl_fd_rdhup MUST mirror the ha_start_election_async no-config voter-only guard: a config-less node (myconf->config_version_tag==0) with candidates_count>1 must NOT start an RDHUP re-election (it would tie-grant with another tag=0 node — freshness gate only denies a LOWER tag — and elect a config-less primary that pushes an empty config). Guard sits AFTER the WAL-truncate block, BEFORE elect_ctx alloc/hook/raft_on_leader_lost; sets role=CTRL_STANDBY and returns. RDHUP integration/failover tests that assert election-after-RDHUP use make_cluster/memset which leaves config_version_tag=0 + candidates_count=2 — they were incidentally testing the buggy condition; give them a nonzero tag (0x1234) so they exercise the legitimate CONFIGURED-node path. RDHUP handlers called with c==NULL early-return before the guard (unit-mode tests assert only the CTRL_ELECTING transition). CTRL_PRIMARY enum==0, CTRL_STANDBY==1, CTRL_ELECTING==3 (test printf "role=0==CTRL_PRIMARY" is expected). Full controller suite: meson driver SIGTERMs at 300s — run ./xce.build/controller/controller_unit_test directly with ASAN_OPTIONS=detect_leaks=0.

  • (2026-06-13, barrier-rollout 5.2) push_config_to_standbys now RETURNS the configAck barrier result (0/-ETIMEDOUT/-EINVAL), push count via acked_out; the configAck branch of repl_wait_majority_ack reads entry_lsn as the push tag and r->ce->n_peers (set-once, lock-free) as the denominator, ce==NULL -> -EINVAL fail-loud. ha.c/join.c streaming pushes discard the return BY DESIGN (log-only); only do_upgrade_config_with_bytes treats it as authoritative.

  • [2026-06-12] Census parsing caveat: controller_unit_test logs (tinylog) and Unity result lines BOTH go to stdout and can splice mid-line under the multi-threaded config-push tests — a distinct-Unity-line count silently drops spliced results. Robust census = union of Unity result names AND the test_x: PASS printf lines, cross-checked against the per-group N Tests M Failures summary blocks (which are splice-immune).

  • [2026-06-12] Config-barrier test trick: when a test needs a covering config ACK for a tag only known AFTER conf_save_config hashes the re-encoded blob (upgradeConfig path), pre-ack with acked_config_lsn = UINT64_MAX — covers any push tag under both legacy (>0) and per-push (>= tag) counting.

  • [2026-06-12] A zeroed stack cbuf_t DEADLOCKS on cbuf_put (ncap==0 → len==ncap → cond_wait forever); any fixture driving do_upgrade_config_with_bytes' restart path must cbuf_init(&c->wbuf, …) first.

  • [2026-06-13, single-leader-role 1.1] A WAL type with value 31..99 (below WAL_ENTRY_TYPE_MAX=100) is observationally IDENTICAL to a generic entry in the current tree for append (persist_wal_async writes any type), apply_wal_entry (common-path LSN advance via default: arm), and replay_wal (its OWN type filter only skips type > WAL_CHECKPOINT(100)). So a behavioral RED test for "WAL_NOOP append/apply works" PASSES even before WAL_NOOP exists. The ONLY genuinely-broken behavior pre-impl is wal_classify (totality assert(0) aborts on the unclassified value). RED carrier for append/apply cases = couple them to wal_classify(WAL_NOOP)→RETAIN via the fork helper (the enum value + classify row land in the SAME task 1.2 change per design).

  • [2026-06-13] persist_t starts current_lsn==0 (calloc-zeroed; persist_init does NOT reset it). The FIRST persist_wal_async gets lsn==0; in replay_wal the hdr.lsn <= from_lsn (0<=0) skip means current_lsn never advances for a lone first entry. Replay fixtures asserting an LSN advance must write a preceding entry (or otherwise land the entry-under-test at lsn>0). replay_wal (persist.c:783) is SEPARATE from apply_wal_entry and has its own zero-region detector (lsn==0 && type==0 && payload_len==0 → break) and type filter.

  • Census EXACT method (2026-06-12): count DISTINCT Unity file:line:test:PASS triples (grep -aoE + sort -u), not raw 2mPASS occurrences — tinylog interleaving garbles ~9 verdict lines per run and duplicates others; distinct-triple count matched the 494-baseline arithmetic exactly (495 after task 4.3's +1).

  • repl registry reconnect semantics: repl_remove_standby vacates the slot (ep=NULL, keeps last_disconnected_lsn guard); repl_add_standby reuses the first empty slot and memsets it (acked_lsn=0). With both standbys removed, majority_candidate_locked zero-fills to n_peers-1 → candidate 0 → committed_lsn frozen (RETAIN entries safely pending); after re-add, ONE covering ack = 2-of-3 majority.

  • repl_wait_majority_ack walAck track is engine-only since barrier-rollout 2.2: entry_lsn==0 returns 0 immediately (documented once-per-lifetime LSN-0 first entry), everything else delegates to commit_submit unconditionally — ce==NULL surfaces as -EINVAL from commit_submit (unreachable in production; cluster_init_phase2 always wires the engine). The counting loop that remains in the function is configAck-ONLY (no self-count, n_active==0 shortcut is Req 8.5 config behavior). persist_wal_sync_append's residual non-engine branch can only store committed_lsn for the LSN-0 no-op case. (2026-06-12, task 2.2)

  • commit_submit(COMMIT_MODE_RETAIN) is fully lock-free: it returns 0 BEFORE pthread_mutex_lock(&ce->lock) (commit.c:233-239), so it is safe to call from hook functions even while entity write locks are held — only WAIT mode takes ce->lock. (barrier-rollout task 1.2, 2026-06-12)

  • ha-commit-engine: once repl.c references commit_* symbols (task 5.2), commit.c MUST be in controller_src (meson.build) or the production xce_controller link fails — the "linkage lands with 5.3" note in tasks.md only held while repl.c had no engine references. Lock order ce->lock → r->lock: repl.c calls engine hooks only AFTER releasing r->lock.

  • Use clang-format for C/C++ formatting; shared config lives at xce/.clang-format.

  • Logging macros in utils.h: TRACE (= tlog_debug3, verbose), INFO, WARN. There is NO DEBUG3 macro — use TRACE for debug-level logging in repl.c and other controller files. (2026-06-10)

  • msg_handler_t in netio.h is int (*)(void*) — NOT int (*)(endpoint_t*). Functions assigned to ep->msg_handler must use void *ep parameter. (Task 2.2, 2026-05-30)

  • xce_request() is defined in request.c but not declared in request.h; files that need it must add extern int xce_request(endpoint_t *ep, xce_request_t *req); locally (pattern from config_repl.c).

  • Test seam pattern for ha.c XCE_TESTING: declare int (*hook)(ep, req) = NULL under #ifdef XCE_TESTING, expose as extern in test file, test sets it before calling function under test. Mirrors xce_retry_send_hook in request.c.

  • capnp UAF fix pattern (Task 2.3): INVOKE_MSG_HANDLER fires ep->msg_handler(ep) directly from iomgr thread; packet is in ep->fsm_in.pkt. Do raw capn_init_mem decode, read_RequestVoteReply into local struct BEFORE capn_free, then capn_free + free_packet + free(pkt). Use local scalars (not capnp pointers) after free. (2026-05-30)

  • capnp serialization for tests: use new_RequestVoteReply(seg) / new_RequestReply(seg) / new_Request(seg) — NOT raw capn_new_struct. Follow pattern from xce_encode / xce_replconfigpush_req_raw in request.c.

  • To avoid C99 "goto jumps over declaration" error: declare all local vars at top of function (or in a sub-block), not after the first goto target label.

  • hook seam forward-declaration order: declare the hook variable BEFORE the function that references it (even within the same XCE_TESTING block). Use a dispatch helper static function to decouple seam from production path.

  • term fencing must live in the handler, not the dispatch path (2026-05-31): vote_reply_msg_handler delegates to vote_reply_dispatch which calls either xce_vote_reply_hook (XCE_TESTING) or raft_on_vote_reply (production). If term fencing only happens inside raft_cb_become_follower (production), it is silently skipped in the test hook path. Pattern: do explicit CAS loop (atomic_compare_exchange_weak) on ha->term before calling vote_reply_dispatch, so the update applies regardless of which dispatch path runs.

  • ha_start_election_async ownership rule (Task 2.4, 2026-05-30): in the production path (c != NULL), ha_elect_ctx_t is owned by election_timeout_fn which frees it on fire. In the test-mode path (c == NULL, peers == NULL), no delaymgr task owns ctx — must free ctx inline after ha_elect_evaluate_and_signal fires. ASAN will catch the leak if forgotten.

  • ha_start_election_async single-node shortcircuit (Task 2.4): n_peers==1 means n_other=0 — allocate ha_elect_ctx_t with 0 flexible entries (no votes[] slots needed), set votes_granted=1, call evaluate directly, then free ctx immediately. No iomgr or delaymgr involvement for single-node.

  • election_timeout_fn must lock ctx->lock before calling ha_elect_evaluate_and_signal (Task 2.4): the evaluate function has a done_posted guard but its callers in the reply path hold ctx->lock; timeout fn must also hold the lock to avoid races with concurrent reply processing.

  • single-node shortcircuit enters CTRL_PRIMARY_COMMITTING, NOT CTRL_PRIMARY (Task 2.4 review fix): task 4.4 committing timeout promotes to PRIMARY for degraded/single-node mode (Req 8.4). Tests must assert CTRL_PRIMARY_COMMITTING.

  • ha_elect_evaluate_and_signal out-of-bounds guard (Task 2.4 review fix): ctx->votes[] is a C99 flexible array with n_peers entries. When n_peers==0 (single-node), votes[0] is out-of-bounds. Must check n_peers==0 FIRST and short-circuit to PRIMARY (quorum=1, self-vote=1 always wins). Guard added before any votes[0] access.

  • client_ssl_ctx for outgoing vote connections (Task 2.4 review fix): endpoint_connect() creates SSL from iomgr->ssl_ctx (server context). After connect returns, replace ep->ssl with SSL_new(client_ssl_ctx), SSL_set_fd to the same fd, SSL_set_connect_state. If SSL_new fails, fall through (handshake fails → vote denied via err_handler).

  • vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.

  • timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.

  • repl_add_standby idempotency (2026-06-10): scan all MAX_STANDBYS slots for ep match BEFORE the nstandbys guard and empty-slot search. Return the existing slot index immediately on match (no state change). This prevents double-registration when raft_cb_become_leader promotes vote eps and on_standby_connected fires for a reconnecting late-joiner. Covered by test_repl_add_standby_idempotent.

  • Heartbeat freshness source of truth (2026-06-09): ha_heartbeat_is_fresh derives from ha->last_activity (whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The old last_heartbeat timespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate is role==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha); RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh.

  • raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so raft_init must set voted_for_term = initial_term when initial_voted_for != -1. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6.

  • repl_add_standby returns the SLOT INDEX (>=0) on success, not 0 (2026-06-11, ha-commit-engine task 1.1): asserting == 0 breaks on the second standby. Assert >= 0 or the expected index.

  • Unity harness quirks in controller_unit_test (2026-06-11): (a) failure lines misattribute the FILE — Unity.TestFile comes from whichever group's UNITY_BEGIN ran, so a failure printed as test_repl_unit.c:294 may live in test_p2_integration.c; locate the actual test by grepping for the test NAME. (b) Each group's UNITY_BEGIN resets counters, so the process exit code only reflects the LAST group — failures in earlier groups are masked (exit=0); meson reports green despite real failures. Grep output for FAIL with grep FAIL on the raw log (ANSI codes sit between ':' and 'FAIL', so grep ":FAIL" misses them).

  • 3 pre-existing controller suite failures on persistent_ha baseline (2026-06-11, verified by stash/rebuild without ha-commit-engine changes, see bug-411): test_committed_lsn_does_not_advance_on_timeout (rc -110 vs expected 0 — persist_wal_sync_append now returns -ETIMEDOUT, test expects old rc==0+sync_rc contract), test_repl_ack_after_promote_gate_allows_jobsub and test_rdhup_primary_dispatch_gate_opens (both -10013: ha_has_write_quorum false, standby_endpoints empty with n_peers=2).

  • ha-commit-engine: task 1.2 shipped commit_on_membership_change as a FULL re-evaluation (commit_advance_locked + cond broadcast), and majority_candidate_locked already excludes in_catch_up + ep==NULL slots. Task 3.1 membership tests are green-on-arrival; task 3.2 is a no-op (verified 2026-06-11, tests 17-20 in test_commit_engine.c).

  • key_find (xkvmgr_find) returns 1 when found, NOT 0 — queue_ctrl/worker_ctrl/job_ctrl leak this into rc: if (rc == 0) hook gates never fire on the found path (bug-447); any code reusing key_find's rc for success checks must normalise it.

  • Controller test binary reply observation: no xce_reply hook exists in src/lib; observe replies on a zeroed no-iomgr endpoint_t — xce_reply endpoint_enq()s onto ep->outq; drain with endpoint_deq + raw capnp read_Request/read_RequestReply (test_commit_barrier_reply.c).

  • makeJob needs job_obj_init() once per process AND xce_job_t.submit != NULL; makeString needs xstring_table_init (test_worker_join.c inits it unguarded at group entry — probe makeString for NULL before initialising to stay -t-selection safe).

  • (2026-06-13, barrier-rollout 6.1) mark_job_start/finish/fail/update are STATIC in job.c — testsuite reaches them ONLY via the non-static wrappers on_job_start/on_job_finish/on_job_fail (and on_job_postdone → mark_job_update). Those wrappers return 0 unconditionally regardless of mark_job_* rc, so the guard's "success-no-op rc" observable is asserted on the wrapper rc (== 0). No public caller produces a terminal→non-terminal UPDATE (on_job_postdone OR's PDONE = forward; job_ctrl SIGCONT only clears suspend bits) — mark_job_start (state := JOB_STAT_RUN) is the faithful publicly-reachable terminal-clearing witness for the shared backward rule. current_lsn is the "did a hook fire / WAL append" oracle: a guarded-out no-op runs no hooks → no persist_wal_async → current_lsn frozen; combine with cbrp_count_jobstate_entries (on-disk scan) + a key_find live-state read. Controller suite baseline 2026-06-13: 502 distinct PASS triples pre-6.1; +3 with 6.1's 2 preservation pins landing green (forward-update, PERR-merge) = 505, 3 FAIL = the 3 new RED guard tests.

  • (2026-06-13, single-leader-role 2.2) The own-term promotion noop uses persist_wal_async(&c->persist, WAL_NOOP, NULL, 0) which ALWAYS calls repl_broadcast → xce_wal_batch_req_raw(s->ep) for every registered, non-catch-up standby. Any test fixture that registers fake int endpoints (the g_*_eps[] pattern) via repl_add_standby MUST install repl_broadcast_send_hook (drop hook) before driving a real append, or ASAN aborts with global-buffer-overflow reading ep->peer (offset 36 past the 4-byte int). Precedent: test_commit_engine.c ce_drop_send_hook, test_p2_integration.c p2_drop_send_hook. The task-2.1 fixture lifecycle_cluster_setup (test_ha_role_lifecycle.c) OMITS this hook AND off-by-ones the current_lsn prime (sets appended_lsn+1; persist_wal_async takes current_lsn then ++, so to land the noop at appended_lsn the prime must be appended_lsn). Both are fixture bugs (bug-477/478), out of ha.c boundary. Production noop_lsn = current_lsn-1 read under p->lock AFTER the append is correct.

  • (2026-06-13, single-leader-role 2.2) raft_cb_become_leader does NOT call ha_do_promote — the raft FSM already synced ha->term at entry (~ha.c:463 atomic_store of the raft-reported term); calling ha_do_promote would double-increment (research §4 finding 2). The CTRL_PRIMARY store sits OUTSIDE the if(c) block so the c==NULL test shim reaches it. The non-raft road (ha_promote_to_primary) DOES call ha_do_promote (it owns its term). Both roads share ha_register_persist_hooks (BEFORE noop append) + ha_arm_own_term_gate (noop + commit_set_writable_lsn).

  • (2026-06-13, single-leader-role COMPLETE 11/11, validate-impl GO) CTRL_PRIMARY_COMMITTING REMOVED — ctrl_role_t now 4 values (CTRL_PRIMARY=0, CTRL_STANDBY=1, CTRL_INIT=2, CTRL_ELECTING=3). Deleted without replacement: committing_timeout_fn (60s watchdog), ha_check_majority_convergence, ha_repl_ack_received/repl_ack_count, committing_timeout_ms/HA_DEFAULT_COMMITTING_TIMEOUT_MS. OPCode_replCatchUpDone is now a gate-allowed INFO no-op. Dispatch-gate PRIMARY branch + barrier-rollout fact allowlist preserved byte-for-byte; commit.c consumed with ZERO diff. Repo-wide completion grep (CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence, excl .git/.kiro/specs) = ZERO.

  • (2026-06-13) OPERATIONAL: controller meson test wrapper ROUTINELY hits its 300s timeout under load (barrier commit-deadline waits) — Timeout: 1 is NOT a failure. Authoritative green = direct binary src/xce.build/controller/controller_unit_test (EXIT 0, every group "N Tests 0 Failures"). A pre-existing ~153KB suite-wide LSan leak is SUPPRESSED by ASAN_OPTIONS=detect_leaks=0 at controller/meson.build:163 — LSan exit=1 only appears on a RAW direct-binary run without that env; baseline, reproduces at pre-spec commits, not a per-change regression.

  • (2026-06-14, reserve-lsn-0 / bug-543 FIX) LSN 0 is now RESERVED as the none/empty sentinel — persist_init() sets current_lsn = 1 on a fresh store (persist.c, the ONLY fresh-init site; recovery overwrites from snapshot/WAL tip). First-ever WAL entry (promotion NOOP) is lsn 1. This was a ONE-LINE production change + a comment block; everything downstream ALREADY treated 0 as "none/empty" and lsn>=1 as real, so NO change was needed in commit.c (highest_appended_lsn current_lsn>0?-1:0, commit_submit rejects lsn 0, wait_done writable_lsn==0=gate-off), the streamer (hdr.lsn>standby_lsn: 1>0=true now streams the first entry → no hole), ha.c own-term gate (noop_lsn=current_lsn-1=1, writable_lsn=1 effective), or the wire/capnp format. The pre-alloc zero-region sentinel hdr.lsn==0 && type==0 && payload_len==0 (persist.c:874, config_repl.c:1580, truncate:1666) is STRENGTHENED — no real entry is ever lsn 0. Segment naming follows current_lsn at open time, so the first segment is seg-...001.wal (not 000); recovery/truncate scan by sscanf, no hard-coded seg-000 assumption.

  • (2026-06-14, reserve-lsn-0 TEST FALLOUT) Base-shift broke tests that did a "priming append so the entry lands at LSN >= 1" — that priming was ONLY to escape the old lsn-0 special-case and is now redundant. Fix = REMOVE the priming append so the real entry lands at lsn 1 (test_commit_engine tests 32/34, test_commit_barrier_rollout fact-hook, test_commit_barrier_reply cbrp_setup, test_p2_integration single_node) OR bump a hard-coded current_lsn==1 assertion to ==2 if the priming must stay (test_egress_filter ef_setup). The ENGINE-LESS walAck path only "worked" via repl_wait_majority_ack's entry_lsn==0 → return 0 hatch (repl.c:437); with the first entry now lsn 1 it routes to commit_submit(ce=NULL) → -EINVAL, so engine-less commit tests MUST wire a commit engine (test_p2_integration test 9: added commit_engine_init(n_peers=3) + repl_set_commit_engine + commit_engine_fini). Tests asserting RELATIVE lsn (lsn_before+1, current_lsn>0, h1.lsn>h0.lsn) or byte-offset truncation math are base-shift-immune.

  • (2026-06-13, kiro-impl process) A RED-only test pin is only valid if its OWN impl task can turn it green via the SAME seam the test drives. Misfires caught in review: boot-replay pin RED on an lsn-0 replay_wal skip artifact; hooks pin via the c==NULL raft shim where ha_register_persist_hooks(&c->hooksys) can never run (re-point to the real-cluster ha_promote_to_primary seam). Deleting machinery → DELETE the machinery-subject tests (design "no shims, rewritten or deleted not stubbed"), never add file-local no-op stubs. Unity runner prints the PREVIOUS passing test's file:line as a stale prefix on FAIL lines → attribute failures by test NAME.

Do-Not-Repeat

  • [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.

  • [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).

  • [2026-06-15] SOLVED (bug-563): "jobsub ok but joblist/workerlist empty on the HA primary; reslist works". ROOT CAUSE: cluster_start() is the ONLY setter of c->running = 1 and it is ORPHANED — zero callers (grep cluster_start(). The HA myapp_run path starts iomgr/vtaskmgr/delaymgr/persist itself but never calls cluster_start, so c->running stays 0 on the primary. check_controller() (the streaming gate xce_reply_get uses for joblist/workerlist) returns -EINVAL when !c->running; xce_reply_get then sets the chunk count nc = 0, shipping an EMPTY list even though collect_jobs() found the rows. reslist passes checkf=NULL so it bypasses the gate — that asymmetry is the tell. Single-node "works" because the legacy cluster_init path called cluster_start. FIX: c.running = 1; in myapp_run() after persist_start() (src/controller/myapp.c). RULE: when only SOME list ops are empty but the data is present server-side, check the xce_reply_get check_* predicate — a <0 return silently zeroes the chunk (nc=0), not an error. To prove it: log nc/checkstatus in the xce_reply_get send loop. Reproduce HA bugs locally with /xce-build + a 3-node deploy-ha under ASAN rather than theorizing. (My earlier UAF/stale-binary guess here was WRONG — disproven by "crash gone, binary current".) Still-latent: cluster_start also armed check_workers_timeout (delaymgr) — that side-effect remains unrestored on the HA path.

  • [2026-06-16] When adding a NEW opcode to the HA protocol, it MUST be wired into ha_dispatch_gate() (src/controller/ha.c) — the role-based admission gate runs in myreq_handler (cluster.c:251) BEFORE the per-opcode handler, and an unlisted opcode is rejected (-XCE_ERR_STANDBY/-XCE_ERR_NOT_READY) so the handler never runs. bug-605: ha-prevote added OPCode_preVoteRequest (opcode 38) + handler (cluster.c:440) but not the gate; STANDBY voters silently dropped every incoming PreVote, the candidate counted the failed exchange as a denial, and a cold cluster looped PreVote forever with NO primary. INVARIANT: a PreVote is strictly non-mutating, so it must be admitted in EXACTLY the roles a real requestVote is (CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY). Diagnostic tell: voters log zero "PreVote DENIED" yet the candidate keeps "losing" — means the request never reached the handler (gate drop), not a policy denial. Confirm with gdb breakpoint on ha_dispatch_gate rc.

  • [2026-06-16] HA election timeouts must be RANDOMIZED/STAGGERED on the FAILOVER path, not just at startup. bug-606: when the primary dies, the surviving standbys lose it at the same instant and (before the fix) campaigned in perfect lock-step — both win PreVote (neither sees a primary), both vote self at the same term, split the vote (quorum unreachable with 2 survivors), time out on a FIXED watchdog (raft.c:105) and retry on the FIXED heartbeat_timeout liveness cadence → term climbs unbounded, no leader. The jitter machinery (election_priority_step_ms=500, election_jitter_max_ms=200, xorshift64 on jitter_seed) existed but was wired ONLY into the startup heartbeat-wait (ha_wait_for_heartbeat_or_timeout / myapp.c stagger); the PreVote failover path (liveness_poll_fn→ha_on_primary_ctrl_fd_rdhup→ha_begin_failover→ha_start_prevote_async) bypassed it, and raft's own staggered-start (raft_on_leader_lost→on_start_election_after_delay, raft.c:421) is also bypassed by the PreVote path. FIX: ha_election_stagger_ms() + a delaymgr-scheduled, guarded PreVote start in ha_begin_failover (skip if a primary appears during the stagger). RULE: any new re-election/failover trigger must apply per-node stagger (priority*step + fresh random jitter, re-randomized each attempt) AND a "primary alive?" guard before campaigning. Diagnostic tell: identical ms-aligned election timestamps across nodes + term climbing by 1 each cycle = lock-step split vote, not a vote-policy bug. step MUST exceed jitter_max so adjacent priorities never overlap.

  • [2026-06-14] A partially-initialized xce_reply_t is a UAF hazard. The Cap'n-C generated encode_RequestReply (controller.capnp.c) reads the COMMON TRAILER fields (redirectHost, redirectPort) on EVERY reply regardless of data_which — redirect_host is strlen'd whenever != NULL. Handlers declare xce_reply_t reply; on the stack and only set status/data, so those trailer fields were stack garbage; a garbage redirect_host aliasing a freed capnp segment = heap-use-after-free (seen on the OPCode_nop/state-report path, cluster.c). FIX (bug-562): init_reply_by_req() now memsets the whole struct first. RULE: any xce_reply_t built WITHOUT init_reply_by_req (e.g. ha_send_wal_ack_reply) MUST memset it. Generated union encoders are not safe to feed a partially-init struct — common/trailer fields are always serialized.

  • [2026-06-04] capnpc-c 1.0.1 has a bug with :group inside a union when $C.codecgen is active: the encode/decode generator emits d->FIELDNAME (flat) instead of d->data.GROUPNAME.FIELDNAME, AND places the case as dead code after the prior break. Do NOT use inline :group in a union that has $C.codecgen. Instead, define a named outer struct (struct Foo $C.mapname("foo_t")) and reference it as a pointer union member. The named struct generates correct encode_Foo_ptr/decode_Foo_ptr calls. The read_RequestReply/write_RequestReply wire layer works correctly with groups but the codec layer does not.

  • ha-consistency-invariants decomposed into 4 sub-specs (2026-06-11, user decision): umbrella spec keeps requirements/design/research as the authoritative contract (generates NO tasks); implementation via ha-commit-engine → ha-commit-barrier-rollout → ha-single-leader-role (HARD ordering: barrier before COMMITTING removal), with ha-convergence-recovery parallel after engine. Briefs reference umbrella docs with per-spec requirement ID subsets. Dependency order lives in steering/roadmap.md "## Specs (dependency order)" (kiro-spec-batch parseable). Rationale: ~30-38 tasks in one spec exceeds the ~20-task decomposition threshold; phases are independently shippable (each leaves the system no less consistent).

  • Commit-first-then-reply contract (2026-06-11, user, ha-consistency-invariants): job/queue/host(worker) control operations MUST follow validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply. No success reply before commit. Known violators to restructure: do_jobcontrol_op (job.c:921-934, replies with only async WAL), handle_jobdispatch (job.c:459-471, replies before processing — classification pending). handle_jobsubmit is the reference implementation.

  • Four design decisions locked (2026-06-11, user, ha-consistency-invariants): (1) fact entries follow pure Raft — log-is-the-queue, append to WAL + retry replication forever, NO commit timeout for facts; (2) crash-loss of uncommitted facts accepted until worker-rejoin-resync lands (strictly better than today); (3) demotion eviction = process SELF-RESTART after truncation (rebuild-in-place only if restart proves disruptive); (4) own-term commit via explicit NO-OP WAL entry at promotion. Rationale: "follow Raft as it has been well verified."

  • Fact-vs-request commit semantics (2026-06-11, user decision, ha-consistency-invariants Req 2.2/2.8): job state transitions are REPORTED FACTS, not requests. Request-shaped ops (submit, queue ctrl, config) reject to requester on commit timeout; fact-shaped ops (worker-reported transitions) are retained and retried until committed — including across quorum loss; primary never discards an uncommitted fact while running+leading. Backstop for facts never received / lost across failover: worker-rejoin-resync (roadmap backlog, designated official backstop).

  • vcpus is operator tuning, not a spec constant (2026-06-11, user): --vcpus should be sized per deployment by host capability + cluster size + workload; 8 is only the fallback default. Specs/designs must express thread-pool-relative mechanisms as functions of n_vcpus, never absolute counts, and put sizing advice in operator docs.

  • vtask pool sizing facts (verified 2026-06-11): default is 8 (NOT 4 — ha-async-io.md was stale, now fixed) via --vcpus CLI override, cluster.c:1082. WAL ACK processing runs on the IOMGR thread (peer_ctrl_msg_handler → repl_ack_decode_and_dispatch → repl_on_ack, ha.c:2770-2792) — so blocking commit-waits on vtask threads can never deadlock their own wakeups; all-workers-blocked is a throughput problem only. Pool size should scale with write concurrency/cores, NOT with n_peers (ACKs gather in parallel; Little's law). Mitigation pattern: commit-wait admission cap (n_vcpus - K), async continuation as durable fix.

  • xzarray COW shadow is DORMANT infrastructure (discovered 2026-06-11): xzarray_freeze/xzarray_thaw (zarray.c:695-757) are built, in the varray vtable, unit-tested (test_zarray_cow.c) — but have ZERO production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). It is slot-level STRUCTURAL COW only: slots hold item pointers, so in-place field mutations are visible through both shadow and active views; freeze is O(1) pointer swap, reads merge active-over-shadow, thaw merges back with a should_retain GC filter, single generation (-EALREADY on double freeze). Intended consumer: the never-implemented entity-bearing checkpoint (Task 11 / sentinel-only snapshots).

Decision Log

  • Recovery-oriented reframing supersedes ARIES/undo direction (2026-06-10, user insight, ha-consistency-invariants): replication here serves FAILOVER/RECOVERY, not transactions. Structural fact: a single-writer leader never truncates its own log while leader, so provisional in-memory state on a live primary either commits later or the node is demoted — discard coincides exactly with role/lifecycle events. Therefore NO per-entry undo log; discard handled by recovery mechanics (truncate divergent suffix + coarse rebuild via existing persist_recover/replay_wal at demotion/rejoin). Only the watermark egress filter survives (entity WAL-LSN tag, committed-only export to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch), not isolation. Also: Raft commitIndex is volatile — prefer leader-driven log reconciliation at rejoin over persisting committed_lsn.

  • STM evaluated and rejected for provisional-state mechanism (2026-06-10, ha-consistency-invariants): user asked if STM could help; analysis concluded full STM is a mismatch (commit point = majority network ACK up to seconds — STM transactions can't span syscalls; libitm/TSX dead in C; single-writer + RTTI per-object locks already cover the concurrency). Adopted the useful subset instead: "committed-view + undo log" (ARIES/MVCC-lite) — entities tagged with WAL LSN, committed-only egress filter for dataex/scheduler, per-in-flight undo records for prompt rollback on abort. Dual-zarray COW checkpoint mechanism is the in-house precedent for maintaining a second consistent view.

  • Provisional-state model for Req 2.4 (2026-06-10, user decision, ha-consistency-invariants): in-memory controller state is a PROVISIONAL materialized view; the persisted majority-committed WAL (+snapshot) is the authoritative cluster state. Chosen over hook inversion (apply-after-commit). Mutation order stays mutate-first; instead, commit gates (1) requester confirmation and (2) irrevocable-action egress — dataex/scheduler export must filter uncommitted entities (closes the dispatch-of-uncommitted-job race). Status queries MAY observe provisional state (documented deviation; read consistency out of scope). Consequence: §C convergence/reconciliation work (truncate-on-demotion, phantom eviction, persisted committed_lsn) is mandatory core, not optional — "provisional" is only true if something discards it.

  • Raft + CP adopted as governing HA model (2026-06-10, user decision, supersedes "No Raft" in product.md): consistency requirements are DEDUCED from the Raft paper + CP, not chosen per-feature. Concretely: (1) no CTRL_PRIMARY_COMMITTING state — Raft has only Follower/Candidate/Leader; new-leader safety comes from the own-term commit rule (§5.4.2), not a fourth state; (2) majority ACK is the hard commit barrier for ALL state-changing writes — reject on timeout/quorum loss, never local-apply-and-degrade (committed_lsn ≈ commitIndex); (3) replication fans out to all standbys with leader-driven continuous catch-up; majority is the commit threshold, not the fan-out target. Single-writer log retained as XCE's simplification. Spec: .kiro/specs/ha-consistency-invariants/brief.md. Implementation ordering constraint: universal commit barrier MUST land before removing COMMITTING.

  • Replication payload wire format (2026-06-09, ha-repl-hardening 7.1): chose the established _raw sender convention (manual capnp build via capn_new_list8/capn_setv8 + xce_decode per-opcode raw extraction into raw_payload/raw_payload_len struct fields) over extending the typemap codegen to map capnp Data ↔ raw buffers. Rationale: replConfigPush/upgradeConfig already prove the pattern in production; the codegen route risks regressions across all opcodes and fights the known capnpc-c 1.0.1 codecgen bugs. Schema change: Request.body union gained walcatchupentrymsg @28 / snapshottransfermsg @29 — these messages previously had NO union arm at all (sent as body_which=nomsg, i.e. nothing crossed the wire).

  • Convergence threshold (2026-06-09, ha-repl-hardening 7.5): chose requirements.md Req 2.3 formula — ceil(n_peers/2)−1 converged standbys (integer form n_peers/2) — over the code's previous all-standbys rule. Rationale: primary + converged standbys form a cluster majority, matching Raft quorum; committed entries are already on a majority per-entry, and safe_committed_lsn derives from a majority of voters, so promotion with one dead minority standby is safe and avoids a 60s degraded-mode timeout. Lagging standbys keep catching up via the heartbeat decision loop after promotion.

  • Election liveness fix (2026-06-09): skip_election path in myapp.c had two bugs: (1) voted_for_name != "" triggered even for self-votes and stale terms — fixed by adding strcmp(name,"self")!=0 && voted_for_term==ha->term; (2) ha_arm_heartbeat_timeout_vtask was never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.c handle_request_vote_req also simplified to term-only deny.

    • CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.

  • Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.

  • (2026-06-11, ha-convergence-recovery spec) repl_complete_catchup (repl.c:521) has NO production caller — in_catch_up is set by broadcast-overflow eviction (repl.c:240) and never cleared; re-inclusion trigger is wired by the ACK-driven catch-up continuation.

  • (2026-06-11) Scheduler-bound job egress has TWO seams, not one: full push xcesched_push_jobs (sched.c:169, on scheduler join) AND incremental JOB_NEW push sched_job_new_fun→xcesched_push_joblist (sched.c:236-248). Status reads (apiserver.c:83, opget.c:173) share collect_jobs (dataex.c:294) and must stay unfiltered per umbrella deviation.

  • (2026-06-11) No saved-argv/execv machinery exists anywhere in src/controller or src/lib — demotion self-restart is net-new; job_t is a thin wrapper { xce_job_t job; } at src/include/xce/types.h:483-485 (controller-local fields go on the wrapper, wire type untouched).

  • (2026-06-16, ha-prevote cleanup) The HA election carrier ha_elect_ctx_t (ha->elect_ctx) is deliberately leaked one-per-round on BOTH the real-election and PreVote paths — vote endpoints hold a bare vctx->ctx pointer freed async by endpoint_fini, so freeing the carrier risks UAF; there is NO real ha_election_cleanup(). Pre-existing + bounded (rare failover events). Deferred by user decision to follow-up spec .kiro/specs/ha-elect-ctx-lifecycle/ (refcount-the-carrier or teardown-ordered free, both paths). Do NOT attempt a drive-by free — UAF on failover is worse than the bounded leak. The ha-prevote 3.1 log-reason nit is RESOLVED (commit 47f7338): ha_heartbeat_is_fresh un-static'd + declared in ha.h; cluster.c denial-reason now mirrors raft_cb_vote_eligible order exactly.

  • (2026-06-16, ha-prevote impl) The controller "controller test" 300s TIMEOUT under meson test is a WALL-CLOCK WRAPPER ARTIFACT, not a hang/failure: running the binary directly (ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test) completes all 564 tests incl. the slow test_commit_barrier_reply (61 tests, real 500ms commit barriers) with 0 failures. For a true controller pass/fail, run the binary directly (no cap) or filter with -t <suite>; do NOT treat the meson timeout as a regression. ha-prevote feature COMPLETE: 11 tasks, raft suite 123/0, controller 564/0, real-vote path byte-for-byte unchanged (0 raft.c deletions), commits 0d66026..e6b234a.

  • (2026-06-15, ha-prevote design review) PRODUCTION vote-reply tally path = raft_on_vote_reply in the raft library, NOT ha_elect_ctx_t. vote_reply_dispatch (ha.c ~2630-2650) calls raft_on_vote_reply directly in production; the ha_elect_ctx_t / handle_request_vote_reply / ha_elect_evaluate_and_signal path runs ONLY behind #ifdef XCE_TESTING via xce_vote_reply_hook (plus the startup/first-boot ha_elect_ctx_t.done semaphore). The raft library owns quorum tracking and fires on_become_leader/on_become_follower. Do-Not-Repeat: when mirroring the election reply path, mirror raft_on_vote_reply (raft lib), not ha_elect_ctx_t. (An earlier exploration mislabeled ha_elect_ctx_t as the operative tally; caught in design validation.)

  • (2026-06-15, user decision, ha-prevote RN-2) Leader-stickiness window for PreVote = reuse ha_heartbeat_is_fresh() / heartbeat_timeout_ms (10s), gated on role==CTRL_STANDBY. Rationale: the stickiness window MUST equal XCE's failure-detection window or you get either a disruption gap (window too short) or a liveness stall (window too long). XCE splits textbook Raft's overloaded "election timeout" — HEARTBEAT_TIMEOUT_MS (10s) is the leader-failure detector; ELECTION_TIMEOUT_MS (5s) is only the candidate vote-collection deadline. Raft's "minimum election timeout" disruption rule maps to HEARTBEAT_TIMEOUT_MS, NOT ELECTION_TIMEOUT_MS. Reusing it keeps stickiness == failure-detect == existing real-vote gate (raft_cb_vote_eligible), one predicate; role-gate handles RDHUP/connection-loss for free. A shorter window buys no failover speedup (nobody campaigns before failure-detect anyway; real crashes RDHUP immediately) and only opens a disruption gap on flaky links. Rejected a separate prevote_stickiness_ms constant (lets the two timeouts drift). Full comparison table in .kiro/specs/ha-prevote/research.md §6.

  • (2026-06-15, user decision, ha-prevote) Product is PRE-RELEASE: no backward/rolling-upgrade compatibility is required for any HA/wire change. All controllers in a cluster deploy from the same build, so there is no mixed-version (old-binary ↔ new-binary) cluster. Wire-format changes (new capnp opcodes/fields, vote-RPC changes) may be made freely without negotiation/fallback paths. Do NOT raise inter-version compat as a requirement or research item. (Dropped RN-5 from ha-prevote/research.md on this basis.)

  • commit_submit(COMMIT_MODE_RETAIN) is REGISTRATION-FREE (commit.c:233-239, verified 2026-06-11): returns 0 immediately, records NOTHING per entry — commit state derives entirely from registry watermarks at ACK time. RETAIN routing at the hook seam therefore has NO engine-side observable distinct from the append itself; tests of RETAIN routing must assert classification + non-blocking return + lsn > committed_lsn pending + ACK-driven advancement, never a fake "registered" counter. (ha-commit-barrier-rollout 1.1)

  • XCE_TESTING fail-fast (assert/abort) paths are testable via fork(): child silences stderr, calls the guarded fn, _exit(0); parent asserts WIFSIGNALED && WTERMSIG == SIGABRT. Precedent: lib testsuite test.c:1783; ASAN-compatible with detect_leaks=0. (ha-commit-barrier-rollout 1.1)

  • ha_has_write_quorum (ha.c:1507) is STATIC — testsuite reaches it only through ha_dispatch_gate(ha, opcode) (extern-declared in test files, pattern: test_ha_repl_ack.c). Gate fixtures: ha_init + n_peers + atomic role store + heap zarray standby_endpoints (test_ha_connection_reuse.c pattern). Catch-up exclusion at the gate is untestable until the ha_state_t.repl back-pointer lands (ha-commit-engine 4.2) — assert via repl_active_standby_count instead. (2026-06-11, task 4.1)

  • persist_commit_barrier watermark = current_lsn - 1 (matching persist.c:1021 persist_wal_sync_append read); covered ⇔ current_lsn == 0 OR committed_lsn >= current_lsn - 1. Design §Commit Barrier Helper "lsn = p->current_lsn" is shorthand — the parenthetical persist.c:1019-1021 reference pins it to current_lsn - 1 (engine clamps committed to highest appended, so waiting on current_lsn itself would never complete). (2026-06-12, task 2.1)

  • (2026-06-13, single-leader-role 4.2) ha_dispatch_gate's OUTER default: return 0 ADMITS — so after deleting the case CTRL_PRIMARY_COMMITTING: block, a role manually set to COMMITTING falls through to admit (not NOT_READY). That is exactly why test_ha_startup.c:test_dispatch_gate_committing_denies_job_ops flips from PASS→FAIL (Expected -10013 Was 0) — category-(a), 5.1-owned. The 6 PRIMARY||COMMITTING compound sites in ha.c: raft_cb_vote_eligible(~215), handle_request_vote term-stepdown(~1984)+role-deny(~2004), peer_ctrl_msg_handler case label(~2820), vote_ep_err_handler(~3088), primary_heartbeat_fn re-arm(~3883). Collapsing all to PRIMARY-only makes a manually-COMMITTING test role a NON-leader. Census after 4.2: exactly 5 FAIL, ALL COMMITTING-state category-(a) (test_ha_startup ×2, test_ha_rdhup_integration, test_ha_repl_push, test_ha_connection_reuse), zero category-(b). enum value CTRL_PRIMARY_COMMITTING=4 stays in ha.h (5.2 deletes it); no ha.c CODE branches on it post-4.2 (only comments in 2.2/3.1-owned functions remain). Unity attributes vote-gate cases to test_ha_failover.c by group UNITY_BEGIN — locate by NAME. FAIL bytes: name:\x1b[41mFAIL\x1b[0m (color BETWEEN ':' and FAIL) — grep raw FAIL, count was exactly 5.

libxce async failover (2026-06-16)

  • xce_request(ep,req) is fire-and-forget enqueue; reply status is only available at the cbuf_get round trip in xceapi.c. xce_request_with_retry() production branch is a no-op stub (retries only under the test hook). Any client-side retry/failover MUST wrap the send+wait round trip at the api_ctx_t level, not xce_request. Controller redirect (E_XCE_REDIRECT, join.c) sends status-only; redirect_host/port schema fields exist (controller.capnp:568) but are never populated. See spec libxce-failover, bug-608.
  • xceapi.c per-op guard rejects ctx->state != XCE_CTX_RUN with -EINVAL; the transport-error path (xcectx.c:148) drives ctx to XCE_CTX_ERR -> this is the exact source of the failover "Invalid argument" symptom. Any failover must restore XCE_CTX_RUN after reconnect.
  • ctx->wbuf is a single-slot status-only channel (cbuf_init ...,1,sizeof(int), xcectx.c:185): no seq correlation, so client is single-in-flight per ctx. Retries must drain/seq-match to avoid consuming a stale reply.
  • Job submit uses a CLIENT-generated jobid (job.c:56) + server EEXIST/E_XCE_DUP_JOB replay guard (job.c:95) -> writes are idempotent on re-send; DUP on a retried submit means success.
  • libxce layering: xcecli_* (client.c) is the UNIVERSAL client surface — worker (controller_connector.c), scheduler (scheduler.c), apiserver cgo (clib.go), xce_test all call xcecli_, never xceapi xce_ directly. Only api_ctx_t (client.c) holds the master list; xce_ctx_t (xceapi/xcectx.h) does not. => client-side cross-cutting concerns (failover/retry) belong in client.c wrapping xce_, keeping xce_ one-shot/fail-fast and xceapi unchanged.

Decision: full-state rejoin on controller failover (2026-06-16)

  • On failover, stateful daemons re-join the new primary and apiserver/scheduler receive FULL state (full job list), not a delta. Accepted because failover is rare, some handshake is unavoidable (new primary has no live connection to the daemon until rejoin), and full-state-on-rejoin is simpler/correct and still better than IBM LSF (full-history replace, no snapshot). Lightweight re-attach / delta transfer deferred (YAGNI) — roadmap backlog. Revisit only if measured as a scale problem.