configAck branch: per-push majority (acked_config_lsn >= push_tag, 1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate, ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead. push_config_to_standbys returns the barrier result; upgrade failure -> send_upgrade_error (rc forwarded), no success ACK, no restart, committed_config_tag success-only. Remediation r1 (two CRITICAL review findings): - F1: wire production config-ACK ingestion — replconfigpushreplymsg decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack on the promoted-vote ep; + defensive cluster.c OPCode_reply case. Closes the unlanded half of ha-config-bootstrap 8.1; without this the barrier would brick every multi-member config upgrade. No schema change. - F2: keep the barrier off the iomgr thread — export push_config_transport_only (non-blocking) for the 5 ha.c + join.c streaming sites; barrier path now vtask-only (do_upgrade_config). Wire-path test drives real capnp ACK bytes through the production decode branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0 FAIL; pre-existing fixture leak only (less than baseline).
26 KiB
Cerebrum
OpenWolf's learning memory. Updated automatically as the AI learns from interactions. Do not edit manually unless correcting an error. Last updated: 2026-05-24
User Preferences
Key Learnings
-
(2026-06-13, barrier-rollout 5.2) push_config_to_standbys now RETURNS the configAck barrier result (0/-ETIMEDOUT/-EINVAL), push count via acked_out; the configAck branch of repl_wait_majority_ack reads entry_lsn as the push tag and r->ce->n_peers (set-once, lock-free) as the denominator, ce==NULL -> -EINVAL fail-loud. ha.c/join.c streaming pushes discard the return BY DESIGN (log-only); only do_upgrade_config_with_bytes treats it as authoritative.
-
[2026-06-12] Census parsing caveat: controller_unit_test logs (tinylog) and Unity result lines BOTH go to stdout and can splice mid-line under the multi-threaded config-push tests — a distinct-Unity-line count silently drops spliced results. Robust census = union of Unity result names AND the
test_x: PASSprintf lines, cross-checked against the per-groupN Tests M Failuressummary blocks (which are splice-immune). -
[2026-06-12] Config-barrier test trick: when a test needs a covering config ACK for a tag only known AFTER conf_save_config hashes the re-encoded blob (upgradeConfig path), pre-ack with acked_config_lsn = UINT64_MAX — covers any push tag under both legacy (>0) and per-push (>= tag) counting.
-
[2026-06-12] A zeroed stack cbuf_t DEADLOCKS on cbuf_put (ncap==0 → len==ncap → cond_wait forever); any fixture driving do_upgrade_config_with_bytes' restart path must cbuf_init(&c->wbuf, …) first.
-
Census EXACT method (2026-06-12): count DISTINCT Unity
file:line:test:PASStriples (grep -aoE+ sort -u), not raw2mPASSoccurrences — tinylog interleaving garbles ~9 verdict lines per run and duplicates others; distinct-triple count matched the 494-baseline arithmetic exactly (495 after task 4.3's +1). -
repl registry reconnect semantics: repl_remove_standby vacates the slot (ep=NULL, keeps last_disconnected_lsn guard); repl_add_standby reuses the first empty slot and memsets it (acked_lsn=0). With both standbys removed, majority_candidate_locked zero-fills to n_peers-1 → candidate 0 → committed_lsn frozen (RETAIN entries safely pending); after re-add, ONE covering ack = 2-of-3 majority.
-
repl_wait_majority_ack walAck track is engine-only since barrier-rollout 2.2: entry_lsn==0 returns 0 immediately (documented once-per-lifetime LSN-0 first entry), everything else delegates to commit_submit unconditionally — ce==NULL surfaces as -EINVAL from commit_submit (unreachable in production; cluster_init_phase2 always wires the engine). The counting loop that remains in the function is configAck-ONLY (no self-count, n_active==0 shortcut is Req 8.5 config behavior). persist_wal_sync_append's residual non-engine branch can only store committed_lsn for the LSN-0 no-op case. (2026-06-12, task 2.2)
-
commit_submit(COMMIT_MODE_RETAIN)is fully lock-free: it returns 0 BEFOREpthread_mutex_lock(&ce->lock)(commit.c:233-239), so it is safe to call from hook functions even while entity write locks are held — only WAIT mode takes ce->lock. (barrier-rollout task 1.2, 2026-06-12) -
ha-commit-engine: once repl.c references commit_* symbols (task 5.2), commit.c MUST be in controller_src (meson.build) or the production xce_controller link fails — the "linkage lands with 5.3" note in tasks.md only held while repl.c had no engine references. Lock order ce->lock → r->lock: repl.c calls engine hooks only AFTER releasing r->lock.
-
Use
clang-formatfor C/C++ formatting; shared config lives atxce/.clang-format. -
Logging macros in utils.h: TRACE (= tlog_debug3, verbose), INFO, WARN. There is NO DEBUG3 macro — use TRACE for debug-level logging in repl.c and other controller files. (2026-06-10)
-
msg_handler_tin netio.h isint (*)(void*)— NOTint (*)(endpoint_t*). Functions assigned toep->msg_handlermust usevoid *epparameter. (Task 2.2, 2026-05-30) -
xce_request()is defined in request.c but not declared in request.h; files that need it must addextern int xce_request(endpoint_t *ep, xce_request_t *req);locally (pattern from config_repl.c). -
Test seam pattern for ha.c XCE_TESTING: declare
int (*hook)(ep, req) = NULLunder#ifdef XCE_TESTING, expose as extern in test file, test sets it before calling function under test. Mirrorsxce_retry_send_hookin request.c. -
capnp UAF fix pattern (Task 2.3): INVOKE_MSG_HANDLER fires ep->msg_handler(ep) directly from iomgr thread; packet is in ep->fsm_in.pkt. Do raw capn_init_mem decode, read_RequestVoteReply into local struct BEFORE capn_free, then capn_free + free_packet + free(pkt). Use local scalars (not capnp pointers) after free. (2026-05-30)
-
capnp serialization for tests: use new_RequestVoteReply(seg) / new_RequestReply(seg) / new_Request(seg) — NOT raw capn_new_struct. Follow pattern from xce_encode / xce_replconfigpush_req_raw in request.c.
-
To avoid C99 "goto jumps over declaration" error: declare all local vars at top of function (or in a sub-block), not after the first goto target label.
-
hook seam forward-declaration order: declare the hook variable BEFORE the function that references it (even within the same XCE_TESTING block). Use a dispatch helper static function to decouple seam from production path.
-
term fencing must live in the handler, not the dispatch path (2026-05-31): vote_reply_msg_handler delegates to vote_reply_dispatch which calls either xce_vote_reply_hook (XCE_TESTING) or raft_on_vote_reply (production). If term fencing only happens inside raft_cb_become_follower (production), it is silently skipped in the test hook path. Pattern: do explicit CAS loop (atomic_compare_exchange_weak) on ha->term before calling vote_reply_dispatch, so the update applies regardless of which dispatch path runs.
-
ha_start_election_async ownership rule (Task 2.4, 2026-05-30): in the production path (c != NULL), ha_elect_ctx_t is owned by election_timeout_fn which frees it on fire. In the test-mode path (c == NULL, peers == NULL), no delaymgr task owns ctx — must free ctx inline after ha_elect_evaluate_and_signal fires. ASAN will catch the leak if forgotten.
-
ha_start_election_async single-node shortcircuit (Task 2.4): n_peers==1 means n_other=0 — allocate ha_elect_ctx_t with 0 flexible entries (no votes[] slots needed), set votes_granted=1, call evaluate directly, then free ctx immediately. No iomgr or delaymgr involvement for single-node.
-
election_timeout_fn must lock ctx->lock before calling ha_elect_evaluate_and_signal (Task 2.4): the evaluate function has a done_posted guard but its callers in the reply path hold ctx->lock; timeout fn must also hold the lock to avoid races with concurrent reply processing.
-
single-node shortcircuit enters CTRL_PRIMARY_COMMITTING, NOT CTRL_PRIMARY (Task 2.4 review fix): task 4.4 committing timeout promotes to PRIMARY for degraded/single-node mode (Req 8.4). Tests must assert CTRL_PRIMARY_COMMITTING.
-
ha_elect_evaluate_and_signal out-of-bounds guard (Task 2.4 review fix): ctx->votes[] is a C99 flexible array with n_peers entries. When n_peers==0 (single-node), votes[0] is out-of-bounds. Must check n_peers==0 FIRST and short-circuit to PRIMARY (quorum=1, self-vote=1 always wins). Guard added before any votes[0] access.
-
client_ssl_ctx for outgoing vote connections (Task 2.4 review fix): endpoint_connect() creates SSL from iomgr->ssl_ctx (server context). After connect returns, replace ep->ssl with SSL_new(client_ssl_ctx), SSL_set_fd to the same fd, SSL_set_connect_state. If SSL_new fails, fall through (handshake fails → vote denied via err_handler).
-
vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.
-
timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.
-
repl_add_standby idempotency (2026-06-10): scan all MAX_STANDBYS slots for ep match BEFORE the nstandbys guard and empty-slot search. Return the existing slot index immediately on match (no state change). This prevents double-registration when raft_cb_become_leader promotes vote eps and on_standby_connected fires for a reconnecting late-joiner. Covered by test_repl_add_standby_idempotent.
-
Heartbeat freshness source of truth (2026-06-09):
ha_heartbeat_is_freshderives fromha->last_activity(whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The oldlast_heartbeattimespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate isrole==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha); RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh. -
raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so
raft_initmust setvoted_for_term = initial_termwheninitial_voted_for != -1. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6. -
repl_add_standby returns the SLOT INDEX (>=0) on success, not 0 (2026-06-11, ha-commit-engine task 1.1): asserting
== 0breaks on the second standby. Assert>= 0or the expected index. -
Unity harness quirks in controller_unit_test (2026-06-11): (a) failure lines misattribute the FILE — Unity.TestFile comes from whichever group's UNITY_BEGIN ran, so a failure printed as test_repl_unit.c:294 may live in test_p2_integration.c; locate the actual test by grepping for the test NAME. (b) Each group's UNITY_BEGIN resets counters, so the process exit code only reflects the LAST group — failures in earlier groups are masked (exit=0); meson reports green despite real failures. Grep output for FAIL with
grep FAILon the raw log (ANSI codes sit between ':' and 'FAIL', sogrep ":FAIL"misses them). -
3 pre-existing controller suite failures on persistent_ha baseline (2026-06-11, verified by stash/rebuild without ha-commit-engine changes, see bug-411): test_committed_lsn_does_not_advance_on_timeout (rc -110 vs expected 0 — persist_wal_sync_append now returns -ETIMEDOUT, test expects old rc==0+sync_rc contract), test_repl_ack_after_promote_gate_allows_jobsub and test_rdhup_primary_dispatch_gate_opens (both -10013: ha_has_write_quorum false, standby_endpoints empty with n_peers=2).
-
ha-commit-engine: task 1.2 shipped commit_on_membership_change as a FULL re-evaluation (commit_advance_locked + cond broadcast), and majority_candidate_locked already excludes in_catch_up + ep==NULL slots. Task 3.1 membership tests are green-on-arrival; task 3.2 is a no-op (verified 2026-06-11, tests 17-20 in test_commit_engine.c).
-
key_find (xkvmgr_find) returns 1 when found, NOT 0 — queue_ctrl/worker_ctrl/job_ctrl leak this into rc:
if (rc == 0)hook gates never fire on the found path (bug-447); any code reusing key_find's rc for success checks must normalise it. -
Controller test binary reply observation: no xce_reply hook exists in src/lib; observe replies on a zeroed no-iomgr endpoint_t — xce_reply endpoint_enq()s onto ep->outq; drain with endpoint_deq + raw capnp read_Request/read_RequestReply (test_commit_barrier_reply.c).
-
makeJob needs job_obj_init() once per process AND xce_job_t.submit != NULL; makeString needs xstring_table_init (test_worker_join.c inits it unguarded at group entry — probe makeString for NULL before initialising to stay -t-selection safe).
Do-Not-Repeat
-
[2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.
-
[2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).
-
[2026-06-04] capnpc-c 1.0.1 has a bug with
:groupinside aunionwhen$C.codecgenis active: the encode/decode generator emitsd->FIELDNAME(flat) instead ofd->data.GROUPNAME.FIELDNAME, AND places the case as dead code after the priorbreak. Do NOT use inline:groupin a union that has$C.codecgen. Instead, define a named outer struct (struct Foo $C.mapname("foo_t")) and reference it as a pointer union member. The named struct generates correctencode_Foo_ptr/decode_Foo_ptrcalls. Theread_RequestReply/write_RequestReplywire layer works correctly with groups but the codec layer does not. -
ha-consistency-invariants decomposed into 4 sub-specs (2026-06-11, user decision): umbrella spec keeps requirements/design/research as the authoritative contract (generates NO tasks); implementation via ha-commit-engine → ha-commit-barrier-rollout → ha-single-leader-role (HARD ordering: barrier before COMMITTING removal), with ha-convergence-recovery parallel after engine. Briefs reference umbrella docs with per-spec requirement ID subsets. Dependency order lives in steering/roadmap.md "## Specs (dependency order)" (kiro-spec-batch parseable). Rationale: ~30-38 tasks in one spec exceeds the ~20-task decomposition threshold; phases are independently shippable (each leaves the system no less consistent).
-
Commit-first-then-reply contract (2026-06-11, user, ha-consistency-invariants): job/queue/host(worker) control operations MUST follow validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply. No success reply before commit. Known violators to restructure: do_jobcontrol_op (job.c:921-934, replies with only async WAL), handle_jobdispatch (job.c:459-471, replies before processing — classification pending). handle_jobsubmit is the reference implementation.
-
Four design decisions locked (2026-06-11, user, ha-consistency-invariants): (1) fact entries follow pure Raft — log-is-the-queue, append to WAL + retry replication forever, NO commit timeout for facts; (2) crash-loss of uncommitted facts accepted until worker-rejoin-resync lands (strictly better than today); (3) demotion eviction = process SELF-RESTART after truncation (rebuild-in-place only if restart proves disruptive); (4) own-term commit via explicit NO-OP WAL entry at promotion. Rationale: "follow Raft as it has been well verified."
-
Fact-vs-request commit semantics (2026-06-11, user decision, ha-consistency-invariants Req 2.2/2.8): job state transitions are REPORTED FACTS, not requests. Request-shaped ops (submit, queue ctrl, config) reject to requester on commit timeout; fact-shaped ops (worker-reported transitions) are retained and retried until committed — including across quorum loss; primary never discards an uncommitted fact while running+leading. Backstop for facts never received / lost across failover: worker-rejoin-resync (roadmap backlog, designated official backstop).
-
vcpus is operator tuning, not a spec constant (2026-06-11, user):
--vcpusshould be sized per deployment by host capability + cluster size + workload; 8 is only the fallback default. Specs/designs must express thread-pool-relative mechanisms as functions of n_vcpus, never absolute counts, and put sizing advice in operator docs. -
vtask pool sizing facts (verified 2026-06-11): default is 8 (NOT 4 — ha-async-io.md was stale, now fixed) via
--vcpusCLI override, cluster.c:1082. WAL ACK processing runs on the IOMGR thread (peer_ctrl_msg_handler→repl_ack_decode_and_dispatch→repl_on_ack, ha.c:2770-2792) — so blocking commit-waits on vtask threads can never deadlock their own wakeups; all-workers-blocked is a throughput problem only. Pool size should scale with write concurrency/cores, NOT with n_peers (ACKs gather in parallel; Little's law). Mitigation pattern: commit-wait admission cap (n_vcpus - K), async continuation as durable fix. -
xzarray COW shadow is DORMANT infrastructure (discovered 2026-06-11):
xzarray_freeze/xzarray_thaw(zarray.c:695-757) are built, in the varray vtable, unit-tested (test_zarray_cow.c) — but have ZERO production callers (kvmgr.c:297vfreeze = NULL /* wired in P0-2 */). It is slot-level STRUCTURAL COW only: slots hold item pointers, so in-place field mutations are visible through both shadow and active views; freeze is O(1) pointer swap, reads merge active-over-shadow, thaw merges back with ashould_retainGC filter, single generation (-EALREADY on double freeze). Intended consumer: the never-implemented entity-bearing checkpoint (Task 11 / sentinel-only snapshots).
Decision Log
-
Recovery-oriented reframing supersedes ARIES/undo direction (2026-06-10, user insight, ha-consistency-invariants): replication here serves FAILOVER/RECOVERY, not transactions. Structural fact: a single-writer leader never truncates its own log while leader, so provisional in-memory state on a live primary either commits later or the node is demoted — discard coincides exactly with role/lifecycle events. Therefore NO per-entry undo log; discard handled by recovery mechanics (truncate divergent suffix + coarse rebuild via existing persist_recover/replay_wal at demotion/rejoin). Only the watermark egress filter survives (entity WAL-LSN tag, committed-only export to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch), not isolation. Also: Raft commitIndex is volatile — prefer leader-driven log reconciliation at rejoin over persisting committed_lsn.
-
STM evaluated and rejected for provisional-state mechanism (2026-06-10, ha-consistency-invariants): user asked if STM could help; analysis concluded full STM is a mismatch (commit point = majority network ACK up to seconds — STM transactions can't span syscalls; libitm/TSX dead in C; single-writer + RTTI per-object locks already cover the concurrency). Adopted the useful subset instead: "committed-view + undo log" (ARIES/MVCC-lite) — entities tagged with WAL LSN, committed-only egress filter for dataex/scheduler, per-in-flight undo records for prompt rollback on abort. Dual-zarray COW checkpoint mechanism is the in-house precedent for maintaining a second consistent view.
-
Provisional-state model for Req 2.4 (2026-06-10, user decision, ha-consistency-invariants): in-memory controller state is a PROVISIONAL materialized view; the persisted majority-committed WAL (+snapshot) is the authoritative cluster state. Chosen over hook inversion (apply-after-commit). Mutation order stays mutate-first; instead, commit gates (1) requester confirmation and (2) irrevocable-action egress — dataex/scheduler export must filter uncommitted entities (closes the dispatch-of-uncommitted-job race). Status queries MAY observe provisional state (documented deviation; read consistency out of scope). Consequence: §C convergence/reconciliation work (truncate-on-demotion, phantom eviction, persisted committed_lsn) is mandatory core, not optional — "provisional" is only true if something discards it.
-
Raft + CP adopted as governing HA model (2026-06-10, user decision, supersedes "No Raft" in product.md): consistency requirements are DEDUCED from the Raft paper + CP, not chosen per-feature. Concretely: (1) no CTRL_PRIMARY_COMMITTING state — Raft has only Follower/Candidate/Leader; new-leader safety comes from the own-term commit rule (§5.4.2), not a fourth state; (2) majority ACK is the hard commit barrier for ALL state-changing writes — reject on timeout/quorum loss, never local-apply-and-degrade (committed_lsn ≈ commitIndex); (3) replication fans out to all standbys with leader-driven continuous catch-up; majority is the commit threshold, not the fan-out target. Single-writer log retained as XCE's simplification. Spec: .kiro/specs/ha-consistency-invariants/brief.md. Implementation ordering constraint: universal commit barrier MUST land before removing COMMITTING.
-
Replication payload wire format (2026-06-09, ha-repl-hardening 7.1): chose the established
_rawsender convention (manual capnp build via capn_new_list8/capn_setv8 + xce_decode per-opcode raw extraction into raw_payload/raw_payload_len struct fields) over extending the typemap codegen to map capnp Data ↔ raw buffers. Rationale: replConfigPush/upgradeConfig already prove the pattern in production; the codegen route risks regressions across all opcodes and fights the known capnpc-c 1.0.1 codecgen bugs. Schema change: Request.body union gained walcatchupentrymsg @28 / snapshottransfermsg @29 — these messages previously had NO union arm at all (sent as body_which=nomsg, i.e. nothing crossed the wire). -
Convergence threshold (2026-06-09, ha-repl-hardening 7.5): chose requirements.md Req 2.3 formula — ceil(n_peers/2)−1 converged standbys (integer form n_peers/2) — over the code's previous all-standbys rule. Rationale: primary + converged standbys form a cluster majority, matching Raft quorum; committed entries are already on a majority per-entry, and safe_committed_lsn derives from a majority of voters, so promotion with one dead minority standby is safe and avoids a 60s degraded-mode timeout. Lagging standbys keep catching up via the heartbeat decision loop after promotion.
-
Election liveness fix (2026-06-09):
skip_electionpath in myapp.c had two bugs: (1)voted_for_name != ""triggered even for self-votes and stale terms — fixed by addingstrcmp(name,"self")!=0 && voted_for_term==ha->term; (2)ha_arm_heartbeat_timeout_vtaskwas never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.chandle_request_vote_reqalso simplified to term-only deny.- CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.
-
Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.
-
(2026-06-11, ha-convergence-recovery spec)
repl_complete_catchup(repl.c:521) has NO production caller —in_catch_upis set by broadcast-overflow eviction (repl.c:240) and never cleared; re-inclusion trigger is wired by the ACK-driven catch-up continuation. -
(2026-06-11) Scheduler-bound job egress has TWO seams, not one: full push
xcesched_push_jobs(sched.c:169, on scheduler join) AND incremental JOB_NEW pushsched_job_new_fun→xcesched_push_joblist(sched.c:236-248). Status reads (apiserver.c:83, opget.c:173) sharecollect_jobs(dataex.c:294) and must stay unfiltered per umbrella deviation. -
(2026-06-11) No saved-argv/execv machinery exists anywhere in src/controller or src/lib — demotion self-restart is net-new;
job_tis a thin wrapper{ xce_job_t job; }at src/include/xce/types.h:483-485 (controller-local fields go on the wrapper, wire type untouched). -
commit_submit(COMMIT_MODE_RETAIN) is REGISTRATION-FREE (commit.c:233-239, verified 2026-06-11): returns 0 immediately, records NOTHING per entry — commit state derives entirely from registry watermarks at ACK time. RETAIN routing at the hook seam therefore has NO engine-side observable distinct from the append itself; tests of RETAIN routing must assert classification + non-blocking return + lsn > committed_lsn pending + ACK-driven advancement, never a fake "registered" counter. (ha-commit-barrier-rollout 1.1)
-
XCE_TESTING fail-fast (assert/abort) paths are testable via fork(): child silences stderr, calls the guarded fn, _exit(0); parent asserts WIFSIGNALED && WTERMSIG == SIGABRT. Precedent: lib testsuite test.c:1783; ASAN-compatible with detect_leaks=0. (ha-commit-barrier-rollout 1.1)
-
ha_has_write_quorum (ha.c:1507) is STATIC — testsuite reaches it only through ha_dispatch_gate(ha, opcode) (extern-declared in test files, pattern: test_ha_repl_ack.c). Gate fixtures: ha_init + n_peers + atomic role store + heap zarray standby_endpoints (test_ha_connection_reuse.c pattern). Catch-up exclusion at the gate is untestable until the ha_state_t.repl back-pointer lands (ha-commit-engine 4.2) — assert via repl_active_standby_count instead. (2026-06-11, task 4.1)
-
persist_commit_barrier watermark = current_lsn - 1 (matching persist.c:1021 persist_wal_sync_append read); covered ⇔ current_lsn == 0 OR committed_lsn >= current_lsn - 1. Design §Commit Barrier Helper "lsn = p->current_lsn" is shorthand — the parenthetical persist.c:1019-1021 reference pins it to current_lsn - 1 (engine clamps committed to highest appended, so waiting on current_lsn itself would never complete). (2026-06-12, task 2.1)