myxce/.wolf/cerebrum.md

Cerebrum

OpenWolf's learning memory. Updated automatically as the AI learns from interactions. Do not edit manually unless correcting an error. Last updated: 2026-05-24

User Preferences

Key Learnings

• (2026-06-23, bug-1028 FIXED) There MUST be exactly ONE authoritative source for a WAL entry's Raft term. The leader's library send path (ha_read_wal_entry_at, ha.c:1472-1476) and the standby's logOk/on_get_entry_term both source terms from persist_entry_term (in-memory index, legacy-0→WAL_LEGACY_TERM). The heartbeat catch-up streamer (ha_stream_wal_to_standby, config_repl.c) — the third consumer — was sourcing terms from raw on-disk hdr.term (0 for legacy entries). A standby's prev_log_term=1 (normalised) ≠ streamer's prev_log_term=0 (raw) → logOk rejects the new-term NOOP forever → applied_lsn stuck → primary re-streams every heartbeat cycle. Fix: all three hdr.term reads in the streamer replaced with persist_entry_term(&persist, hdr.lsn); function gained a const persist_t* parameter. This is the pattern: every source of the per-LSN (term,index) tuple MUST go through persist_entry_term; NEVER read hdr.term raw from disk — it is not normalised.

• (2026-06-23, bug-854 FIXED) The XCECLI_FAILOVER_CALL retry loop classifies a status-0 reply from ANY reachable node as SUCCESS and returns — it only rotates on failure. But ctrlstatus is answerable in every role, so a standby always answers 0, making xcecli_ctrlstatus pin to the first reachable node forever. The outer xcecli_ctrlstatus_primary loop captures a redirect hint from the non-authoritative reply but never forces rotation to the next master — so the configured master list is never traversed. Fix: xcecli_ctrlstatus_primary now calls xce_failover_rotate(ctx) + xcecli_ctx_recover(ctx) under failover.lock on every non-authoritative reply; the floor of max_attempts was raised to nmasters+1 to cover the full ring. General pattern: any outer retry loop that queries a role-agnostic status must force endpoint rotation on a non-authoritative reply — do not rely on the inner failover wrapper, because status-0 preempts it.

• (2026-06-22, ha-term-safety task 3.1) The single primary-append seam is ha_primary_log_append(cluster_t*,type,payload,len,&out_lsn) (ha.c, decl ha.h ~333): packs the raft_entry_blob_t envelope and calls raft_log_append(&c->ha.raft,...) so the LIBRARY stamps s->term + assigns LSN + fires on_persist_entry. CRITICAL: raft_log_append returns -EPERM unless s->role==RAFT_LEADER — and ha_do_promote/the non-raft ha_promote_to_primary road sets ha->role=CTRL_PRIMARY but NOT ha->raft.role (only the raft election road via on_become_leader sets RAFT_LEADER). So ha_primary_log_append MUST have a -EPERM→legacy persist_wal_async fallback (stamping ha->term via persist_set_term first) or init-time/non-raft/engine-less promotion breaks. Retired the hand-rolled bypass by routing persist.c wal_hook_append_classified (RETAIN job/queue/worker hooks) + persist_wal_sync_append (WAL_JOB_SUBMIT WAIT) through ha_primary_log_append GUARDED on p->cluster != NULL — standalone persist (lib-level tests, persist_init(p,&cfg,NULL)) keeps the direct persist_wal_async path. p->cluster is the back-ref (persist.h, set in persist_init); ha.h is reachable in persist.c via xce/cluster_def.h. Co-tracking invariant p->current_lsn == s->log_len + 1 holds because raft_init (2.5) seeds log_len from durable floor and every primary append advances both by 1. To make a controller TEST drive raft_log_append: ha_init_raft then set c->ha.raft.role=RAFT_LEADER + c->ha.raft.term=T directly (the raft-sim convention). on_persist_entry calls persist_wal_async (NOT the hooks) so no recursion. ha_arm_own_term_gate is static → test seam ha_arm_own_term_gate_for_test. Validation: pre-existing test_wal_catchup_* (7, bug-924 overlayfs) fail IDENTICALLY on baseline (git stash + rebuild proves it); every other group 0 fail.

• (2026-06-20, waldump tooling) The xce_test binary (src/controller/meson.build) builds from testsuite/test.c ONLY + libxce_dep — it does NOT link persist.c/job.c (those are in the controller_unit_test target). The investigation doc's claim that xce_test "links persist.c/job.c" was WRONG. BUT capnp read_Job is generated into controller.capnp.c which IS compiled into libxce, so it's reachable transitively. To USE capnp/zlib symbols from test.c you must add c_capnproto_dep + zlib_dep to xce_test_deps (static-lib deps don't auto-propagate through declare_dependency(link_with: libxce)). For a zero-alloc jobid decode read struct Job.jobid directly after read_Job — skip the allocating decode_Job. WAL CRC reader template = persist_test_durable_lsn_on_disk (persist.c ~888): seg files seg-%016llx.wal under <node>/data/wal, 40B wal_entry_hdr_t, crc32 over WAL_HDR_CRC_SIZE(=32) header bytes + payload, prev_crc chain, all-zero hdr = clean EOF (pre-alloc). Offline waldump lives in testsuite/waldump.{c,h} + cmd_waldump in test.c top_commands.

• (2026-06-18, ha-commit-durability task 5.2 VALIDATION) Deterministic E2E on the FIXED build = GREEN, stable 3/3 runs (9/9 probes PASS), in-window primary fdatasync=2 on EVERY probe (vs 0/RED on the unfixed 1.3 build) — the RED→GREEN flip is the deterministic acceptance and is confirmed. Run via nix develop ./src --command bash -c 'bash src/tools/test-commit-durability.sh --deterministic <deploy>' background+poll worktree log; cold election ~15s, 3 probes, exit 0. The ROLLING SIGKILL SOAK is host-verification-needed in-sandbox: rounds 1-2 PASS (jobs survive failover), then round 3 onward "no new primary elected within 40s" and every "DATA LOSS" report is a query against a cluster with NO live primary (same 5 jobs frozen, never re-acked). ROOT CAUSE of the collapse is the durability fix's OWN promotion fence working AS DESIGNED: ctrl-1 won PreVote for term=5, computed safe_committed_lsn=13, then persist_recover_fatal_if_unrecoverable FATAL-terminated ("recovery-truncated log ... no healthy peer reachable ... durable_floor_lsn=13 < hwm=13 ... terminating rather than serving divergent state, Req 2.4") — refusing to let a structurally-truncated node lead and overwrite the good timeline (exactly the bug-661 failure mode the fix prevents). Under aggressive rolling-kill+immediate-restart the fence repeatedly self-terminates recovering nodes → cluster drops below quorum → liveness collapse, NOT durable loss. PROOF it's not loss: committed entries still on disk after the run (ctrl-3 WAL has seg-1 + seg-15; ctrl-2 seg-1) and 0 live controllers at end. So the soak's exit=1 / "DATA LOSS" verdict is a HARNESS ARTIFACT (it equates empty-joblist-with-no-primary to loss) — the soak harness conflates election-stall liveness failure with durability loss. The deterministic flip is the must-have and passed; the live multi-round soak needs a real host (slower election timing, no sandbox reaping) to distinguish fence-driven safety self-termination from a true regression. Did NOT patch production (5.2 is validation-only).

• (2026-06-18, ha-commit-durability task 5.1) End-to-end durable-majority commit verified via 3 integration tests in test_commit_durability.c (NO production wiring gap found — the 2.x/4.x wiring already connects append→flusher fdatasync→durable_lsn→durable-advance wake→majority_candidate_locked self-cap→committed_lsn→commit barrier). (1) test_durable_majority_commit_end_to_end: REAL persist + REAL flusher + commit_engine(n_peers=3) + 2 registered standbys; with persist_test_set_force_fsync_fail(1) a FULL standby-ACK majority does NOT advance committed_lsn (durable precondition gate, Req 1.1/1.4) and a WAIT submit on its own thread does NOT release; after clearing the fault + persist_flush, durable_lsn advances, the durable-advance cb re-derives majority and the WAIT releases rc=0. GOTCHA: persist_wal_async fans out via repl_broadcast to registered standby eps — dummy int eps cause an ASAN global-buffer-overflow in dataq_enq; FIX = install the XCE_TESTING repl_broadcast_send_hook (drop hook, same as test_commit_engine.c) before appending, NULL it after. (2) test_sole_durable_contributor_loss_survives: models the N=3 durable-majority property (committed ⇒ ≥2 durable copies) — write the same op durably to two independent persist dirs (one via persist_wal_async+flush, one via persist_wal_write_replicated which fdatasyncs inline), rm -rf one, prove the survivor still recovers it CLEAN; live multi-host failover proof deferred to 5.2 (Req 1.5/2.5). (3) test_durability_work_off_iomgr_preserves_liveness (Req 1.6): new XCE_TESTING seam persist_test_last_fdatasync_thread() records pthread_self() at the successful flusher fdatasync site (flusher_flush_once, under p->lock); test asserts it != caller thread, and that heartbeat-style atomic reads on the caller thread complete <50ms while flushes are in flight. RED PROVEN by injecting the pre-2.2 bug (cap=highest_appended_lsn): the commit.c:162 assert(cand <= persist_durable_lsn) aborts when the ACK-majority tries to advance committed over a non-durable entry. 4.3's bounded-window catch-up TIMER left as 5.3 follow-up (immediate fail-loud is safety-conservative). Suite Fail:0 ASAN x2; seam nm-excluded from xce_controller. 34 durability tests (was 31).

• (2026-06-18, ha-commit-durability task 4.3) Pre-promotion eligibility gate + fence release. Added TWO functions in ha.c (decls in ha.h): ha_promotion_gate_decision(bool truncated, uint64_t durable_floor, uint64_t safe_hwm, bool peer_reachable, bool catch_up_succeeded) -> recover_decision_t (PURE — builds recover_policy_inputs_t with durable_below_hwm = durable_floor < safe_hwm and delegates to persist_recover_decision, so the 3.2/3.3 decision table is the SINGLE source of truth, DRY) and ha_evaluate_promotion_fence(ha, ...) (LIVE — ELIGIBLE clears ha->in_catch_up, FENCE sets it, FATAL routes through persist_recover_fatal_if_unrecoverable). WIRED into raft_cb_become_leader (ha.c ~703) AFTER persist_truncate_wal_above(safe_lsn) (so persist_durable_floor reflects the 2.2 clamp) and BEFORE the CTRL_PRIMARY store; on != ELIGIBLE it logs WARN and returns WITHOUT becoming primary (so a test-hooked FATAL never falls through to leadership). safe_lsn lifted to fn-scope var promotion_safe_lsn (the cluster HWM from voter replies — ha_compute_safe_committed_lsn arithmetic UNTOUCHED). peer_reachable derived from surviving granted vote eps (ectx->votes[i].vote_granted && .ep != NULL). GATE SKIPPED when c==NULL (the raft_cb_become_leader_for_test shim has no persist) — that is WHY existing election/role/convergence suites (all driven via the c==NULL shim or ha_promote_to_primary, a SEPARATE function with no gate) pass unchanged with ZERO fixture edits. Turned 3.2 rows 1/2/3/5/6 GREEN (0 Ignored) by driving the live gate (rows include a fence-set then catch-up→release sequence proving Req 2.6); added test_promotion_gate_hollow_refused_at_minimal_quorum (Req 2.5). test_commit_durability.c needed #include "ha.h" (ha.h forward-declares cluster_t, self-contained). LIVE-TIMER FATAL DEFERRED to 5.1 (allowed): the gate evaluates catch_up_succeeded=false at the synchronous promotion instant, so a truncated/behind node with NO peer FATALs immediately (correct terminal) — the bounded-window TIMER that gives DEFAULT_RECOVERY_CATCHUP_WINDOW_MS to find a peer before FATAL is the live-timer aspect 5.1 owns; the decision + fence + release are all live HERE. Suite: 31/0 durability (0 Ignored), full controller direct-binary = only the pre-existing barrier baseline fail (overlay-/tmp EIO), meson controller test OK Fail:0.

• (2026-06-18, ha-commit-durability task 3.3) Boot-policy FATAL on bounded-unrecoverable recovery. SPLIT into a PURE decision + a SIDE-EFFECTING terminate so the decision is unit-testable without aborting: recover_decision_t {ELIGIBLE=0,FENCE=1,FATAL=2} + recover_policy_inputs_t {verdict, durable_below_hwm, healthy_peer_reachable, catch_up_succeeded, durable_floor_lsn, cluster_hwm_lsn} in persist.h; persist_recover_decision(const inputs*) (pure, no side effects) + persist_recover_fatal_if_unrecoverable(const inputs*) (logs truncation boundary at ERROR then terminates) in persist.c. FATAL ONLY when (truncated OR below-HWM) AND no healthy peer AND !catch_up_succeeded (rows 4,4b). A reachable healthy peer is NEVER FATAL (rows 3,5 → FENCE) — this is the invariant 3.3 owns; the full FENCE→ELIGIBLE eligibility is 4.3. TERMINATE SEAM: added persist_fatal_hook (extern void()(const char), declared in persist.h, DEFINED =NULL in persist.c — NOT under XCE_TESTING so production links it as a NULL global → production path is tlog_exit(); abort()). Mirrors the existing ha_fatal_hook convention (ha_convergence.c:58 / ha.h:1045). Tests install the hook to capture FATAL without dying (test_commit_durability.c dur_fatal_hook + g_fatal_hook_calls counter). Bounded window = DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000 (persist.h) — FINITE, never indefinite. Production nm-verified: persist_recover_decision/persist_recover_fatal_if_unrecoverable are real T symbols, XCE_TESTING seams excluded. Rows 4/4b + a focused peer-never-fatal test PASS; rows 1/2/3/5/6 stay TEST_IGNORE for 4.3 (suite GREEN). tlog_exit() is reachable in persist.c transitively (no extra include needed — ERROR/WARN already pull tlog via xcelog.h chain).

• (2026-06-18, ha-commit-durability task 3.1) Recovery-integrity verdict: recover_verdict_t {RECOVER_CLEAN=0, RECOVER_TRUNCATED=1} in persist.h (enum declared BEFORE struct persist_s since the struct field uses it); recovery_verdict field on persist_t (NOT ha_state_t — keeps the boundary in persistence, avoids pulling in ha.c); replay_wal gained a recover_verdict_t* verdict_out out-param (single caller: persist_recover at persist.c). Classification: clean stops = hn==0 EOF, all-zero pre-alloc sentinel, end-of-segments → CLEAN; structural corruption = CRC chain break (persist.c ~1463/1471), CRC mismatch (~1500), short payload read (~1490), AND 0<hn<sizeof(hdr) partial-header / hn<0 read-error → TRUNCATED. Verdict init CLEAN once at fn entry, only ever SET to TRUNCATED (a later clean segment never resets it). Snapshot-sentinel + clean replay is CLEAN (load_snapshot sets checkpoint_lsn from the 16-byte sentinel; replay only adds entries ABOVE it, no sub-sentinel entries = normal). 3.1 ONLY classifies + sets the flag (accessors persist_recovery_verdict / persist_recovery_truncated) — does NOT FATAL (task 3.3) and does NOT fence promotion (task 4.3 consumes it). Verdict is about LOCAL log structure, NOT being behind the cluster (a clean log ending below high-water is CLEAN; freshness/promotion fence catches "behind"). Test crafts corruption by append+flush then pwrite-flipping the on-disk first entry's crc field; WAL seg path is <tmpdir>/<name>/data/wal/seg-%016llx.wal (name="ctrl-test" in make_durability_persist — first entry is LSN 1 ⇒ seg-...0001.wal, LSN 0 reserved).

• (2026-06-18, controller test ordering) PRE-EXISTING FAILURE (NOT a regression): test_fact_hook_append_is_retained_and_nonblocking (test_commit_barrier_rollout.c:446 "a covering ACK must advance committed_lsn over the retained fact") FAILS even on a clean baseline (git-stash-confirmed) and FAILS in isolation (controller_unit_test -t test_commit_barrier_rollout) — a timing-sensitive commit-barrier test, unrelated to recovery/persist changes. Also: leftover /tmp/xce_* dirs cause a cascade (errno=5 EIO fdatasync noise + extra spurious FAILs like test_demotion_replay_yields_committed_only); rm -rf /tmp/xce_* before each suite run clears it and the run is stable at exactly that one pre-existing fail. When validating a persistence task, confirm a failing barrier test is the baseline one before treating it as your regression.

• (2026-06-18, ha-commit-durability task 1.1) Controller CLI flags follow a fixed 6-point argp pattern in src/controller/main.c: (1) #define OPT_* key above ASCII, (2) field in struct arguments, (3) entry in options[], (4) case in parse_opt (validate, e.g. atoi<0 → argp_error), (5) default-zero init in main() (0 = "not passed"), (6) assign into myconf (struct in src/include/myconf.h, NOT src/controller/). persist_config_t overrides live in src/include/persist.h as static inline helpers next to persist_config_defaults(); there are TWO persist_config_defaults() callers to keep in sync — cluster.c phase2 (~L1267) and myapp.c step 6 (~L164). GOTCHA: capnp generates a same-named PersistConfig_set_walSyncMs (wire serialization) — the compiler's "did you mean" suggests it; it is NOT the runtime override helper. Test home: testsuite/test_cluster_persist.c (already covers persist_config_defaults + wal_sync_ms).

• (2026-06-17, bug-660 slow-election convergence) ha->last_activity is DUAL-PURPOSE and that conflation is a trap: it's the watchdog grace clock (reset to now on every ha_arm_heartbeat_timeout_vtask, incl. ha_prevote_rearm_liveness after each lost PreVote) AND the source for ha_heartbeat_is_fresh (the reason=fresh_heartbeat PreVote/vote deny). So a node with NO live primary keeps refreshing its own "fresh heartbeat" every failover cycle and denies every legitimate higher-term candidate → elections take 11+ rounds / minutes to converge (a standby denied for 2.5 min after its primary was killed). FIX: the eligibility gate must also require a LIVE primary connection. raft_cb_vote_eligible (ha.c:311) checked is_fresh ALONE; handle_request_vote (ha.c:2205) already required primary_ctrl_fd >= 0 && is_fresh. Adding ha->primary_ctrl_fd >= 0 && to ha.c:311 (+ cluster.c:526 reason logging) makes a dead/absent primary (fd=-1 on init / RDHUP / watchdog-fire; real fd only set at config_repl.c:1451 on first ctrlhello) GRANT → converge in ~1 round. The existing fresh-deny tests already set primary_ctrl_fd=999 and documented "the deny gate requires a live connection AND a fresh heartbeat" — production just didn't enforce it. Diagnostic signature: ag "PreVote DENIED" logs/ shows reason=fresh_heartbeat long after the primary's last log line. Test: test_pre_vote_dead_primary_grants_despite_fresh_activity (controller_unit_test -t test_ha_vote_gate).

• (2026-06-15, ha-rejoin bug-340) A killed+restarted controller loops elections forever because the PRIMARY never evicts the dead/wedged outbound standby endpoint. primary_heartbeat_fn (ha.c) has two steps: (1) NOP each ep in ha->standby_endpoints, (2) reconnect to any peer NOT present in standby_endpoints. A wedged ep (peer died → handle_error did EPOLL_CTL_DEL but left it in the array; epoll MOD then returns ENOENT → the WARN endpoint_update_epoll rc1=-1 spam) makes step 2 think the peer is still connected → reconnect SUPPRESSED → restarted node never gets a ctrlhello → re-elects every heartbeat_timeout (term climbs unbounded). xce_nop_req returns >=0 on enqueue even when the epoll re-arm fails, so the heartbeat layer can't see the wedge. FIX: evict in step 1 using endpoint_t.ts_last_in (existing wall-clock field, stamped by handle_input) — if ts_last_in>0 && now-ts_last_in>heartbeat_timeout, zarray_del from standby_endpoints + repl_remove_standby; reconnect then re-establishes the channel and ctrl-1 settles via ha_discover_primary (NO term gate — accepts any primary msg while ELECTING regardless of its own high term). Gotchas: (a) ha_arm_heartbeat_timeout_vtask ALREADY resets last_activity=now, so 'become_follower doesn't reset liveness' is NOT a bug — the ~timeout loop period proves the watchdog is correct; the sole cause is the broken channel. (b) standby_endpoints holds TWO ep types (promoted vote_ep_ctx_t with ctx!=NULL, and heartbeat standby_ep_ctx_t with ctx==NULL) sharing first 4 fields; don't free vote-ep data (UAF — live vctx pointers). (c) Run one suite: controller_unit_test -t test_ha_liveness, ASAN_OPTIONS=detect_leaks=0.

• (2026-06-17, bug-646) The WARN endpoint_update_epoll rc1=-1 spam (same one called out in the bug-340 learning above) was a SILENT-SWALLOW: endpoint_update_epoll (netio.c) logged the iomgr_update_poll failure but returned 0, leaving the poll interest unarmed. iomgr_update_poll uses EPOLL_CTL_MOD → ENOENT when the fd isn't registered (e.g. after a failed reconnect, or a wedged ep that handle_error EPOLL_CTL_DEL'd but left in the array). Consequence on the CLIENT: a jobsub after failover hangs forever in cbuf_get (xceapi.c, no timeout) while io_worker sleeps in epoll_wait — because the request bytes were never flushed (EPOLLOUT never armed) and no socket-drop fired handle_disconnect. FIXED: endpoint_update_epoll now returns -errno; endpoint_enq runs handle_error(ep) on arm-failure (wakes the waiter via handle_disconnect, failover rotates); handle_output propagates too. Non-obvious facts that ruled out other causes: (1) server commit barrier is bounded — DEFAULT_SYNC_TIMEOUT_MS=500 (config.h:93); (2) ha_dispatch_gate (ha.c:1628, called at cluster.c:251) ALREADY replies E_XCE_STANDBY / -XCE_ERR_NOT_READY for writes to a non-primary / quorum-less-primary — so "writes aren't role-gated" is NOT a gap; (3) client connection-break IS handled (handle_disconnect injects -ECONNRESET into wbuf). Test: test_netio_arm_failure.c.

• (2026-06-17, bug-648 + VERIFIED SSL close semantics) OpenSSL 3.x peer-close behavior, confirmed against official docs (SSL_get_error/SSL_read man pages) AND a live probe on this build (OpenSSL 3.6.0): a peer close makes SSL_read() return 0 (never <0 for a clean FIN, never WANT_READ) and the meaning is carried by SSL_get_error(), NOT the return value or errno. Mapping: clean close_notify → SSL_ERROR_ZERO_RETURN (errno 0); unexpected EOF (FIN, no close_notify) → SSL_ERROR_SSL since OpenSSL 3.0 (pre-3.0 it was SSL_ERROR_SYSCALL + errno 0 — documented in the SSL_get_error NOTES); RST → SSL_ERROR_SYSCALL errno=ECONNRESET(104). errno is authoritative ONLY for SSL_ERROR_SYSCALL; for SSL_ERROR_SSL the error QUEUE is authoritative and any errno is incidental — we measured EPIPE(32), which is the leaked side effect of OpenSSL's internal alert-WRITE (raw write() on a broken conn → EPIPE; raw read() → EOF/0 or ECONNRESET, never EPIPE). Consequence: the old rc = -errno in io_fsm_do swallowed every documented EOF case (errno 0 → 0). FIX: io_classify_io_error() (pure, non-static, unit-tested in test_netio_close_propagation.c) returns 0 only for WANT_READ/WANT_WRITE + EAGAIN syscall, else negative ((ecode!=0)?-ecode:-ECONNRESET). DESIGN CONCLUSION (user, reaffirmed feedback-fail-loud-over-timeout): no client reply-wait timeout — close detection is made deterministic at the source instead. CAVEAT: on THIS 3.6 build a process-death FIN gave SSL_ERROR_SSL/errno=32 which the OLD code already propagated, so bug-648 is likely NOT the sole cause of the controller-kill permanent hang — that still implicates the reconnect/SSL-handshake-stall path (client TCP-connected, state RUN, handshake never completes → request queued, never flushed, io_worker idle in epoll_wait). Verify live before closing that out.

• (2026-06-15, ha-election bug-339) Candidate-advertised RequestVote LSN MUST equal the voter-side freshness floor. Voter floor = ha_vote_self_freshness(ha, config_tag, applied_lsn) = max(applied_lsn, last_known_committed_lsn) (cluster.c OPCode_requestVote). On a STANDBY, persist.committed_lsn NEVER advances (only the primary commit engine writes it at persist.c:1192); standbys advance applied_lsn (ha_apply_wal_catchup_entry) + last_known_committed_lsn (heartbeat replies). So a candidate that advertises persist.committed_lsn under-advertises vs its own floor → after a primary dies, every surviving standby advertises stale-low LSN while each peer's floor is high → freshness gate (cand_lsn<my_lsn) denies on BOTH sides → no quorum → infinite RDHUP→ELECTING→STANDBY loop, terms climb in lockstep, nobody ever become_leader. Boot path (myapp.c:217) already uses applied_lsn; only the RDHUP failover path (ha_on_primary_ctrl_fd_rdhup) was inconsistent. Fix: compute cand_tag/cand_lsn via ha_vote_self_freshness once and feed BOTH the elect_ctx (votes[0]+self_*) AND the ha_rdhup_election_hook args. Symmetric advertised==floor guarantees the freshest node is granted by all peers. Diagnostic signature in logs: both nodes only raft_cb_become_follower (never become_leader) + RDHUP every ~heartbeat_timeout.

• (2026-06-14, election-safety) ha_on_primary_ctrl_fd_rdhup MUST mirror the ha_start_election_async no-config voter-only guard: a config-less node (myconf->config_version_tag==0) with candidates_count>1 must NOT start an RDHUP re-election (it would tie-grant with another tag=0 node — freshness gate only denies a LOWER tag — and elect a config-less primary that pushes an empty config). Guard sits AFTER the WAL-truncate block, BEFORE elect_ctx alloc/hook/raft_on_leader_lost; sets role=CTRL_STANDBY and returns. RDHUP integration/failover tests that assert election-after-RDHUP use make_cluster/memset which leaves config_version_tag=0 + candidates_count=2 — they were incidentally testing the buggy condition; give them a nonzero tag (0x1234) so they exercise the legitimate CONFIGURED-node path. RDHUP handlers called with c==NULL early-return before the guard (unit-mode tests assert only the CTRL_ELECTING transition). CTRL_PRIMARY enum==0, CTRL_STANDBY==1, CTRL_ELECTING==3 (test printf "role=0==CTRL_PRIMARY" is expected). Full controller suite: meson driver SIGTERMs at 300s — run ./xce.build/controller/controller_unit_test directly with ASAN_OPTIONS=detect_leaks=0.

• (2026-06-13, barrier-rollout 5.2) push_config_to_standbys now RETURNS the configAck barrier result (0/-ETIMEDOUT/-EINVAL), push count via acked_out; the configAck branch of repl_wait_majority_ack reads entry_lsn as the push tag and r->ce->n_peers (set-once, lock-free) as the denominator, ce==NULL -> -EINVAL fail-loud. ha.c/join.c streaming pushes discard the return BY DESIGN (log-only); only do_upgrade_config_with_bytes treats it as authoritative.

• [2026-06-12] Census parsing caveat: controller_unit_test logs (tinylog) and Unity result lines BOTH go to stdout and can splice mid-line under the multi-threaded config-push tests — a distinct-Unity-line count silently drops spliced results. Robust census = union of Unity result names AND the test_x: PASS printf lines, cross-checked against the per-group N Tests M Failures summary blocks (which are splice-immune).

• [2026-06-12] Config-barrier test trick: when a test needs a covering config ACK for a tag only known AFTER conf_save_config hashes the re-encoded blob (upgradeConfig path), pre-ack with acked_config_lsn = UINT64_MAX — covers any push tag under both legacy (>0) and per-push (>= tag) counting.

• [2026-06-12] A zeroed stack cbuf_t DEADLOCKS on cbuf_put (ncap==0 → len==ncap → cond_wait forever); any fixture driving do_upgrade_config_with_bytes' restart path must cbuf_init(&c->wbuf, …) first.

• [2026-06-13, single-leader-role 1.1] A WAL type with value 31..99 (below WAL_ENTRY_TYPE_MAX=100) is observationally IDENTICAL to a generic entry in the current tree for append (persist_wal_async writes any type), apply_wal_entry (common-path LSN advance via default: arm), and replay_wal (its OWN type filter only skips type > WAL_CHECKPOINT(100)). So a behavioral RED test for "WAL_NOOP append/apply works" PASSES even before WAL_NOOP exists. The ONLY genuinely-broken behavior pre-impl is wal_classify (totality assert(0) aborts on the unclassified value). RED carrier for append/apply cases = couple them to wal_classify(WAL_NOOP)→RETAIN via the fork helper (the enum value + classify row land in the SAME task 1.2 change per design).

• [2026-06-13] persist_t starts current_lsn==0 (calloc-zeroed; persist_init does NOT reset it). The FIRST persist_wal_async gets lsn==0; in replay_wal the hdr.lsn <= from_lsn (0<=0) skip means current_lsn never advances for a lone first entry. Replay fixtures asserting an LSN advance must write a preceding entry (or otherwise land the entry-under-test at lsn>0). replay_wal (persist.c:783) is SEPARATE from apply_wal_entry and has its own zero-region detector (lsn==0 && type==0 && payload_len==0 → break) and type filter.

• Census EXACT method (2026-06-12): count DISTINCT Unity file:line:test:PASS triples (grep -aoE + sort -u), not raw 2mPASS occurrences — tinylog interleaving garbles ~9 verdict lines per run and duplicates others; distinct-triple count matched the 494-baseline arithmetic exactly (495 after task 4.3's +1).

• repl registry reconnect semantics: repl_remove_standby vacates the slot (ep=NULL, keeps last_disconnected_lsn guard); repl_add_standby reuses the first empty slot and memsets it (acked_lsn=0). With both standbys removed, majority_candidate_locked zero-fills to n_peers-1 → candidate 0 → committed_lsn frozen (RETAIN entries safely pending); after re-add, ONE covering ack = 2-of-3 majority.

• repl_wait_majority_ack walAck track is engine-only since barrier-rollout 2.2: entry_lsn==0 returns 0 immediately (documented once-per-lifetime LSN-0 first entry), everything else delegates to commit_submit unconditionally — ce==NULL surfaces as -EINVAL from commit_submit (unreachable in production; cluster_init_phase2 always wires the engine). The counting loop that remains in the function is configAck-ONLY (no self-count, n_active==0 shortcut is Req 8.5 config behavior). persist_wal_sync_append's residual non-engine branch can only store committed_lsn for the LSN-0 no-op case. (2026-06-12, task 2.2)

• commit_submit(COMMIT_MODE_RETAIN) is fully lock-free: it returns 0 BEFORE pthread_mutex_lock(&ce->lock) (commit.c:233-239), so it is safe to call from hook functions even while entity write locks are held — only WAIT mode takes ce->lock. (barrier-rollout task 1.2, 2026-06-12)

• ha-commit-engine: once repl.c references commit_* symbols (task 5.2), commit.c MUST be in controller_src (meson.build) or the production xce_controller link fails — the "linkage lands with 5.3" note in tasks.md only held while repl.c had no engine references. Lock order ce->lock → r->lock: repl.c calls engine hooks only AFTER releasing r->lock.

• Use clang-format for C/C++ formatting; shared config lives at xce/.clang-format.

• Logging macros in utils.h: TRACE (= tlog_debug3, verbose), INFO, WARN. There is NO DEBUG3 macro — use TRACE for debug-level logging in repl.c and other controller files. (2026-06-10)

• msg_handler_t in netio.h is int (*)(void*) — NOT int (*)(endpoint_t*). Functions assigned to ep->msg_handler must use void *ep parameter. (Task 2.2, 2026-05-30)

• xce_request() is defined in request.c but not declared in request.h; files that need it must add extern int xce_request(endpoint_t *ep, xce_request_t *req); locally (pattern from config_repl.c).

• Test seam pattern for ha.c XCE_TESTING: declare int (*hook)(ep, req) = NULL under #ifdef XCE_TESTING, expose as extern in test file, test sets it before calling function under test. Mirrors xce_retry_send_hook in request.c.

• capnp UAF fix pattern (Task 2.3): INVOKE_MSG_HANDLER fires ep->msg_handler(ep) directly from iomgr thread; packet is in ep->fsm_in.pkt. Do raw capn_init_mem decode, read_RequestVoteReply into local struct BEFORE capn_free, then capn_free + free_packet + free(pkt). Use local scalars (not capnp pointers) after free. (2026-05-30)

• capnp serialization for tests: use new_RequestVoteReply(seg) / new_RequestReply(seg) / new_Request(seg) — NOT raw capn_new_struct. Follow pattern from xce_encode / xce_replconfigpush_req_raw in request.c.

• To avoid C99 "goto jumps over declaration" error: declare all local vars at top of function (or in a sub-block), not after the first goto target label.

• hook seam forward-declaration order: declare the hook variable BEFORE the function that references it (even within the same XCE_TESTING block). Use a dispatch helper static function to decouple seam from production path.

• term fencing must live in the handler, not the dispatch path (2026-05-31): vote_reply_msg_handler delegates to vote_reply_dispatch which calls either xce_vote_reply_hook (XCE_TESTING) or raft_on_vote_reply (production). If term fencing only happens inside raft_cb_become_follower (production), it is silently skipped in the test hook path. Pattern: do explicit CAS loop (atomic_compare_exchange_weak) on ha->term before calling vote_reply_dispatch, so the update applies regardless of which dispatch path runs.

• ha_start_election_async ownership rule (Task 2.4, 2026-05-30): in the production path (c != NULL), ha_elect_ctx_t is owned by election_timeout_fn which frees it on fire. In the test-mode path (c == NULL, peers == NULL), no delaymgr task owns ctx — must free ctx inline after ha_elect_evaluate_and_signal fires. ASAN will catch the leak if forgotten.

• ha_start_election_async single-node shortcircuit (Task 2.4): n_peers==1 means n_other=0 — allocate ha_elect_ctx_t with 0 flexible entries (no votes[] slots needed), set votes_granted=1, call evaluate directly, then free ctx immediately. No iomgr or delaymgr involvement for single-node.

• election_timeout_fn must lock ctx->lock before calling ha_elect_evaluate_and_signal (Task 2.4): the evaluate function has a done_posted guard but its callers in the reply path hold ctx->lock; timeout fn must also hold the lock to avoid races with concurrent reply processing.

• single-node shortcircuit enters CTRL_PRIMARY_COMMITTING, NOT CTRL_PRIMARY (Task 2.4 review fix): task 4.4 committing timeout promotes to PRIMARY for degraded/single-node mode (Req 8.4). Tests must assert CTRL_PRIMARY_COMMITTING.

• ha_elect_evaluate_and_signal out-of-bounds guard (Task 2.4 review fix): ctx->votes[] is a C99 flexible array with n_peers entries. When n_peers==0 (single-node), votes[0] is out-of-bounds. Must check n_peers==0 FIRST and short-circuit to PRIMARY (quorum=1, self-vote=1 always wins). Guard added before any votes[0] access.

• client_ssl_ctx for outgoing vote connections (Task 2.4 review fix): endpoint_connect() creates SSL from iomgr->ssl_ctx (server context). After connect returns, replace ep->ssl with SSL_new(client_ssl_ctx), SSL_set_fd to the same fd, SSL_set_connect_state. If SSL_new fails, fall through (handshake fails → vote denied via err_handler).

• vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.

• timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.

• repl_add_standby idempotency (2026-06-10): scan all MAX_STANDBYS slots for ep match BEFORE the nstandbys guard and empty-slot search. Return the existing slot index immediately on match (no state change). This prevents double-registration when raft_cb_become_leader promotes vote eps and on_standby_connected fires for a reconnecting late-joiner. Covered by test_repl_add_standby_idempotent.

• Heartbeat freshness source of truth (2026-06-09): ha_heartbeat_is_fresh derives from ha->last_activity (whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The old last_heartbeat timespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate is role==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha); RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh.

• raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so raft_init must set voted_for_term = initial_term when initial_voted_for != -1. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6.

• repl_add_standby returns the SLOT INDEX (>=0) on success, not 0 (2026-06-11, ha-commit-engine task 1.1): asserting == 0 breaks on the second standby. Assert >= 0 or the expected index.

• Unity harness quirks in controller_unit_test (2026-06-11): (a) failure lines misattribute the FILE — Unity.TestFile comes from whichever group's UNITY_BEGIN ran, so a failure printed as test_repl_unit.c:294 may live in test_p2_integration.c; locate the actual test by grepping for the test NAME. (b) Each group's UNITY_BEGIN resets counters, so the process exit code only reflects the LAST group — failures in earlier groups are masked (exit=0); meson reports green despite real failures. Grep output for FAIL with grep FAIL on the raw log (ANSI codes sit between ':' and 'FAIL', so grep ":FAIL" misses them).

• 3 pre-existing controller suite failures on persistent_ha baseline (2026-06-11, verified by stash/rebuild without ha-commit-engine changes, see bug-411): test_committed_lsn_does_not_advance_on_timeout (rc -110 vs expected 0 — persist_wal_sync_append now returns -ETIMEDOUT, test expects old rc==0+sync_rc contract), test_repl_ack_after_promote_gate_allows_jobsub and test_rdhup_primary_dispatch_gate_opens (both -10013: ha_has_write_quorum false, standby_endpoints empty with n_peers=2).

• ha-commit-engine: task 1.2 shipped commit_on_membership_change as a FULL re-evaluation (commit_advance_locked + cond broadcast), and majority_candidate_locked already excludes in_catch_up + ep==NULL slots. Task 3.1 membership tests are green-on-arrival; task 3.2 is a no-op (verified 2026-06-11, tests 17-20 in test_commit_engine.c).

• key_find (xkvmgr_find) returns 1 when found, NOT 0 — queue_ctrl/worker_ctrl/job_ctrl leak this into rc: if (rc == 0) hook gates never fire on the found path (bug-447); any code reusing key_find's rc for success checks must normalise it.

• Controller test binary reply observation: no xce_reply hook exists in src/lib; observe replies on a zeroed no-iomgr endpoint_t — xce_reply endpoint_enq()s onto ep->outq; drain with endpoint_deq + raw capnp read_Request/read_RequestReply (test_commit_barrier_reply.c).

• makeJob needs job_obj_init() once per process AND xce_job_t.submit != NULL; makeString needs xstring_table_init (test_worker_join.c inits it unguarded at group entry — probe makeString for NULL before initialising to stay -t-selection safe).

• (2026-06-13, barrier-rollout 6.1) mark_job_start/finish/fail/update are STATIC in job.c — testsuite reaches them ONLY via the non-static wrappers on_job_start/on_job_finish/on_job_fail (and on_job_postdone → mark_job_update). Those wrappers return 0 unconditionally regardless of mark_job_* rc, so the guard's "success-no-op rc" observable is asserted on the wrapper rc (== 0). No public caller produces a terminal→non-terminal UPDATE (on_job_postdone OR's PDONE = forward; job_ctrl SIGCONT only clears suspend bits) — mark_job_start (state := JOB_STAT_RUN) is the faithful publicly-reachable terminal-clearing witness for the shared backward rule. current_lsn is the "did a hook fire / WAL append" oracle: a guarded-out no-op runs no hooks → no persist_wal_async → current_lsn frozen; combine with cbrp_count_jobstate_entries (on-disk scan) + a key_find live-state read. Controller suite baseline 2026-06-13: 502 distinct PASS triples pre-6.1; +3 with 6.1's 2 preservation pins landing green (forward-update, PERR-merge) = 505, 3 FAIL = the 3 new RED guard tests.

• (2026-06-13, single-leader-role 2.2) The own-term promotion noop uses persist_wal_async(&c->persist, WAL_NOOP, NULL, 0) which ALWAYS calls repl_broadcast → xce_wal_batch_req_raw(s->ep) for every registered, non-catch-up standby. Any test fixture that registers fake int endpoints (the g_*_eps[] pattern) via repl_add_standby MUST install repl_broadcast_send_hook (drop hook) before driving a real append, or ASAN aborts with global-buffer-overflow reading ep->peer (offset 36 past the 4-byte int). Precedent: test_commit_engine.c ce_drop_send_hook, test_p2_integration.c p2_drop_send_hook. The task-2.1 fixture lifecycle_cluster_setup (test_ha_role_lifecycle.c) OMITS this hook AND off-by-ones the current_lsn prime (sets appended_lsn+1; persist_wal_async takes current_lsn then ++, so to land the noop at appended_lsn the prime must be appended_lsn). Both are fixture bugs (bug-477/478), out of ha.c boundary. Production noop_lsn = current_lsn-1 read under p->lock AFTER the append is correct.

• (2026-06-13, single-leader-role 2.2) raft_cb_become_leader does NOT call ha_do_promote — the raft FSM already synced ha->term at entry (~ha.c:463 atomic_store of the raft-reported term); calling ha_do_promote would double-increment (research §4 finding 2). The CTRL_PRIMARY store sits OUTSIDE the if(c) block so the c==NULL test shim reaches it. The non-raft road (ha_promote_to_primary) DOES call ha_do_promote (it owns its term). Both roads share ha_register_persist_hooks (BEFORE noop append) + ha_arm_own_term_gate (noop + commit_set_writable_lsn).

• (2026-06-13, single-leader-role COMPLETE 11/11, validate-impl GO) CTRL_PRIMARY_COMMITTING REMOVED — ctrl_role_t now 4 values (CTRL_PRIMARY=0, CTRL_STANDBY=1, CTRL_INIT=2, CTRL_ELECTING=3). Deleted without replacement: committing_timeout_fn (60s watchdog), ha_check_majority_convergence, ha_repl_ack_received/repl_ack_count, committing_timeout_ms/HA_DEFAULT_COMMITTING_TIMEOUT_MS. OPCode_replCatchUpDone is now a gate-allowed INFO no-op. Dispatch-gate PRIMARY branch + barrier-rollout fact allowlist preserved byte-for-byte; commit.c consumed with ZERO diff. Repo-wide completion grep (CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence, excl .git/.kiro/specs) = ZERO.

• (2026-06-13) OPERATIONAL: controller meson test wrapper ROUTINELY hits its 300s timeout under load (barrier commit-deadline waits) — Timeout: 1 is NOT a failure. Authoritative green = direct binary src/xce.build/controller/controller_unit_test (EXIT 0, every group "N Tests 0 Failures"). A pre-existing ~153KB suite-wide LSan leak is SUPPRESSED by ASAN_OPTIONS=detect_leaks=0 at controller/meson.build:163 — LSan exit=1 only appears on a RAW direct-binary run without that env; baseline, reproduces at pre-spec commits, not a per-change regression.

• (2026-06-14, reserve-lsn-0 / bug-543 FIX) LSN 0 is now RESERVED as the none/empty sentinel — persist_init() sets current_lsn = 1 on a fresh store (persist.c, the ONLY fresh-init site; recovery overwrites from snapshot/WAL tip). First-ever WAL entry (promotion NOOP) is lsn 1. This was a ONE-LINE production change + a comment block; everything downstream ALREADY treated 0 as "none/empty" and lsn>=1 as real, so NO change was needed in commit.c (highest_appended_lsn current_lsn>0?-1:0, commit_submit rejects lsn 0, wait_done writable_lsn==0=gate-off), the streamer (hdr.lsn>standby_lsn: 1>0=true now streams the first entry → no hole), ha.c own-term gate (noop_lsn=current_lsn-1=1, writable_lsn=1 effective), or the wire/capnp format. The pre-alloc zero-region sentinel hdr.lsn==0 && type==0 && payload_len==0 (persist.c:874, config_repl.c:1580, truncate:1666) is STRENGTHENED — no real entry is ever lsn 0. Segment naming follows current_lsn at open time, so the first segment is seg-...001.wal (not 000); recovery/truncate scan by sscanf, no hard-coded seg-000 assumption.

• (2026-06-14, reserve-lsn-0 TEST FALLOUT) Base-shift broke tests that did a "priming append so the entry lands at LSN >= 1" — that priming was ONLY to escape the old lsn-0 special-case and is now redundant. Fix = REMOVE the priming append so the real entry lands at lsn 1 (test_commit_engine tests 32/34, test_commit_barrier_rollout fact-hook, test_commit_barrier_reply cbrp_setup, test_p2_integration single_node) OR bump a hard-coded current_lsn==1 assertion to ==2 if the priming must stay (test_egress_filter ef_setup). The ENGINE-LESS walAck path only "worked" via repl_wait_majority_ack's entry_lsn==0 → return 0 hatch (repl.c:437); with the first entry now lsn 1 it routes to commit_submit(ce=NULL) → -EINVAL, so engine-less commit tests MUST wire a commit engine (test_p2_integration test 9: added commit_engine_init(n_peers=3) + repl_set_commit_engine + commit_engine_fini). Tests asserting RELATIVE lsn (lsn_before+1, current_lsn>0, h1.lsn>h0.lsn) or byte-offset truncation math are base-shift-immune.

• (2026-06-13, kiro-impl process) A RED-only test pin is only valid if its OWN impl task can turn it green via the SAME seam the test drives. Misfires caught in review: boot-replay pin RED on an lsn-0 replay_wal skip artifact; hooks pin via the c==NULL raft shim where ha_register_persist_hooks(&c->hooksys) can never run (re-point to the real-cluster ha_promote_to_primary seam). Deleting machinery → DELETE the machinery-subject tests (design "no shims, rewritten or deleted not stubbed"), never add file-local no-op stubs. Unity runner prints the PREVIOUS passing test's file:line as a stale prefix on FAIL lines → attribute failures by test NAME.

Do-Not-Repeat

• (2026-06-21, raft-sim tasks 9.1-9.4) In raft_sim.c, leader per-peer next_index[peer] is initialized to Len(log)+1 at become_leader and ONLY advances via raft_on_append_entries_reply success (= matchIndex+1). It does NOT change when a send to a partitioned peer is dropped (no reply comes back). So to exercise nextIndex BACK-OFF in a sim test you cannot just partition+drop — you must create a real (prev_log_index) mismatch: replicate so next_index runs ahead, then shrink the follower's actual log (call c.nodes[k].cbs.on_truncate_from(&c.nodes[k], idx)) so the leader's stale next_index points past the follower's true tail; the next AE is then rejected and walks back. To bump a sim leader's term for "higher-term leader" tests, set c.nodes[0].state.term = N directly AFTER a clean single election (raft_log_append stamps entries from s->term; next_index/match_index are term-independent) — do NOT chain raft_on_election_timeout+raft_start_election on a node that is already LEADER (election_timeout is CANDIDATE→FOLLOWER, a no-op on a leader, so the term never climbs). For a duplicate-AE idempotency assertion, rewind next_index[peer] to re-send an ALREADY-PRESENT entry (already-have branch = no log growth); re-sending at the live next_index sends the NEXT entry, which is not a duplicate.

• (2026-06-21, bug-850) Do NOT fix "a rejoining standby races live replication ahead of catch-up" by defaulting repl_add_standby to in_catch_up=1. It has a huge blast radius: a fresh standby then does NOT count toward repl_active_standby_count write quorum until it converges, so EVERY commit-WAIT path stalls — the full controller suite TIMED OUT at 300s (commit.c "WAIT deadline expired ... committed_lsn=0"), because unit tests register standbys + commit directly without running the heartbeat dispatch that would clear the flag. The correct, surgical fix is a CONTIGUITY GUARD at the persist layer (Raft log-matching): persist_wal_write_replicated rejects a forward gap (lsn > current_lsn) with -EAGAIN (no write, no current_lsn jump); the apply path treats -EAGAIN as a quiet skip (no ack, no ERROR spam) so the ACK-driven catch-up continuation re-streams from current_lsn and the log fills in order. Zero quorum-semantics change → zero suite regression (129/0/0). Relies on bug-848 (advertise durable_floor == current_lsn-1) so catch-up always starts AT current_lsn and is never spuriously rejected.

• (2026-06-18) When a task changes commit semantics to be ASYNC-durable (self caps at durable_lsn, advanced by the demand-driven group-commit flusher), every test that SYNCHRONOUSLY asserts "covering ACK ⇒ committed_lsn advances" must first force durability (persist_flush) or bounded-poll. ha-commit-durability 2.2 primed durable_lsn in the in-memory barrier_setup_vcpus stubs but MISSED the real-persist test_fact_hook_append_is_retained_and_nonblocking → 6/6 deterministic FAIL. Do NOT trust an impl-note that blames a test failure on a /tmp-overlay fdatasync-EIO without re-verifying: fdatasync returned rc=0 on /tmp here and the test passed pre-spec — it was a real durable-gating regression. Also: meson 'Fail:0' masks Unity sub-group failures — always gate on ANSI-stripped ':FAIL:' count from the test binary's stdout, not the meson summary.

• (2026-06-22, ha-term-safety task 3.3 remediation) Do NOT run clang-format -i on whole XCE controller source files (ha.c/ha.h/test files): these files are NOT clang-format-clean against the repo's config, so a full reformat rewrites ~1700 lines and BURIES the targeted change (and violates a "preserve the existing diff" instruction). I then made it worse with git checkout HEAD -- <files> to undo the format — but the uncommitted task-3.3 work lived ONLY in the working tree, so that wiped it (no stash, unrecoverable from git). Recovery worked only because the full original content was already in my context. Lesson: (1) only format the exact changed regions or skip formatting; (2) NEVER git checkout HEAD -- <file> to discard formatting noise when the file also holds uncommitted feature work — instead revert just the format hunks, or stash first. The reflog showed branch HEAD was already at task-3.2 (754e586), and the initial-context "recent commit 2c5d298" was a stale snapshot — the uncommitted 3.3 diff sat on top of 3.2.

• (2026-06-22, ACK peer-index, bug-960) The walAck branch of repl_ack_decode_and_dispatch (ha.c) fires for TWO endpoint types whose stored peer_idx means DIFFERENT things: promoted-vote eps (vote_ep_ctx_t, ctx!=NULL) carry a self-EXCLUDED votes index (relative); heartbeat standby eps (standby_ep_ctx_t, ctx==NULL) carry an ALREADY-ABSOLUTE candidate index. The authoritative branch is ha.c:6187-6189 (sctx->ctx != NULL ? votes_idx : i). The pre-existing ACK consumer ha_walack_on_ack(c, vctx->peer_idx, …) uses peer_idx DIRECTLY (no remap) — proof the ACK path's index is already absolute. Any code consuming the ACK peer_idx for the library (match_index[]/next_index[], which raft.c writes UNCHECKED) MUST branch on vctx->ctx != NULL before remapping, or it double-maps the heartbeat path → wrong-peer false quorum / OOB write. self_idx == ha->my_priority (raft_cfg.self_idx = my_priority everywhere).

• (2026-06-24, bug-1060 — RECURRENCE of the 2026-06-22 entry above) Ran clang-format -i on whole ha.c/ha.h/test_ha_election.c AGAIN → spurious whole-file reformat (3268/969/635 lines) that buried a ~180-line fix. Confirmed spurious by checking the repo's HEAD ha.c is NOT idempotent under the nix clang-format (a .clang-format exists at src/.clang-format but the pinned formatter version differs from whatever produced HEAD). Recovery THIS time succeeded because: (a) I had already printed the full pre-format git diff ha.c (5 hunks: bug-1028/1042/1043 pre-existing + my bug-1060) into context; (b) ha.h's only genuine new symbol vs HEAD was ha_sync_raft_log_from_recovery (found via comm on grep -oE '\bha_[a-z_]+\b' of HEAD-vs-current — everything else was pointer-style/comment reflow); (c) test_ha_election.c was NOT in the session-start M list so it held only my changes. Then git checkout HEAD -- <3 files> + re-applied the logical hunks by Edit. LESSON STANDS: do NOT clang-format whole files here; format only changed regions or skip. The git checkout HEAD -- rescue is ONLY safe after you have positively accounted for EVERY uncommitted change in those files (capture the pre-format git diff FIRST, and confirm which files had pre-existing work via the session-start git status).

• [2026-06-17] Do NOT invoke handle_error()/an endpoint err_handler from the SYNCHRONOUS send path (endpoint_enq / request.c senders). The err_handler is an io_worker-thread callback that may take ha->lock (vote_ep_err_handler, ha.c:3478) or ctx-level locks; the send path runs on the caller's thread, which often already holds that lock (controller primary_heartbeat_fn holds ha->lock while sending heartbeat NOPs) → self-deadlock on a non-recursive mutex (bug-652, a regression from bug-646). Correct pattern: endpoint_enq RETURNS a retriable error (-ECONNRESET) and the caller propagates it (request.c senders return it — bug-650); the real teardown stays on the io_worker thread (io_handler→handle_error), which holds no caller locks. General rule: a libxce send must never run endpoint teardown/callbacks inline — only surface the error.

• [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.

• [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).

• [2026-06-15] SOLVED (bug-563): "jobsub ok but joblist/workerlist empty on the HA primary; reslist works". ROOT CAUSE: cluster_start() is the ONLY setter of c->running = 1 and it is ORPHANED — zero callers (grep cluster_start(). The HA myapp_run path starts iomgr/vtaskmgr/delaymgr/persist itself but never calls cluster_start, so c->running stays 0 on the primary. check_controller() (the streaming gate xce_reply_get uses for joblist/workerlist) returns -EINVAL when !c->running; xce_reply_get then sets the chunk count nc = 0, shipping an EMPTY list even though collect_jobs() found the rows. reslist passes checkf=NULL so it bypasses the gate — that asymmetry is the tell. Single-node "works" because the legacy cluster_init path called cluster_start. FIX: c.running = 1; in myapp_run() after persist_start() (src/controller/myapp.c). RULE: when only SOME list ops are empty but the data is present server-side, check the xce_reply_get check_* predicate — a <0 return silently zeroes the chunk (nc=0), not an error. To prove it: log nc/checkstatus in the xce_reply_get send loop. Reproduce HA bugs locally with /xce-build + a 3-node deploy-ha under ASAN rather than theorizing. (My earlier UAF/stale-binary guess here was WRONG — disproven by "crash gone, binary current".) Still-latent: cluster_start also armed check_workers_timeout (delaymgr) — that side-effect remains unrestored on the HA path.

• [2026-06-16] When adding a NEW opcode to the HA protocol, it MUST be wired into ha_dispatch_gate() (src/controller/ha.c) — the role-based admission gate runs in myreq_handler (cluster.c:251) BEFORE the per-opcode handler, and an unlisted opcode is rejected (-XCE_ERR_STANDBY/-XCE_ERR_NOT_READY) so the handler never runs. bug-605: ha-prevote added OPCode_preVoteRequest (opcode 38) + handler (cluster.c:440) but not the gate; STANDBY voters silently dropped every incoming PreVote, the candidate counted the failed exchange as a denial, and a cold cluster looped PreVote forever with NO primary. INVARIANT: a PreVote is strictly non-mutating, so it must be admitted in EXACTLY the roles a real requestVote is (CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY). Diagnostic tell: voters log zero "PreVote DENIED" yet the candidate keeps "losing" — means the request never reached the handler (gate drop), not a policy denial. Confirm with gdb breakpoint on ha_dispatch_gate rc.

• [2026-06-16] HA election timeouts must be RANDOMIZED/STAGGERED on the FAILOVER path, not just at startup. bug-606: when the primary dies, the surviving standbys lose it at the same instant and (before the fix) campaigned in perfect lock-step — both win PreVote (neither sees a primary), both vote self at the same term, split the vote (quorum unreachable with 2 survivors), time out on a FIXED watchdog (raft.c:105) and retry on the FIXED heartbeat_timeout liveness cadence → term climbs unbounded, no leader. The jitter machinery (election_priority_step_ms=500, election_jitter_max_ms=200, xorshift64 on jitter_seed) existed but was wired ONLY into the startup heartbeat-wait (ha_wait_for_heartbeat_or_timeout / myapp.c stagger); the PreVote failover path (liveness_poll_fn→ha_on_primary_ctrl_fd_rdhup→ha_begin_failover→ha_start_prevote_async) bypassed it, and raft's own staggered-start (raft_on_leader_lost→on_start_election_after_delay, raft.c:421) is also bypassed by the PreVote path. FIX: ha_election_stagger_ms() + a delaymgr-scheduled, guarded PreVote start in ha_begin_failover (skip if a primary appears during the stagger). RULE: any new re-election/failover trigger must apply per-node stagger (priority*step + fresh random jitter, re-randomized each attempt) AND a "primary alive?" guard before campaigning. Diagnostic tell: identical ms-aligned election timestamps across nodes + term climbing by 1 each cycle = lock-step split vote, not a vote-policy bug. step MUST exceed jitter_max so adjacent priorities never overlap.

• [2026-06-14] A partially-initialized xce_reply_t is a UAF hazard. The Cap'n-C generated encode_RequestReply (controller.capnp.c) reads the COMMON TRAILER fields (redirectHost, redirectPort) on EVERY reply regardless of data_which — redirect_host is strlen'd whenever != NULL. Handlers declare xce_reply_t reply; on the stack and only set status/data, so those trailer fields were stack garbage; a garbage redirect_host aliasing a freed capnp segment = heap-use-after-free (seen on the OPCode_nop/state-report path, cluster.c). FIX (bug-562): init_reply_by_req() now memsets the whole struct first. RULE: any xce_reply_t built WITHOUT init_reply_by_req (e.g. ha_send_wal_ack_reply) MUST memset it. Generated union encoders are not safe to feed a partially-init struct — common/trailer fields are always serialized.

• [2026-06-04] capnpc-c 1.0.1 has a bug with :group inside a union when $C.codecgen is active: the encode/decode generator emits d->FIELDNAME (flat) instead of d->data.GROUPNAME.FIELDNAME, AND places the case as dead code after the prior break. Do NOT use inline :group in a union that has $C.codecgen. Instead, define a named outer struct (struct Foo $C.mapname("foo_t")) and reference it as a pointer union member. The named struct generates correct encode_Foo_ptr/decode_Foo_ptr calls. The read_RequestReply/write_RequestReply wire layer works correctly with groups but the codec layer does not.

• ha-consistency-invariants decomposed into 4 sub-specs (2026-06-11, user decision): umbrella spec keeps requirements/design/research as the authoritative contract (generates NO tasks); implementation via ha-commit-engine → ha-commit-barrier-rollout → ha-single-leader-role (HARD ordering: barrier before COMMITTING removal), with ha-convergence-recovery parallel after engine. Briefs reference umbrella docs with per-spec requirement ID subsets. Dependency order lives in steering/roadmap.md "## Specs (dependency order)" (kiro-spec-batch parseable). Rationale: ~30-38 tasks in one spec exceeds the ~20-task decomposition threshold; phases are independently shippable (each leaves the system no less consistent).

• Commit-first-then-reply contract (2026-06-11, user, ha-consistency-invariants): job/queue/host(worker) control operations MUST follow validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply. No success reply before commit. Known violators to restructure: do_jobcontrol_op (job.c:921-934, replies with only async WAL), handle_jobdispatch (job.c:459-471, replies before processing — classification pending). handle_jobsubmit is the reference implementation.

• Four design decisions locked (2026-06-11, user, ha-consistency-invariants): (1) fact entries follow pure Raft — log-is-the-queue, append to WAL + retry replication forever, NO commit timeout for facts; (2) crash-loss of uncommitted facts accepted until worker-rejoin-resync lands (strictly better than today); (3) demotion eviction = process SELF-RESTART after truncation (rebuild-in-place only if restart proves disruptive); (4) own-term commit via explicit NO-OP WAL entry at promotion. Rationale: "follow Raft as it has been well verified."

• Fact-vs-request commit semantics (2026-06-11, user decision, ha-consistency-invariants Req 2.2/2.8): job state transitions are REPORTED FACTS, not requests. Request-shaped ops (submit, queue ctrl, config) reject to requester on commit timeout; fact-shaped ops (worker-reported transitions) are retained and retried until committed — including across quorum loss; primary never discards an uncommitted fact while running+leading. Backstop for facts never received / lost across failover: worker-rejoin-resync (roadmap backlog, designated official backstop).

• vcpus is operator tuning, not a spec constant (2026-06-11, user): --vcpus should be sized per deployment by host capability + cluster size + workload; 8 is only the fallback default. Specs/designs must express thread-pool-relative mechanisms as functions of n_vcpus, never absolute counts, and put sizing advice in operator docs.

• vtask pool sizing facts (verified 2026-06-11): default is 8 (NOT 4 — ha-async-io.md was stale, now fixed) via --vcpus CLI override, cluster.c:1082. WAL ACK processing runs on the IOMGR thread (peer_ctrl_msg_handler → repl_ack_decode_and_dispatch → repl_on_ack, ha.c:2770-2792) — so blocking commit-waits on vtask threads can never deadlock their own wakeups; all-workers-blocked is a throughput problem only. Pool size should scale with write concurrency/cores, NOT with n_peers (ACKs gather in parallel; Little's law). Mitigation pattern: commit-wait admission cap (n_vcpus - K), async continuation as durable fix.

• xzarray COW shadow is DORMANT infrastructure (discovered 2026-06-11): xzarray_freeze/xzarray_thaw (zarray.c:695-757) are built, in the varray vtable, unit-tested (test_zarray_cow.c) — but have ZERO production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). It is slot-level STRUCTURAL COW only: slots hold item pointers, so in-place field mutations are visible through both shadow and active views; freeze is O(1) pointer swap, reads merge active-over-shadow, thaw merges back with a should_retain GC filter, single generation (-EALREADY on double freeze). Intended consumer: the never-implemented entity-bearing checkpoint (Task 11 / sentinel-only snapshots).

Decision Log

• Recovery-oriented reframing supersedes ARIES/undo direction (2026-06-10, user insight, ha-consistency-invariants): replication here serves FAILOVER/RECOVERY, not transactions. Structural fact: a single-writer leader never truncates its own log while leader, so provisional in-memory state on a live primary either commits later or the node is demoted — discard coincides exactly with role/lifecycle events. Therefore NO per-entry undo log; discard handled by recovery mechanics (truncate divergent suffix + coarse rebuild via existing persist_recover/replay_wal at demotion/rejoin). Only the watermark egress filter survives (entity WAL-LSN tag, committed-only export to dataex/scheduler) — justified as failover correctness (prevent orphan dispatch), not isolation. Also: Raft commitIndex is volatile — prefer leader-driven log reconciliation at rejoin over persisting committed_lsn.

• STM evaluated and rejected for provisional-state mechanism (2026-06-10, ha-consistency-invariants): user asked if STM could help; analysis concluded full STM is a mismatch (commit point = majority network ACK up to seconds — STM transactions can't span syscalls; libitm/TSX dead in C; single-writer + RTTI per-object locks already cover the concurrency). Adopted the useful subset instead: "committed-view + undo log" (ARIES/MVCC-lite) — entities tagged with WAL LSN, committed-only egress filter for dataex/scheduler, per-in-flight undo records for prompt rollback on abort. Dual-zarray COW checkpoint mechanism is the in-house precedent for maintaining a second consistent view.

• Provisional-state model for Req 2.4 (2026-06-10, user decision, ha-consistency-invariants): in-memory controller state is a PROVISIONAL materialized view; the persisted majority-committed WAL (+snapshot) is the authoritative cluster state. Chosen over hook inversion (apply-after-commit). Mutation order stays mutate-first; instead, commit gates (1) requester confirmation and (2) irrevocable-action egress — dataex/scheduler export must filter uncommitted entities (closes the dispatch-of-uncommitted-job race). Status queries MAY observe provisional state (documented deviation; read consistency out of scope). Consequence: §C convergence/reconciliation work (truncate-on-demotion, phantom eviction, persisted committed_lsn) is mandatory core, not optional — "provisional" is only true if something discards it.

• Raft + CP adopted as governing HA model (2026-06-10, user decision, supersedes "No Raft" in product.md): consistency requirements are DEDUCED from the Raft paper + CP, not chosen per-feature. Concretely: (1) no CTRL_PRIMARY_COMMITTING state — Raft has only Follower/Candidate/Leader; new-leader safety comes from the own-term commit rule (§5.4.2), not a fourth state; (2) majority ACK is the hard commit barrier for ALL state-changing writes — reject on timeout/quorum loss, never local-apply-and-degrade (committed_lsn ≈ commitIndex); (3) replication fans out to all standbys with leader-driven continuous catch-up; majority is the commit threshold, not the fan-out target. Single-writer log retained as XCE's simplification. Spec: .kiro/specs/ha-consistency-invariants/brief.md. Implementation ordering constraint: universal commit barrier MUST land before removing COMMITTING.

• Replication payload wire format (2026-06-09, ha-repl-hardening 7.1): chose the established _raw sender convention (manual capnp build via capn_new_list8/capn_setv8 + xce_decode per-opcode raw extraction into raw_payload/raw_payload_len struct fields) over extending the typemap codegen to map capnp Data ↔ raw buffers. Rationale: replConfigPush/upgradeConfig already prove the pattern in production; the codegen route risks regressions across all opcodes and fights the known capnpc-c 1.0.1 codecgen bugs. Schema change: Request.body union gained walcatchupentrymsg @28 / snapshottransfermsg @29 — these messages previously had NO union arm at all (sent as body_which=nomsg, i.e. nothing crossed the wire).

• Convergence threshold (2026-06-09, ha-repl-hardening 7.5): chose requirements.md Req 2.3 formula — ceil(n_peers/2)−1 converged standbys (integer form n_peers/2) — over the code's previous all-standbys rule. Rationale: primary + converged standbys form a cluster majority, matching Raft quorum; committed entries are already on a majority per-entry, and safe_committed_lsn derives from a majority of voters, so promotion with one dead minority standby is safe and avoids a 60s degraded-mode timeout. Lagging standbys keep catching up via the heartbeat decision loop after promotion.

• Election liveness fix (2026-06-09): skip_election path in myapp.c had two bugs: (1) voted_for_name != "" triggered even for self-votes and stale terms — fixed by adding strcmp(name,"self")!=0 && voted_for_term==ha->term; (2) ha_arm_heartbeat_timeout_vtask was never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.c handle_request_vote_req also simplified to term-only deny.

  • CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.

• Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.

• (2026-06-11, ha-convergence-recovery spec) repl_complete_catchup (repl.c:521) has NO production caller — in_catch_up is set by broadcast-overflow eviction (repl.c:240) and never cleared; re-inclusion trigger is wired by the ACK-driven catch-up continuation.

• (2026-06-11) Scheduler-bound job egress has TWO seams, not one: full push xcesched_push_jobs (sched.c:169, on scheduler join) AND incremental JOB_NEW push sched_job_new_fun→xcesched_push_joblist (sched.c:236-248). Status reads (apiserver.c:83, opget.c:173) share collect_jobs (dataex.c:294) and must stay unfiltered per umbrella deviation.

• (2026-06-11) No saved-argv/execv machinery exists anywhere in src/controller or src/lib — demotion self-restart is net-new; job_t is a thin wrapper { xce_job_t job; } at src/include/xce/types.h:483-485 (controller-local fields go on the wrapper, wire type untouched).

• (2026-06-16, ha-prevote cleanup) The HA election carrier ha_elect_ctx_t (ha->elect_ctx) is deliberately leaked one-per-round on BOTH the real-election and PreVote paths — vote endpoints hold a bare vctx->ctx pointer freed async by endpoint_fini, so freeing the carrier risks UAF; there is NO real ha_election_cleanup(). Pre-existing + bounded (rare failover events). Deferred by user decision to follow-up spec .kiro/specs/ha-elect-ctx-lifecycle/ (refcount-the-carrier or teardown-ordered free, both paths). Do NOT attempt a drive-by free — UAF on failover is worse than the bounded leak. The ha-prevote 3.1 log-reason nit is RESOLVED (commit 47f7338): ha_heartbeat_is_fresh un-static'd + declared in ha.h; cluster.c denial-reason now mirrors raft_cb_vote_eligible order exactly.

• (2026-06-17, bug-646 client-hang fix) User rejected adding a client reply-wait timeout (cbuf_timedget) as a fix for the failover hang: "a timeout may hide potential issues and make things more complex." Principle adopted: fail loud at the source, never mask with a backstop timeout. The hang was a netio silent-swallow (endpoint_update_epoll returned 0 on iomgr_update_poll/EPOLL_CTL_MOD failure), so the fix surfaces the error and runs the error path (wake the waiter via handle_disconnect → failover rotates), rather than papering over the missing reply with a deadline. When choosing between "add a timeout" and "propagate the real error," prefer propagation on this codebase.

• (2026-06-16, ha-prevote impl) The controller "controller test" 300s TIMEOUT under meson test is a WALL-CLOCK WRAPPER ARTIFACT, not a hang/failure: running the binary directly (ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test) completes all 564 tests incl. the slow test_commit_barrier_reply (61 tests, real 500ms commit barriers) with 0 failures. For a true controller pass/fail, run the binary directly (no cap) or filter with -t <suite>; do NOT treat the meson timeout as a regression. ha-prevote feature COMPLETE: 11 tasks, raft suite 123/0, controller 564/0, real-vote path byte-for-byte unchanged (0 raft.c deletions), commits 0d66026..e6b234a.

• (2026-06-15, ha-prevote design review) PRODUCTION vote-reply tally path = raft_on_vote_reply in the raft library, NOT ha_elect_ctx_t. vote_reply_dispatch (ha.c ~2630-2650) calls raft_on_vote_reply directly in production; the ha_elect_ctx_t / handle_request_vote_reply / ha_elect_evaluate_and_signal path runs ONLY behind #ifdef XCE_TESTING via xce_vote_reply_hook (plus the startup/first-boot ha_elect_ctx_t.done semaphore). The raft library owns quorum tracking and fires on_become_leader/on_become_follower. Do-Not-Repeat: when mirroring the election reply path, mirror raft_on_vote_reply (raft lib), not ha_elect_ctx_t. (An earlier exploration mislabeled ha_elect_ctx_t as the operative tally; caught in design validation.)

• (2026-06-15, user decision, ha-prevote RN-2) Leader-stickiness window for PreVote = reuse ha_heartbeat_is_fresh() / heartbeat_timeout_ms (10s), gated on role==CTRL_STANDBY. Rationale: the stickiness window MUST equal XCE's failure-detection window or you get either a disruption gap (window too short) or a liveness stall (window too long). XCE splits textbook Raft's overloaded "election timeout" — HEARTBEAT_TIMEOUT_MS (10s) is the leader-failure detector; ELECTION_TIMEOUT_MS (5s) is only the candidate vote-collection deadline. Raft's "minimum election timeout" disruption rule maps to HEARTBEAT_TIMEOUT_MS, NOT ELECTION_TIMEOUT_MS. Reusing it keeps stickiness == failure-detect == existing real-vote gate (raft_cb_vote_eligible), one predicate; role-gate handles RDHUP/connection-loss for free. A shorter window buys no failover speedup (nobody campaigns before failure-detect anyway; real crashes RDHUP immediately) and only opens a disruption gap on flaky links. Rejected a separate prevote_stickiness_ms constant (lets the two timeouts drift). Full comparison table in .kiro/specs/ha-prevote/research.md §6.

• (2026-06-15, user decision, ha-prevote) Product is PRE-RELEASE: no backward/rolling-upgrade compatibility is required for any HA/wire change. All controllers in a cluster deploy from the same build, so there is no mixed-version (old-binary ↔ new-binary) cluster. Wire-format changes (new capnp opcodes/fields, vote-RPC changes) may be made freely without negotiation/fallback paths. Do NOT raise inter-version compat as a requirement or research item. (Dropped RN-5 from ha-prevote/research.md on this basis.)

• commit_submit(COMMIT_MODE_RETAIN) is REGISTRATION-FREE (commit.c:233-239, verified 2026-06-11): returns 0 immediately, records NOTHING per entry — commit state derives entirely from registry watermarks at ACK time. RETAIN routing at the hook seam therefore has NO engine-side observable distinct from the append itself; tests of RETAIN routing must assert classification + non-blocking return + lsn > committed_lsn pending + ACK-driven advancement, never a fake "registered" counter. (ha-commit-barrier-rollout 1.1)

• XCE_TESTING fail-fast (assert/abort) paths are testable via fork(): child silences stderr, calls the guarded fn, _exit(0); parent asserts WIFSIGNALED && WTERMSIG == SIGABRT. Precedent: lib testsuite test.c:1783; ASAN-compatible with detect_leaks=0. (ha-commit-barrier-rollout 1.1)

• ha_has_write_quorum (ha.c:1507) is STATIC — testsuite reaches it only through ha_dispatch_gate(ha, opcode) (extern-declared in test files, pattern: test_ha_repl_ack.c). Gate fixtures: ha_init + n_peers + atomic role store + heap zarray standby_endpoints (test_ha_connection_reuse.c pattern). Catch-up exclusion at the gate is untestable until the ha_state_t.repl back-pointer lands (ha-commit-engine 4.2) — assert via repl_active_standby_count instead. (2026-06-11, task 4.1)

• persist_commit_barrier watermark = current_lsn - 1 (matching persist.c:1021 persist_wal_sync_append read); covered ⇔ current_lsn == 0 OR committed_lsn >= current_lsn - 1. Design §Commit Barrier Helper "lsn = p->current_lsn" is shorthand — the parenthetical persist.c:1019-1021 reference pins it to current_lsn - 1 (engine clamps committed to highest appended, so waiting on current_lsn itself would never complete). (2026-06-12, task 2.1)

• (2026-06-13, single-leader-role 4.2) ha_dispatch_gate's OUTER default: return 0 ADMITS — so after deleting the case CTRL_PRIMARY_COMMITTING: block, a role manually set to COMMITTING falls through to admit (not NOT_READY). That is exactly why test_ha_startup.c:test_dispatch_gate_committing_denies_job_ops flips from PASS→FAIL (Expected -10013 Was 0) — category-(a), 5.1-owned. The 6 PRIMARY||COMMITTING compound sites in ha.c: raft_cb_vote_eligible(~215), handle_request_vote term-stepdown(~1984)+role-deny(~2004), peer_ctrl_msg_handler case label(~2820), vote_ep_err_handler(~3088), primary_heartbeat_fn re-arm(~3883). Collapsing all to PRIMARY-only makes a manually-COMMITTING test role a NON-leader. Census after 4.2: exactly 5 FAIL, ALL COMMITTING-state category-(a) (test_ha_startup ×2, test_ha_rdhup_integration, test_ha_repl_push, test_ha_connection_reuse), zero category-(b). enum value CTRL_PRIMARY_COMMITTING=4 stays in ha.h (5.2 deletes it); no ha.c CODE branches on it post-4.2 (only comments in 2.2/3.1-owned functions remain). Unity attributes vote-gate cases to test_ha_failover.c by group UNITY_BEGIN — locate by NAME. FAIL bytes: name:\x1b[41mFAIL\x1b[0m (color BETWEEN ':' and FAIL) — grep raw FAIL, count was exactly 5.

libxce async failover (2026-06-16)

• xce_request(ep,req) is fire-and-forget enqueue; reply status is only available at the cbuf_get round trip in xceapi.c. xce_request_with_retry() production branch is a no-op stub (retries only under the test hook). Any client-side retry/failover MUST wrap the send+wait round trip at the api_ctx_t level, not xce_request. Controller redirect (E_XCE_REDIRECT, join.c) sends status-only; redirect_host/port schema fields exist (controller.capnp:568) but are never populated. See spec libxce-failover, bug-608.
• xceapi.c per-op guard rejects ctx->state != XCE_CTX_RUN with -EINVAL; the transport-error path (xcectx.c:148) drives ctx to XCE_CTX_ERR -> this is the exact source of the failover "Invalid argument" symptom. Any failover must restore XCE_CTX_RUN after reconnect.
• ctx->wbuf is a single-slot status-only channel (cbuf_init ...,1,sizeof(int), xcectx.c:185): no seq correlation, so client is single-in-flight per ctx. Retries must drain/seq-match to avoid consuming a stale reply.
• Job submit uses a CLIENT-generated jobid (job.c:56) + server EEXIST/E_XCE_DUP_JOB replay guard (job.c:95) -> writes are idempotent on re-send; DUP on a retried submit means success.
• libxce layering: xcecli_* (client.c) is the UNIVERSAL client surface — worker (controller_connector.c), scheduler (scheduler.c), apiserver cgo (clib.go), xce_test all call xcecli_, never xceapi xce_ directly. Only api_ctx_t (client.c) holds the master list; xce_ctx_t (xceapi/xcectx.h) does not. => client-side cross-cutting concerns (failover/retry) belong in client.c wrapping xce_, keeping xce_ one-shot/fail-fast and xceapi unchanged.

Decision: full-state rejoin on controller failover (2026-06-16)

• On failover, stateful daemons re-join the new primary and apiserver/scheduler receive FULL state (full job list), not a delta. Accepted because failover is rare, some handshake is unavoidable (new primary has no live connection to the daemon until rejoin), and full-state-on-rejoin is simpler/correct and still better than IBM LSF (full-history replace, no snapshot). Lightweight re-attach / delta transfer deferred (YAGNI) — roadmap backlog. Revisit only if measured as a scale problem.

Session 2026-06-17 (bug-661 reproduction) — learnings

Do-Not-Repeat

• pkill -9 -f xce_controller inside an INLINE Bash one-liner kills its OWN shell — the inline command's cmdline contains the literal "xce_controller". Use pkill -9 -x xce_controller (exact comm match). Scripts run as a file are safe (cmdline = bash script.sh ...).
• Daemons spawned with start.sh ... & inside a Bash TOOL call get reaped when the call returns. Run multi-process HA E2E via the self-managing script with run_in_background: true; write logs to the WORKTREE (e.g. ./dur-run.log), NOT /tmp (the sandbox tears /tmp files down between calls).
• The Bash tool has a ~120s wall limit; long foreground HA runs get SIGKILLed and buffered stdout is lost. Use discrete >> file appends (each echo flushes) or background+poll.

Key Learnings

• bug-661 (committed-job loss on failover) is a RACE: wal_sync_ms is hardcoded DEFAULT_WAL_SYNC_MS=10 (config.h:86), NO CLI/env override. Loss needs the ~10ms un-fsync'd window hit AND a killed+restarted node re-leading before replication catch-up repairs its WAL. Single-node kill+restart soak passed 8 rounds clean; reproduced loss once (4 jobs). Deterministic repro needs a configurable fsync interval (test seam for ha-commit-durability).
• Cold-start 3-node election on /tmp/xce-ha-deploy takes ~15s to first CTRL_PRIMARY.
• xce_test reads piped stdin via readline and exits on EOF — drive it with printf 'set masters ..\nclient\njobsub ..\n' | xce_test.
• (2026-06-18, task 1.3 FINAL) bug-661 deterministic repro = bash src/tools/test-commit-durability.sh --deterministic (strace-based GAP proof, NOT joblist data-loss). WHY NOT data-loss: single-host SIGKILL is physically NON-deterministic — persist_wal_async pwrites INLINE (page cache survives process death; kernel still flushes), and standbys fdatasync-before-WalAck (persist.c:515) so a committed job is always durable off the primary. A rolling-primary-kill data-loss run flips GREEN/RED/RED across identical invocations (REJECTED by review). The DETERMINISTIC slice: with --wal-sync-ms 30000 the primary's fsync_worker never fires in-window, so the primary replies "jobsubmit ok" with ZERO primary fdatasync between submit and ack. The mode straces each controller (strace -f -tt -e trace=fdatasync,fsync, exec-wrap — PTRACE_ATTACH -p is blocked by yama, exec-wrap works) and asserts in-window primary syncs==0 ⇒ RED. CONFIRMED 15/15 RED across 5 fresh-cluster runs; asymmetry primary=0 vs standby=2 in-window. Valid RED→GREEN gate: the fsync-before-ack fix (task 2.x) makes the count ≥1 ⇒ GREEN. The exact defect: commit.c:79 majority_candidate_locked caps the primary self-contribution at highest_appended_lsn (page cache), not a durable_lsn. --wal-sync-ms plumbed via deploy-ha.sh start.sh $* passthrough.
• (2026-06-18, GOTCHA) A SINGLE double-kill (primary + 1 standby) does NOT reproduce bug-661: this codebase enforces STRICT majority — a lone survivor will NOT self-elect (harness log: "no single-voter primary within 40s"). The documented required=1 loss came from ROLLING failover across terms (incident: ctrl-3 became PRIMARY term=4). Also, committed jobs are durable on the STANDBYS (persist_wal_write_replicated fdatasyncs before WalAck, persist.c:515) — only the PRIMARY's copy is un-fsync'd. So the loss needs rolling hollow re-leads faster than catch-up repair, not just one kill. The deterministic harness drives rolling primary kills to hit this.
• (2026-06-18, BUILD) meson/ninja are NOT on PATH in a fresh sandbox shell — they live in the nix store. Build a single target into the EXISTING src/xce.build (no clean) with cd src && nix develop --command bash -c 'meson compile -C xce.build <target>'. xce-build.sh's setup_build does rm -rf (clean) — avoid it for incremental. The controller/xce_test binaries also need the nix runtime libs (openssl etc.), so run the durability harness itself under nix develop ./src --command bash -c '...'.

Decision Log — 2026-06-17: ha-commit-durability mechanism

• INVARIANT (user, authoritative): reply "ok" to client ⇒ WAL entry fdatasync'd on a fault-tolerant majority (self + ⌊N/2⌋ standbys).
• DEFECT: commit.c:majority_candidate_locked counts self as highest_appended_lsn (page-cache), not fsync'd. Standby already persists-before-ACK (persist.c:515); primary does not (relies on fsync_worker timer) → "committed" durable on only 1 node = zero durability fault-tolerance = bug-661.
• DECISION: synchronous persistence — eliminate fsync_worker as the durability mechanism. Primary fdatasyncs before its self-contribution covers an LSN; commit.c self uses durable_lsn not highest_appended_lsn; reply stays gated on commit barrier. Liveness-safe because the submit path already blocks OFF the iomgr thread waiting for majority replication (job.c:568-569).
• THROUGHPUT: per-op fdatasync; array amplification removed by roadmap item job-array-single-wal-record (array → 1 WAL record → 1 fsync). Group-commit (batched fsync) is a deferred strict optimization (same contract), add only if high-rate independent submits measured as a bottleneck.

Decision Log — 2026-06-18: ha-commit-durability design-review reversal (group-commit)

• /kiro-validate-design surfaced: highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage; persist.c wal_classify) — high-frequency, NOT array-collapsible, NOT covered by job-array-single-wal-record, and not client-latency-bound. A pure per-entry synchronous design (delete fsync_worker) would penalize the dominant write path.
• REVERSAL (with user): do group-commit NOW. Repurpose fsync_worker timer→demand-driven flusher: on append-signal, one fdatasync over pending appends, advance durable_lsn, wake commit-eval. commit self-count uses durable_lsn. Roadmap "WAL group-commit batching" item ABSORBED into ha-commit-durability.
• Lesson: when deciding sync-vs-batched durability, classify ALL write modes first (wal_classify), not just the client-acked submit path. The RETAIN fact stream dominates volume.
• Gating items pinned for tasks: (1) freshness-floor formula + raft.tla liveness+safety re-check; (2) FATAL-vs-fence boot decision point; (3) flusher lost-wakeup proof (level-triggered on appended-vs-synced gap).

Key Learning — 2026-06-18: commit-ordering rule documented

• Authoritative commit/durability invariants now live in src/controller/CLAUDE.md (Safety Invariants + Protocol Verification Checklist):
  • Commit Ordering (RULE, true today): committed_lsn advances only as a contiguous LSN-ordered prefix; single-writer p->lock is the serialization point; durability batching flushes a contiguous prefix only (durable_lsn is a prefix watermark, never per-entry); log order must preserve causal deps (submit before start); out-of-order/parallel commit NOT supported (single-log architecture).
  • Commit Durability (TARGET RULE) carries a KNOWN GAP note for bug-661/ha-commit-durability: "committed" does NOT yet imply durable-on-majority until the spec lands. Task 5.3 flips the note to a guarantee when the code lands.
• When a safety invariant changes, document it in src/controller/CLAUDE.md (not just the spec) — but mark not-yet-true guarantees as KNOWN GAP so fresh sessions aren't misled into assuming durability that doesn't exist.

Key Learning — 2026-06-18: build invocation in worktrees + impl preflight

• The flake is at src/flake.nix, NOT the repo root. The xce-build skill's nix develop --command ... only works after cd src. Correct build: cd src && nix develop --command bash tools/xce-build.sh controller (~2-3min, debug+ASAN+tests; nix env is cached, no network). Running from the worktree root fails: "not part of a flake".
• meson/ninja are ONLY inside the nix shell (not on PATH); cc/gcc are on PATH but won't configure the meson tree.
• ha-commit-durability impl preflight (2026-06-18): tasks approved; cleaned baseline by committing 2 pre-existing WIP units (ha-election stagger + libxce netio error propagation); baseline build+controller tests GREEN. Sandbox caveats stand: 120s Bash cap (background+poll builds), daemon spawns reaped (E2E task 5.2 needs host), no confirmed TLA+ toolchain (tasks 4.1/5.3 need host).

Key Learning — 2026-06-18: raft.tla does NOT model durability

• raft/raft.tla (Ongaro's spec) restores log intact from "stable storage" on restart (raft.tla:165-174) — it ASSUMES durable logs, i.e. assumes exactly what bug-661 violates. Running TLC on the unmodified spec PASSES while the durability bug is still present. raft.tla validates election/freshness logic (mlastLogIndex/LastTerm), NOT commit durability.
• The repo has only raft.tla — no .cfg/model/tla2tools.jar/.toolbox/runner; java not on PATH. Project practice = manual conceptual alignment + the CLAUDE.md Protocol Verification Checklist, not automated TLC. So tasks 4.1/5.3 are reasoning tasks (sandbox-doable), not "run TLC" tasks.
• TLA+ Toolbox (bundles TLC) — or tla2tools.jar CLI + Java — is sufficient TOOLING for any model-checking, but proving the DURABILITY invariant would require EXTENDING raft.tla (durableLog vs log; Restart truncates to durableLog). That spec-extension is the real work, not the tool. Tasks 4.1/5.3 now carry this caveat so an implementer won't conclude "TLC passed ⇒ durability proven".

Key Learning — 2026-06-18: /tmp overlay fdatasync EIO breaks durability tests

• This sandbox's /tmp can be an OVERLAY filesystem where fdatasync returns EIO (errno=5), intermittently under load (3000+/run observed). Controller unit tests mkdtemp their WAL dirs into /tmp/xce_*, so their fdatasync hits this.
• CONSEQUENCE after ha-commit-durability 2.1/2.2/2.4: committed_lsn is gated on durable_lsn which advances ONLY on fdatasync rc==0 (fail-loud). On EIO, durable_lsn correctly HOLDS, so committed_lsn cannot advance — this is the gate WORKING AS DESIGNED, but it makes test_fact_hook_append_is_retained_and_nonblocking (test_commit_barrier_rollout.c) FAIL on overlay /tmp ("covering ACK must advance committed_lsn, Expected 1 Was 0"). The SAME test FAILS on the clean pre-spec baseline too on overlay /tmp — NOT a regression from any single task.
• Prior task reviews reported "0 Failures" because they ran on a tmpfs /tmp or a quiet-IO window (EIO is intermittent). Don't trust a single "0 Failures" run on this fs.
• ACTION for final validation (tasks 5.x / /kiro-validate-impl): run the controller suite on a REAL fsync-able filesystem (tmpfs /dev/shm, or point test temp dirs at the worktree) so durability tests aren't masked by spurious EIO. The worktree xce.build is on a real fs (build is fine); it's the TEST WAL dirs in /tmp that are the problem. Always rm -rf /tmp/xce_* before a run (stale dirs also cause an "Expected Non-NULL" mkdtemp cascade).

Key Learning — 2026-06-18: meson "Fail:0" masks Unity sub-group failures

• The controller suite is ONE meson test ("controller test") running the aggregated controller_unit_test binary. A Unity TEST_ASSERT_* failure inside a sub-group (e.g. test_fact_hook_append) is recorded + printed :FAIL: in testlog.txt but is NOT always propagated to the process exit code, so meson test can report Ok:1 Fail:0 while a test actually FAILED. A C assert() (SIGABRT) DOES propagate (so RED-phase asserts are reliable).
• RELIABLE GREEN GATE: grep -c ':FAIL:' src/xce.build/meson-logs/testlog.txt must be 0 — do NOT trust the meson Fail:0 summary alone.
• Combined with the /tmp-overlay-fdatasync-EIO issue, the documented pre-existing test_fact_hook_append_is_retained_and_nonblocking (commit_barrier_rollout) intermittently :FAIL:s on overlay /tmp (EIO holds durable_lsn ⇒ committed can't advance over the retained fact — the durability gate working). Final validation (5.3 / /kiro-validate-impl) MUST run on a real fsync-able fs (worktree/tmpfs) AND grep testlog for :FAIL:.

Decision Log — 2026-06-18: a controller MUST NEVER self-exit (user invariant)

• AUTHORITATIVE (user, 2026-06-18): a controller process exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. It must NEVER self-exit()/abort() as a designed response to an operational condition. Plain assert() for genuine programming-invariant bugs is still OK (that's a "crash").
• CONSEQUENCE: the ha-commit-durability "FATAL → tlog_exit(); abort()" terminal (tasks 3.3/4.3/4.4, persist_recover_fatal_if_unrecoverable + persist_fatal_hook) was WRONG — it self-terminated on an operational condition (recovery-truncated WAL after failover). This was the root cause of the 5.2 soak quorum collapse (self-aborting nodes), NOT a sandbox artifact (correcting the earlier cerebrum framing).
• CORRECT behavior (user-chosen): an unrecoverable member (truncated, can't catch up) logs LOUDLY (operator-visible ERROR, with the truncation boundary) + stays PERMANENTLY non-promotable/fenced + STAYS ALIVE + KEEPS RETRYING catch-up indefinitely. "Don't serve divergent state" (Req 2.4) = refuse to lead, NEVER kill the process. No bounded-window-to-abort deadline. If a peer returns it catches up and clears the fence; otherwise it waits (loud) for an operator.
• REWORK (in progress): remove abort/exit/persist_fatal_hook from 3.3; the decision's FATAL terminal becomes a non-terminating "loud permanent fence"; 4.3/4.4 route to it (stay alive); update Req 2.4 / design §3.2 table / §3.3 wording from "FATAL/terminate" to "loud non-promotable fence, stays alive"; update tests.
• REWORK DONE (2026-06-18, rework-no-self-exit). Concrete shape: (1) enum RECOVER_DECISION_FATAL→RECOVER_DECISION_FENCE_UNRECOVERABLE (persist.h:61). (2) persist_recover_fatal_if_unrecoverable(const inputs*)→persist_recover_loud_fence_if_unrecoverable(persist_t* p, const inputs*) (persist.c:1788) — logs LOUD ERROR (boundary), sets sticky flag, RETURNS; NO tlog_exit/abort. persist_fatal_hook global DELETED. (3) sticky _Atomic bool recovery_unrecoverable on persist_t (persist.h, after recovery_verdict) + persist_recovery_unrecoverable(p) accessor + persist_clear_recovery_unrecoverable(p) (clears on a genuine transition + logs "fence RELEASED"); init false in persist_init (persist.c:451). (4) ha.c ha_evaluate_promotion_fence FENCE_UNRECOVERABLE arm: calls loud-fence with p = ha->raft_c ? &ha->raft_c->persist : NULL (pure-decision callers pass NULL) + sets in_catch_up. (5) ha.c promotion_fence_timeout_fn no longer aborts: on still-unrecoverable it RE-EMITS loud alert (sets sticky flag) + RE-ARMS the timer (periodic operator re-alert, indefinite); on caught-up clears in_catch_up + sticky flag; UAF gotcha — capture t->target_hwm to a local BEFORE free(t) then re-arm (the -Wuse-after-free caught it). ha_arm_promotion_fence_timer fail path emits loud-fence once (stays alive), no abort. (6) become_leader gate: also refuses while persist_recovery_unrecoverable(&c->persist) (belt-and-suspenders), clears flag on ELIGIBLE. (7) Tests: dropped persist_fatal_hook/dur_fatal_hook/g_fatal_hook_calls; rows 4/4b assert decision==FENCE_UNRECOVERABLE + loud-fence helper RETURNS (process stayed alive) + sticky flag SET + clears on persist_clear_recovery_unrecoverable; timer tests use dur_wait_unrecoverable(c,ms) polling the sticky flag; renamed *_no_fatal/_prevents_fatal/_bounded_fatal/_defers_fatal. VERIFIED: controller suite Fail:0 / :FAIL:==0 ASAN ×2; grep abort(/exit(/tlog_exit( ha.c=0 everywhere, persist.c only in comments; production xce_controller nm has the 4 new T symbols and NO persist_recover_fatal/persist_fatal_hook. NOTE: ha_self_restart (Req 1.3/1.4 demotion execv-restart, ha.h:305) is a SEPARATE primitive — its abort-on-exec-fail is OUT OF SCOPE for this rework (it's the convergence/restart path, not the recovery-unrecoverable operational fence) and was left untouched.

xce_test scripting primitives (xce-test-e2e-primitives)

• $jobid/$jobcount/$? auto-vars substitute ONLY as a WHOLE token (preceded by line-start/whitespace). MARK=$jobid is NOT expanded; use a space: echo MARK $jobid. (test.c substitute_autovars, ~line 462)
• jobsub on a clean ack sets $jobid to the fmt_jobid form (non-array N-N-N-N, array N-N-N-N[i]). To capture it in a shell driver, echo MARK $jobid then parse the marker line — do NOT re-impose a numeric id regex (drops array ids).
• In a shell driver, script+=$(printf '...\n') STRIPS the trailing newline (command-subst), concatenating lines into one bad token. Accumulate with explicit $'\n' instead.
• xce_test exit code is the verdict; run with ASAN_OPTIONS=detect_leaks=0 (known client leak) or the exit code is untrustworthy. stderr (errors/sanitizer) does not corrupt stdout capture.

Key Learning (2026-06-21) — WAL entry term is dead (bug-851 keystone)

wal_entry_hdr_t.term (wal_types.h:30) is real but the writer does hdr.term = p->term (persist.c:588/726/1040) and p->term is NEVER assigned from ha->term anywhere in the controller — so every WAL entry persists term=0. waldump confirms term=0 cluster-wide. This makes Raft (term,lsn) log-matching impossible, so reconcile can't truncate divergent same-LSN tails. Under CHAOTIC failover (kill-the-primary repeatedly), a restarted node recovers a stale term and re-elects at the SAME term (PreVote does not persist/propagate the won term to a majority before the leader acts) → two leaders per term → silent committed-prefix divergence. A CLEAN single failover converges fine (848/849/850 validated). Fixing this is HA-core + persistence = full spec workflow, not a quick patch.

Decision Log (2026-06-21) — ha-term-safety: unify config + job state in one log, config bytes out-of-band

For the ha-term-safety spec (bug-851 fix), the user chose a SIMPLE, UNIFORM consistency model: configuration changes and job state share ONE replicated (term,lsn) log. A config change is a LIGHTWEIGHT in-log MARKER carrying only a reference (version_tag + sha256), never the bytes. The real config bytes stay OUT of the WAL records and OUT of the cluster (state) snapshot — transferred/stored out-of-band. Design direction: model the config artifact LIKE A SNAPSHOT for replication (chunked, (term,lsn)-anchored, hash-verified, install-before-activate; reuse snapshot_transfer_s pattern), but it is an admin-supplied INPUT, not compacted log state. The marker (not pure snapshot) is mandatory because it gives config the shared (term,lsn) reconcile path; a snapshot anchored at an lsn with no log entry would recreate a separate config reconcile domain (= today's latent config divergence). committed_config_tag becomes DERIVED from the latest committed marker (single commit index). Controller voting-set (peer add/remove) changes are out of scope here (quorum transition → ha-config-bootstrap). Captured as Requirement 7 in .kiro/specs/ha-term-safety/requirements.md.

Key Learning (2026-06-21) — raft.c is ELECTION-ONLY; log layer is separate & not TLA+-conformant

src/lib/raft/raft.c implements ONLY the election half of raft/raft.tla (Ongaro canonical): currentTerm, votedFor, RequestVote, BecomeLeader, PreVote, UpdateTerm-on-vote-RPCs. It has NO log, NO AppendEntries, NO AdvanceCommitIndex, NO commitIndex. The replicated log = the WAL (persist.c); "AppendEntries" = catch-up streaming/repl.c; "AdvanceCommitIndex" = the commit barrier; conflict reconcile = ha_convergence.c — and NONE of them follow the TLA+ log protocol. Gaps (see .kiro/specs/ha-term-safety/TLA-gap-analysis.md): G1 entry term always 0; G2 replication matches LSN contiguity not prevLogTerm; G3 reconcile truncates by committed-boundary not (term,lsn) conflict; G4 commit is term-blind (no Figure-8 current-term check; own-term noop is partial); G5 standby never adopts leader term from heartbeat/replication (only vote RPCs) → term reuse across restart → two leaders/term; G6 vote freshness is LSN-only not (lastLogTerm,lastLogIndex). G5 confirmed live: rejoined ctrl-1 reported term:2 under a term-3 primary. These compose into bug-851. Config is a THIRD domain entirely outside both (committed_config_tag, no term).

Decision Log (2026-06-21) — make src/lib/raft strictly implement raft.tla (library owns the log)

User chose to EXTEND the raft library to strictly follow raft/raft.tla, using the verified spec as a conformance oracle, instead of patching the controller's hand-rolled log protocol. Scaffolded via roadmap + /kiro-spec-batch as the "Raft TLA+ Conformance" series (3 specs, all generated+auto-approved 2026-06-21, cross-spec review PASSED):

• raft-log-replication (8 reqs/24 tasks): library owns log; per-entry term (G1), AppendEntries prevLogTerm match (G2), (term,index) conflict truncation (G3), UpdateTerm on EVERY RPC (G5). New callbacks: on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from. Entry pts: raft_log_append, raft_on_append_entries_request/reply, raft_send_append_entries. State: log_len, next_index[], match_index[]. Index 1-based = WAL LSN.
• raft-commit-safety (6 reqs/18 tasks): AdvanceCommitIndex Figure-8 current-term rule (G4); (lastLogTerm,lastLogIndex) freshness (G6). New: raft_advance_commit, on_commit_advanced(commit_index) [TRIGGERS the existing barrier, never replaces], commit_index state. raft_on_vote_request/raft_on_pre_vote_request gain (cand_last_log_term, cand_last_log_index); library now owns logOk (locus moved from caller/dispatcher → library; supersedes raft.h:246-248 doc).
• raft-sim-conformance (8 reqs/23 tasks): extend raft_sim.c into in-process oracle driving raft.tla Next-actions, asserting the 4 safety invariants; negative test via compile-time RAFT_SIM_BREAK_* switches (test build only). Would have caught bug-851. Pure library core (no I/O/threads) + controller callback adapters keeps it simulator-verifiable. Preserve as policy: PreVote, priority leadership, sync commit-barrier + persist-before-ACK durability, controller-never-self-terminates. INTENTIONAL safe deviation from raft.tla: conflict branch does truncate-THEN-append in one handler (raft.tla is truncate-only) — documented. ha-term-safety reframed = controller wiring + config R7/R8 + live regression; its R1-R6 realized by the library specs. Implement order: raft-library(done) → log-replication → commit-safety → sim-conformance, then ha-term-safety.

Key Learnings (raft-log-replication, 2026-06-21)

• Library build/test: cd src && nix develop --command bash tools/xce-build.sh lib does a CLEAN ASAN build (rm -rf + meson setup). For fast TDD inner loop reuse the dir: nix develop --command bash -c 'meson compile -C xce.build xcelib_test && meson test -C xce.build "libxce test" --print-errorlogs'. test binary: ./xce.build/lib/xcelib_test (Unity, runs all). raft tests live in testsuite/test_raft.c + raft_sim.c, registered in lib/meson.build.
• src/lib/raft is a PURE library: stores no entry bytes, no commit index, no transport. All side effects via callbacks. Index is 1-based == WAL LSN.
• Simulator gotcha: buffered AE RPCs must capture the leader's term at SEND time (raft_sim_ae_t.leader_term), NOT re-read live state.term at delivery — else reorder/stale-churn tests inject the wrong term and defeat UpdateTerm/stale-drop (G5).
• Adapter invariant: never deliver entry_term==0 with entries_len==1 (already-have branch would false-match; correct leader stamps term>=1).

Key Learnings (ha-term-safety task 2.4, 2026-06-22)

• The controller commit barrier (commit.c) is SELF-DERIVING: majority_candidate_locked computes committed_lsn autonomously from registry watermarks (median) capped at persist_durable_lsn. The own-term writable_lsn gate is ONLY a WAIT-completion predicate, NOT a cap on committed_lsn (so it can't deadlock a commit source).
• Task 2.4 makes the raft library's current-term-gated commit_index the COMMIT SOURCE via a NEW commit_engine_t.raft_commit_index ceiling + commit_set_raft_commit_index() setter; on_commit_advanced callback (ha.c) feeds it into &ha->raft_c->commit. committed_lsn = min(durable-majority, raft_commit_index). Own-term gate RETAINED (design lines 193/201/228) as complementary durability gate.
• FEATURE GATE: raft_commit_index==0 means INERT (library source not yet wired by tasks 3.x) → barrier keeps autonomous durable-majority behavior. This is what stops 2.4 from regressing the whole test_commit_engine suite (which sets no raft index). The ceiling only engages at commit_index>=1.
• Unit-test runner: ./xce.build/controller/controller_unit_test -t <group> [-t <group2>] runs only named groups (argp --run/-t). Far faster than the full binary which times out (bug-924). Full clean build still nix develop --command bash tools/xce-build.sh controller.

Key Learning (2026-06-24, bug-1060) — a new leader MUST NEVER truncate its own log (Raft §5.4 Leader Completeness)

• The post-election safe_committed_lsn (median of granted-voter LSNs) is a valid commit LOWER-BOUND and the promotion-gate HWM, but it is NOT a safe WAL-truncation cutoff: it can sit BELOW entries the new leader itself holds durably. Those entries were acked by this node as a standby ⇒ committed by the prior leader. raft_cb_become_leader calling persist_truncate_wal_above(safe_lsn) destroyed them (granted {self=47, peer=44} → median 44 cut the leader's own committed 45..47), and persist_truncate_wal_above ALSO sets committed_lsn=cutoff (a commitIndex DECREASE) + clamps durable down. The leadership NOP then landed at LSN 45 (the rewound tip) under term+1, diverging vs peers still holding the old job at 45 → contradictory freshness denials (one node fresher by index, the other by term) → permanent prevote deadlock + committed-job loss.
• FIX shape: clamp the truncation cutoff to max(safe_lsn, persist_durable_floor) via pure helper ha_promotion_truncate_floor so truncate is a no-op on the leader's durable log. Do NOT clamp safe_lsn itself inside ha_compute_safe_committed_lsn — promotion_safe_lsn is also the behind-node fence HWM; inflating it to self's tip would make every node trivially pass its own gate. The deadlock was a SYMPTOM; the freshness gate is correct (it refused to elect a leader that would lose committed data). Same class as bug-795/bug-828 ("new leader truncates own committed entries"); bug-828 fixed grant-RECORDING so the median reflects the majority, bug-1060 fixes the remaining case where the median is below the leader's OWN tip.
• persist_truncate_wal_above(cutoff) is a NO-OP when cutoff+1 >= current_lsn (returns early, does NOT touch committed_lsn). Only when it actually cuts does it set committed_lsn=cutoff + clamp durable. So clamping the cutoff to >= durable tip makes it inert on a recovered leader (current_lsn == durable_tip+1).

Key Learning (2026-06-24, bug-1067) — AppendEntries has TWO terms; never conflate them

• Raft's AppendEntries carries (a) the HEADER/leader term — term arg of raft_on_append_entries_request, used for UpdateTerm + the stale-reject if term < s->term → ae_reject_stale — and (b) each ENTRY's STORED term (entry_term arg), used for logOk + on_persist_entry. The XCE library raft_on_append_entries_request already takes them as SEPARATE args; the controller adapter ha_consume_append_entries was setting ae_term = entry_term (the stored term). Correct for a LIVE append (leader appends at its current term ⇒ equal), but for CATCH-UP of a HISTORICAL entry written by a PRIOR leader, entry_term < current leader term ⇒ stale-reject of every historical entry ⇒ cross-term catch-up wedges forever (standby stuck, streamer retransmits). FIX: carry the leader's current term out-of-band per entry (wal_catchup_entry_t.leader_current_term, wire WalCatchupEntry.leaderCurrentTerm @8), stamped by ha_stream_wal_to_standby = ha->term, used as ae_term with fallback to entry_term when 0.
• DO NOT try to fix this receiver-side by using s->term as the AE term: a genuinely STALE leader (leader term < s->term) could then push divergent entries through (the stale-check is the ONLY thing rejecting it). The leader's current term MUST travel on the wire and be checked. Test both: cross-term ACCEPT + stale-leader REJECT.
• wal_catchup_entry_t is HAND-DEFINED in src/include/typemap.h, NOT generated — but capnp compile GENERATES encode/decode (controller.capnp.c) AGAINST it via $C.mapname. So adding a schema field requires editing BOTH controller.capnp (wire) AND typemap.h (the C struct) or the build fails "no member". All scalar mapname fields auto-decode (request.c xce_decode only hand-extracts the Data payload); the leader term rides on each sub-entry and reaches ha_consume via ae.entries[0]=e by reference (no extra plumbing through ha_apply_term_entry_via_library).
• ha_stream_wal_to_standby has MANY callers incl. several in test_config_repl.c (2589/2630/2664/2740) and test_coldstart_lsn0.c (339/387) — grep ALL of them before changing the signature, not just the production caller.

Do-Not-Repeat (2026-06-21)

• Don't trust a green "baseline" reading blindly: the lib suite was actually red on HEAD via a stale test_config.c assertion (DEFAULT_SYNC_TIMEOUT_MS 50 vs committed 500 from commit ca1d9ae). Verify with the exact meson test command, not assumptions.

Do-Not-Repeat (2026-06-22, task 2.4)

• When the commit source is a feature-gated ceiling that is INERT at 0, a test that ACKs a durable majority BEFORE feeding the ceiling will commit autonomously to the durable floor, and the ceiling (monotonic) can never regress committed below it. Establish the raft_commit_index/on_commit_advanced ceiling FIRST (production order: the ACK path drives raft_advance_commit which fires on_commit_advanced), THEN drive the ACK. Initial ACK-then-ceiling ordering cost 3 test failures.
• Do NOT make raft_commit_index an unconditional ceiling: capping at 0 when unset breaks every autonomous-barrier test (test_commit_engine: 25/38 failed). Gate it with != 0.

Key Learnings (appended 2026-06-22, ha-term-safety 4.3 r2)

• Config MARKER (WAL stream) and config BYTES (out-of-band OPCode_replConfigPush) arrive on INDEPENDENT transports. config_committed_marker only returns a marker once it is in the committed prefix (LSN <= committed_lsn); an uncommitted marker is ignored. This creates a push-before-commit window where saved-but-unverified bytes mismatch the older committed marker.
• conf_save_config bumps myconf->config_version_tag (the advertised tag) WITHOUT decoding/activating (cluster_config unchanged). Only load_primary_config decodes into mc->cluster_config. So the activation witness in tests is the cluster_config POINTER (load_primary_config frees+reallocates it), NOT config_version_tag.
• In the single-member commit fixture (commit_engine_init n_peers=1), majority_candidate_locked returns persist_durable_lsn => committed_lsn auto-advances to the durable WAL tip on any commit re-eval (incl. conf_save_config). To hold a flushed marker "uncommitted" in a test, SAVE first then atomic_store committed_lsn back below the marker LSN (models a standby whose committed prefix lags its local WAL tip).
• Off-iomgr scheduling pattern: ha_convergence.c is the canonical example — delay_task_ms(&c->delaymgr, 1, 1, entry, ctx) enqueues (ordq_put only, non-blocking) and the delaymgr worker runs the body off the iomgr ACK thread. Tests drive the queued task by calling its entry fn directly (the fixture leaves delaymgr unstarted). Guard scheduling on c->delaymgr.running to avoid leaking tasks into an undrained queue in the unit-test fixture.

Key Learnings (2026-06-23, /xce-test-ha HA soak — 3 failover bugs)

• (bug-854 dead-first residual) The client client/xcecli_start initial connect was hardcoded to endpoint_connect(masters[0]); if masters[0] is a KILLED node (connection-refused), the whole session fails BEFORE the failover loop (which only runs on an already-connected ctx) can rotate. FIX: loop xcecli_start over ALL masters, re-init the endpoint between attempts (endpoint_fini + endpoint_init + endpoint_set_peer + rebind ctx.iomgr — same as xce_ctx_reconnect), set failover.cur_master to the connected index. Verified: dead-first masters list discovers primary in 46ms (was: hang). Distinct from the earlier bug-854 fix (xcecli_ctrlstatus_primary force-rotation) which only covered the all-live standby-first case.
• (bug-1042 rejoin never converges) ha_init_raft (myapp.c, BEFORE election) sources the raft library initial_log_len from persist_durable_floor — but on a standby-REJOIN boot persist_recover runs LATER (cluster_init_phase2). So a restarted node's library comes up log_len=0 and its logOk rejects EVERY AppendEntries (prevLogIndex>=1 vs empty log) → permanently stuck non-converging standby; primary logs "standby[i] lsn=N < committed_lsn=M -> stream WAL batches" forever. FIX: ha_sync_raft_log_from_recovery(ha,p) after persist_recover (cluster.c), raising raft.log_len/commit_index/term to recovered values (monotonic, clamp commit<=log_len). Mirrors bug-848's applied_lsn sync. Diagnostic tell: ae_reject_logmismatch at prevIdx==recovered_floor on the rejoined node.
• (bug-1043 catch-up apply gap) The term-safe library catch-up route (ha_apply_term_entry_via_library → ha_consume_append_entries → on_persist_entry → persist_wal_write_replicated) writes the WAL but NEVER calls apply_wal_entry/restore_job — so a member that catches up the missing suffix by REPLICATION holds entries on disk but NOT in the queryable job table. On promotion joblist/expect-job miss them → looks like committed-job loss but walcompare is byte-identical (it's an apply gap, not data loss). The LEGACY ha_apply_single_entry path called apply_wal_entry after persist; the term-safe route (ha-term-safety 3.4) dropped it. FIX: re-add apply_wal_entry on the accepted path in ha_apply_term_entry_via_library.
• (HA soak harness design) /xce-test-ha convergence gate must NOT key on "standby[i] converged" log (only re-emitted for a standby that was NOT-converged under the current primary term — a standby restarted already-caught-up never logs it). Use "no NEW 'stream WAL batches' line on the primary for ~6s" == all standbys converged (HA_REPL_CONVERGED is a silent no-op). Detect new-primary election via the INFO line "HA FSM (raft): → CTRL_PRIMARY (term=N)" with term STRICTLY > pre-kill term (user guidance). Only SINGLE-failover is guaranteed — the soak MUST wait full 3-of-3 reconvergence before the next kill, else it tests an unsupported double-fault window and reports false loss.

Do-Not-Repeat (2026-06-23, shell portability)

• This deploy env's realpath is BUSYBOX (/sbin/realpath): it does NOT support -m (treats it as a filename, errors to stderr, and on a relative path whose parent exists still prints a resolved line to stdout → with || echo "$TOP" you get MULTI-LINE garbage in the var). readlink -f also fails on nonexistent paths here. Don't rely on realpath -m/readlink -f for path normalization in scripts that run under nix/this host. Use pure-bash absolutization instead: case "$P" in /*) : ;; *) P="$PWD/$P" ;; esac. (Caught while adding the [dir] arg to run-ha-soak.sh.)

Key Learnings (2026-06-23, /xce-test-ha [dir] arg)

• run-ha-soak.sh deploy dir now resolves: positional $1 > $XCE_HA_TOP env > default /home/claude/tmp/xce-ha-deploy. The skill maps /xce-test-ha [rounds] [dir] → rounds to XCE_HA_ROUNDS env, dir to the positional arg. Because the deploy is rm -rf-destructive and the dir is now caller-chosen, a sentinel .xce-ha-deploy marker is written at deploy time and a guard refuses to wipe a non-empty dir lacking it (fail-loud, protects a typo'd home/source path). Empty/nonexistent dir is always safe.

Key Learnings (2026-06-23, bug-845 WAL double-append fixed)

• The hook system (src/lib/hook.c) had NO dedup: xce_add_hook appended a new slot for a repeat (fn,data); xce_run_hooks fires ALL slots; xce_disable_hook NULLs only ONE. So registering the same hook twice = fires twice + survives one disable. Every WAL_JOB_SUBMIT was double-appended (byte-identical, adjacent LSN) because persist_start() (boot, unconditional) AND ha_register_persist_hooks() (PRIMARY promotion) both register persist_job_hook_fun with the same persist_t. The hooks_registered guard only stops ha_register from re-adding ITSELF — it's blind to persist_start's registration.
• FIX = make xce_add_hook idempotent on exact (fn,data): re-adding an active (fn,data) is a no-op; same fn + DISTINCT data still creates a 2nd slot (legitimate multi-closure, preserved by test). Chosen over "remove persist_start's registration" because MANY controller unit tests (test_e2e_checkpoint, test_ha_role_lifecycle, test_commit_barrier_reply:387 "persist_start registers the production hooks") DEPEND on persist_start registering the hooks and do NOT call ha_register — removing it would break that contract.
• ack gating: handle_jobsubmit (job.c) acks the client iff xce_run_hooks(AFTER_EVENT JOB_NEW) returns 0, and xce_run_hooks returns the LAST hook's rc. With the double-register the ack was (accidentally) gated on the 2nd append's repl_wait_majority_ack (LSN+1) — an extra majority round-trip that MASKED a latent failover durability race. Single-append (correct) removed that margin: one post-fix soak then hit the KNOWN intermittent committed-job-loss (bug-795/828 class — LSN-N cross-node divergence: killed primary had the entry, the 2 survivors did not). Confirmed pre-existing/timing (baseline also passes-or-fails on timing; 3/3 post-fix soaks then passed), NOT caused by the dedup. Validation recipe for HA soak: rebuild controller/xce_controller+xce_test, then run .claude/skills/xce-test-ha/run-ha-soak.sh; judge by waldump single-vs-double entries, not the leaky ASAN exit code.

Do-Not-Repeat (2026-06-23, isolate before blaming your own change)

• When a soak regresses right after your edit, DO NOT assume your edit caused it: the working tree here had pre-existing UNCOMMITTED HA changes (cluster.c/ha.c/persist.c/config_repl.c) that the stale prebuilt binaries (dated 15:01) had NEVER compiled. The first rebuild pulls BOTH your change AND those deltas. Isolate by git stash push -- <your files>, rebuild, re-run. Here it proved the WAL-dup fix was clean and the job-loss was the known intermittent durability race.

Key Learnings (2026-06-23, /xce-test-ha leadership coverage — ctrl-3 never primary)

• WHY ctrl-3 never led: HA election timeout is PRIORITY-STAGGERED. my_priority = index in --peers list (ctrl-1=0, ctrl-2=1, ctrl-3=2; lower = higher priority), and the election delay = my_priority * election_priority_step_ms (ha.h:138, default 500ms → p0 immediate / p1 +500 / p2 +1000). The soak only ever kills ONE node (single-failover), so the surviving pair always contains a lower-index node that out-races ctrl-3. Confirmed: ctrl-3 had 0 CTRL_PRIMARY announcements but DID start 6 elections (it tries, always loses the race). No leadership-transfer/stepdown command exists in xce_test or ha.c, so leadership can't be handed over directly.
• HOW to force a specific (lowest-priority) node T to lead, DETERMINISTICALLY and loss-free (run-ha-soak.sh force_elect_node): exploit the RequestVote freshness gate (a behind node cannot win). P=current primary, S=third node. (1) kill S so it stops receiving entries; (2) submit a batch → commits on {P,T}, S now provably behind; (3) kill P → only T alive briefly (T holds ALL commits — they were on {P,T} — so nothing is lost); (4) restart the STILL-BEHIND S → quorum {T fresh, S stale}; S cannot catch up while there is no primary, so it stays freshness-fenced for the whole election → T wins deterministically (NOT by timing); (5) verify no loss, restart P, reconverge. KEY INSIGHT: a behind node can't catch up without a primary, and there's no primary during the election, so the spoiler is guaranteed fenced — this removes all timing-race flakiness.
• The soak now tracks SEEN_PRIMARY per node and the verdict ASSERTS all 3 led (PASS line: "all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)"); a node that never leads is a FAIL. force_elect_node is generic (works for any never-led node), run post-loop for each missing node after restoring 3-of-3.