fix bug-1060 bug-1067

.wolf/anatomy.md

@@ -1,10 +1,21 @@
 # anatomy.md
 
-> Auto-maintained by OpenWolf. Last scanned: 2026-06-23T04:47:56.802Z
-> Files: 551 tracked | Anatomy hits: 0 | Misses: 0
+> Auto-maintained by OpenWolf. Last scanned: 2026-06-24T08:33:10.048Z
+> Files: 570 tracked | Anatomy hits: 0 | Misses: 0
+
+## ../../../../../../tmp/xce-ha-verify/
+
+- `test.sh` (~291 tok)
+
+## ../../../../../claude/.claude/plans/
+
+- `piped-mixing-glade.md` — Fix bug-854 (client primary-discovery) and bug-1028 (post-failover availability) (~1921 tok)
+- `task-notification-task-id-bkkk2llpr-tas-zany-locket.md` — XCE HA Failover Soak — Result (~343 tok)
 
 ## ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/
 
+- `project_bug1060_leader_completeness.md` (~538 tok)
+- `project_bug1067_ae_two_terms.md` (~716 tok)
 - `project_raft_commit_safety_done.md` (~603 tok)
 - `project_raft_log_replication_done.md` (~507 tok)
 
@@ -19,6 +30,8 @@
 - `package-lock.json` — npm lock file (~1039 tok)
 - `package.json` — Node.js package manifest (~1 tok)
 - `README.md` — Project documentation (~612 tok)
+- `repro_joblist.sh` — Restart the kept failover-e2e cluster and query joblist via the client to see (~201 tok)
+- `repro854.sh` — Reproduce bug-854 client discovery against the kept failover-e2e cluster. (~422 tok)
 - `uninstall-code-review-graph.sh` — Fully uninstall `code-review-graph` (the Python/uv MCP tool). (~672 tok)
 
 ## .claude/
@@ -203,6 +216,11 @@
 
 - `SKILL.md` — XCE HA Cluster Deployment (~1345 tok)
 
+## .claude/skills/xce-test-ha/
+
+- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~6047 tok)
+- `SKILL.md` — XCE HA Failover Soak Test (~2511 tok)
+
 ## .codegraph/
 
 - `.gitignore` — Git ignore rules (~47 tok)
@@ -764,13 +782,14 @@
 
 ## src/controller/
 
-- `cluster.c` — include "config.h" (~16351 tok)
+- `CLAUDE.md` — XCE Controller (~3309 tok)
+- `cluster.c` — include "config.h" (~16451 tok)
 - `commit.c` — Declares persist_t (~5043 tok)
-- `config_repl.c` — define _GNU_SOURCE (~24927 tok)
-- `config_repl.h` — pragma once (~5622 tok)
+- `config_repl.c` — define _GNU_SOURCE (~25407 tok)
+- `config_repl.h` — pragma once (~6008 tok)
 - `ha_convergence.c` — Declares char (~4868 tok)
-- `ha.c` — define _GNU_SOURCE (~77256 tok)
-- `ha.h` — ifndef HA_H (~21010 tok)
+- `ha.c` — define _GNU_SOURCE (~78897 tok)
+- `ha.h` — ifndef HA_H (~21515 tok)
 - `meson.build` (~1352 tok)
 - `myapp.c` — define _GNU_SOURCE (~4691 tok)
 - `persist.c` — define _GNU_SOURCE (~33172 tok)
@@ -784,21 +803,23 @@
 
 - `clean_failover_preservation.sh` — clean_failover_preservation.sh — ha-term-safety task 5.2 live integration test. (~4972 tok)
 - `controller_test.c` — include <argp.h> (~1623 tok)
+- `test_coldstart_lsn0.c` — Declares char (~4405 tok)
 - `test_commit_barrier_rollout.c` — Declares explicitly (~19506 tok)
-- `test_config_repl.c` — define _GNU_SOURCE (~34816 tok)
+- `test_config_repl.c` — define _GNU_SOURCE (~35905 tok)
 - `test_demotion_convergence.c` — define _GNU_SOURCE (~9602 tok)
 - `test_e2e_capnp.c` — include <stdint.h> (~2760 tok)
-- `test_ha_election.c` — Declares char (~12140 tok)
+- `test_ha_election.c` — Declares char (~12969 tok)
 - `test_ha_prevote.c` — Declares char (~11053 tok)
 - `test_ha_vote_gate.c` — define _GNU_SOURCE (~13251 tok)
+- `test_persist_recovery.c` — define _GNU_SOURCE (~3654 tok)
 - `test_repl_apply.c` — include <dirent.h> (~5744 tok)
 - `test_repl_unit.c` — include <assert.h> (~6231 tok)
-- `test_term_safety_adapter.c` — define _GNU_SOURCE (~25125 tok)
+- `test_term_safety_adapter.c` — define _GNU_SOURCE (~26786 tok)
 - `test_walcompare.c` — define _GNU_SOURCE (~3795 tok)
 - `test.c` — include <argp.h> (~27479 tok)
 - `walcompare.c` — define _GNU_SOURCE (~2485 tok)
 - `walcompare.h` — pragma once (~839 tok)
-- `waldump.c` — define _GNU_SOURCE (~4586 tok)
+- `waldump.c` — define _GNU_SOURCE (~4758 tok)
 - `waldump.h` — pragma once (~1006 tok)
 
 ## src/controller/testsuite/clean_failover_preservation.sh (~330 lines, ~3k tokens)
@@ -806,14 +827,19 @@
 
 ## src/include/
 
+- `netio.h` — if !defined(_NETIO_H_) (~1293 tok)
 - `persist.h` — pragma once (~9539 tok)
 - `repl.h` — pragma once (~2163 tok)
-- `typemap.h` — if !defined(_TYPEMAP_H_) (~4480 tok)
+- `typemap.h` — if !defined(_TYPEMAP_H_) (~4632 tok)
 - `wal_types.h` — pragma once (~1400 tok)
 
 ## src/lib/
 
-- `request.c` — include <errno.h> (~15710 tok)
+- `client.c` — include <errno.h> (~6440 tok)
+- `hook.c` — include <errno.h> (~1197 tok)
+- `netio.c` — include <arpa/inet.h> (~6688 tok)
+- `request.c` — include <errno.h> (~15862 tok)
+- `xcectx.c` — include <errno.h> (~3042 tok)
 
 ## src/lib/raft/
 
@@ -824,8 +850,10 @@
 ## src/lib/testsuite/
 
 - `test_config.c` — include <stdio.h> (~490 tok)
+- `test_hooks.c` — include <errno.h> (~1416 tok)
 - `test_raft.c` — include <errno.h> (~68779 tok)
+- `test_wal_wire.c` — include <errno.h> (~2970 tok)
 
 ## src/schema/
 
-- `controller.capnp` — Declares OPCode (~8927 tok)
+- `controller.capnp` — Declares OPCode (~9156 tok)

.wolf/cerebrum.md

@@ -9,6 +9,8 @@
 <!-- How the user likes things done. Code style, tools, patterns, communication. -->
 
 ## Key Learnings
+- (2026-06-23, bug-1028 FIXED) There MUST be exactly ONE authoritative source for a WAL entry's Raft term. The leader's library send path (`ha_read_wal_entry_at`, ha.c:1472-1476) and the standby's `logOk`/`on_get_entry_term` both source terms from `persist_entry_term` (in-memory index, legacy-0→WAL_LEGACY_TERM). The heartbeat catch-up streamer (`ha_stream_wal_to_standby`, config_repl.c) — the third consumer — was sourcing terms from raw on-disk `hdr.term` (0 for legacy entries). A standby's `prev_log_term=1` (normalised) ≠ streamer's `prev_log_term=0` (raw) → `logOk` rejects the new-term NOOP forever → `applied_lsn` stuck → primary re-streams every heartbeat cycle. Fix: all three `hdr.term` reads in the streamer replaced with `persist_entry_term(&persist, hdr.lsn)`; function gained a `const persist_t*` parameter. This is the pattern: every source of the per-LSN (term,index) tuple MUST go through `persist_entry_term`; NEVER read `hdr.term` raw from disk — it is not normalised.
+- (2026-06-23, bug-854 FIXED) The `XCECLI_FAILOVER_CALL` retry loop classifies a status-0 reply from ANY reachable node as SUCCESS and returns — it only rotates on failure. But ctrlstatus is answerable in every role, so a standby always answers 0, making `xcecli_ctrlstatus` pin to the first reachable node forever. The outer `xcecli_ctrlstatus_primary` loop captures a redirect hint from the non-authoritative reply but never forces rotation to the next master — so the configured master list is never traversed. Fix: `xcecli_ctrlstatus_primary` now calls `xce_failover_rotate(ctx)` + `xcecli_ctx_recover(ctx)` under `failover.lock` on every non-authoritative reply; the floor of `max_attempts` was raised to `nmasters+1` to cover the full ring. General pattern: any outer retry loop that queries a role-agnostic status must **force endpoint rotation** on a non-authoritative reply — do not rely on the inner failover wrapper, because status-0 preempts it.
 - (2026-06-22, ha-term-safety task 3.1) The single primary-append seam is `ha_primary_log_append(cluster_t*,type,payload,len,&out_lsn)` (ha.c, decl ha.h ~333): packs the `raft_entry_blob_t` envelope and calls `raft_log_append(&c->ha.raft,...)` so the LIBRARY stamps `s->term` + assigns LSN + fires `on_persist_entry`. CRITICAL: `raft_log_append` returns `-EPERM` unless `s->role==RAFT_LEADER` — and `ha_do_promote`/the non-raft `ha_promote_to_primary` road sets `ha->role=CTRL_PRIMARY` but NOT `ha->raft.role` (only the raft election road via on_become_leader sets RAFT_LEADER). So `ha_primary_log_append` MUST have a `-EPERM`→legacy `persist_wal_async` fallback (stamping `ha->term` via persist_set_term first) or init-time/non-raft/engine-less promotion breaks. Retired the hand-rolled bypass by routing persist.c `wal_hook_append_classified` (RETAIN job/queue/worker hooks) + `persist_wal_sync_append` (WAL_JOB_SUBMIT WAIT) through `ha_primary_log_append` GUARDED on `p->cluster != NULL` — standalone persist (lib-level tests, `persist_init(p,&cfg,NULL)`) keeps the direct persist_wal_async path. `p->cluster` is the back-ref (persist.h, set in persist_init); ha.h is reachable in persist.c via `xce/cluster_def.h`. Co-tracking invariant `p->current_lsn == s->log_len + 1` holds because raft_init (2.5) seeds log_len from durable floor and every primary append advances both by 1. To make a controller TEST drive raft_log_append: `ha_init_raft` then set `c->ha.raft.role=RAFT_LEADER` + `c->ha.raft.term=T` directly (the raft-sim convention). on_persist_entry calls persist_wal_async (NOT the hooks) so no recursion. `ha_arm_own_term_gate` is static → test seam `ha_arm_own_term_gate_for_test`. Validation: pre-existing `test_wal_catchup_*` (7, bug-924 overlayfs) fail IDENTICALLY on baseline (git stash + rebuild proves it); every other group 0 fail.
 - (2026-06-20, waldump tooling) The `xce_test` binary (`src/controller/meson.build`) builds from `testsuite/test.c` ONLY + `libxce_dep` — it does NOT link `persist.c`/`job.c` (those are in the `controller_unit_test` target). The investigation doc's claim that xce_test "links persist.c/job.c" was WRONG. BUT capnp `read_Job` is generated into `controller.capnp.c` which IS compiled into `libxce`, so it's reachable transitively. To USE capnp/zlib symbols from test.c you must add `c_capnproto_dep` + `zlib_dep` to `xce_test_deps` (static-lib deps don't auto-propagate through `declare_dependency(link_with: libxce)`). For a zero-alloc jobid decode read `struct Job.jobid` directly after `read_Job` — skip the allocating `decode_Job`. WAL CRC reader template = `persist_test_durable_lsn_on_disk` (persist.c ~888): seg files `seg-%016llx.wal` under `<node>/data/wal`, 40B `wal_entry_hdr_t`, crc32 over `WAL_HDR_CRC_SIZE`(=32) header bytes + payload, prev_crc chain, all-zero hdr = clean EOF (pre-alloc). Offline `waldump` lives in `testsuite/waldump.{c,h}` + `cmd_waldump` in test.c top_commands.
 - (2026-06-18, ha-commit-durability task 5.2 VALIDATION) Deterministic E2E on the FIXED build = GREEN, stable 3/3 runs (9/9 probes PASS), in-window primary fdatasync=2 on EVERY probe (vs 0/RED on the unfixed 1.3 build) — the RED→GREEN flip is the deterministic acceptance and is confirmed. Run via `nix develop ./src --command bash -c 'bash src/tools/test-commit-durability.sh --deterministic <deploy>'` background+poll worktree log; cold election ~15s, 3 probes, exit 0. The ROLLING SIGKILL SOAK is host-verification-needed in-sandbox: rounds 1-2 PASS (jobs survive failover), then round 3 onward "no new primary elected within 40s" and every "DATA LOSS" report is a query against a cluster with NO live primary (same 5 jobs frozen, never re-acked). ROOT CAUSE of the collapse is the durability fix's OWN promotion fence working AS DESIGNED: ctrl-1 won PreVote for term=5, computed safe_committed_lsn=13, then `persist_recover_fatal_if_unrecoverable` FATAL-terminated ("recovery-truncated log ... no healthy peer reachable ... durable_floor_lsn=13 < hwm=13 ... terminating rather than serving divergent state, Req 2.4") — refusing to let a structurally-truncated node lead and overwrite the good timeline (exactly the bug-661 failure mode the fix prevents). Under aggressive rolling-kill+immediate-restart the fence repeatedly self-terminates recovering nodes → cluster drops below quorum → liveness collapse, NOT durable loss. PROOF it's not loss: committed entries still on disk after the run (ctrl-3 WAL has seg-1 + seg-15; ctrl-2 seg-1) and 0 live controllers at end. So the soak's exit=1 / "DATA LOSS" verdict is a HARNESS ARTIFACT (it equates empty-joblist-with-no-primary to loss) — the soak harness conflates election-stall liveness failure with durability loss. The deterministic flip is the must-have and passed; the live multi-round soak needs a real host (slower election timing, no sandbox reaping) to distinguish fence-driven safety self-termination from a true regression. Did NOT patch production (5.2 is validation-only).
@@ -88,6 +90,8 @@
 - (2026-06-22, ha-term-safety task 3.3 remediation) Do NOT run `clang-format -i` on whole XCE controller source files (ha.c/ha.h/test files): these files are NOT clang-format-clean against the repo's config, so a full reformat rewrites ~1700 lines and BURIES the targeted change (and violates a "preserve the existing diff" instruction). I then made it worse with `git checkout HEAD -- <files>` to undo the format — but the uncommitted task-3.3 work lived ONLY in the working tree, so that wiped it (no stash, unrecoverable from git). Recovery worked only because the full original content was already in my context. Lesson: (1) only format the exact changed regions or skip formatting; (2) NEVER `git checkout HEAD -- <file>` to discard formatting noise when the file also holds uncommitted feature work — instead revert just the format hunks, or stash first. The reflog showed branch HEAD was already at task-3.2 (754e586), and the initial-context "recent commit 2c5d298" was a stale snapshot — the uncommitted 3.3 diff sat on top of 3.2.
 - (2026-06-22, ACK peer-index, bug-960) The walAck branch of `repl_ack_decode_and_dispatch` (ha.c) fires for TWO endpoint types whose stored `peer_idx` means DIFFERENT things: promoted-vote eps (`vote_ep_ctx_t`, `ctx!=NULL`) carry a self-EXCLUDED *votes* index (relative); heartbeat standby eps (`standby_ep_ctx_t`, `ctx==NULL`) carry an ALREADY-ABSOLUTE candidate index. The authoritative branch is ha.c:6187-6189 (`sctx->ctx != NULL ? votes_idx : i`). The pre-existing ACK consumer `ha_walack_on_ack(c, vctx->peer_idx, …)` uses `peer_idx` DIRECTLY (no remap) — proof the ACK path's index is already absolute. Any code consuming the ACK `peer_idx` for the library (match_index[]/next_index[], which raft.c writes UNCHECKED) MUST branch on `vctx->ctx != NULL` before remapping, or it double-maps the heartbeat path → wrong-peer false quorum / OOB write. `self_idx == ha->my_priority` (raft_cfg.self_idx = my_priority everywhere).
 
+- (2026-06-24, bug-1060 — RECURRENCE of the 2026-06-22 entry above) Ran `clang-format -i` on whole ha.c/ha.h/test_ha_election.c AGAIN → spurious whole-file reformat (3268/969/635 lines) that buried a ~180-line fix. Confirmed spurious by checking the repo's HEAD ha.c is NOT idempotent under the nix `clang-format` (a `.clang-format` exists at src/.clang-format but the pinned formatter version differs from whatever produced HEAD). Recovery THIS time succeeded because: (a) I had already printed the full pre-format `git diff ha.c` (5 hunks: bug-1028/1042/1043 pre-existing + my bug-1060) into context; (b) ha.h's only genuine new symbol vs HEAD was `ha_sync_raft_log_from_recovery` (found via `comm` on `grep -oE '\bha_[a-z_]+\b'` of HEAD-vs-current — everything else was pointer-style/comment reflow); (c) test_ha_election.c was NOT in the session-start `M` list so it held only my changes. Then `git checkout HEAD -- <3 files>` + re-applied the logical hunks by Edit. LESSON STANDS: do NOT clang-format whole files here; format only changed regions or skip. The `git checkout HEAD --` rescue is ONLY safe after you have positively accounted for EVERY uncommitted change in those files (capture the pre-format `git diff` FIRST, and confirm which files had pre-existing work via the session-start `git status`).
+
 <!-- Mistakes made and corrected. Each entry prevents the same mistake recurring. -->
 <!-- Format: [YYYY-MM-DD] Description of what went wrong and what to do instead. -->
 
@@ -245,6 +249,17 @@ Pure library core (no I/O/threads) + controller callback adapters keeps it simul
 - FEATURE GATE: raft_commit_index==0 means INERT (library source not yet wired by tasks 3.x) → barrier keeps autonomous durable-majority behavior. This is what stops 2.4 from regressing the whole test_commit_engine suite (which sets no raft index). The ceiling only engages at commit_index>=1.
 - Unit-test runner: `./xce.build/controller/controller_unit_test -t <group> [-t <group2>]` runs only named groups (argp `--run`/`-t`). Far faster than the full binary which times out (bug-924). Full clean build still `nix develop --command bash tools/xce-build.sh controller`.
 
+## Key Learning (2026-06-24, bug-1060) — a new leader MUST NEVER truncate its own log (Raft §5.4 Leader Completeness)
+- The post-election `safe_committed_lsn` (median of granted-voter LSNs) is a valid commit LOWER-BOUND and the promotion-gate HWM, but it is NOT a safe WAL-truncation cutoff: it can sit BELOW entries the new leader itself holds durably. Those entries were acked by this node as a standby ⇒ committed by the prior leader. `raft_cb_become_leader` calling `persist_truncate_wal_above(safe_lsn)` destroyed them (granted `{self=47, peer=44}` → median 44 cut the leader's own committed 45..47), and `persist_truncate_wal_above` ALSO sets `committed_lsn=cutoff` (a commitIndex DECREASE) + clamps durable down. The leadership NOP then landed at LSN 45 (the rewound tip) under term+1, diverging vs peers still holding the old job at 45 → contradictory freshness denials (one node fresher by index, the other by term) → permanent prevote deadlock + committed-job loss.
+- FIX shape: clamp the truncation cutoff to `max(safe_lsn, persist_durable_floor)` via pure helper `ha_promotion_truncate_floor` so truncate is a no-op on the leader's durable log. Do NOT clamp `safe_lsn` itself inside `ha_compute_safe_committed_lsn` — `promotion_safe_lsn` is also the behind-node fence HWM; inflating it to self's tip would make every node trivially pass its own gate. The deadlock was a SYMPTOM; the freshness gate is correct (it refused to elect a leader that would lose committed data). Same class as bug-795/bug-828 ("new leader truncates own committed entries"); bug-828 fixed grant-RECORDING so the median reflects the majority, bug-1060 fixes the remaining case where the median is below the leader's OWN tip.
+- `persist_truncate_wal_above(cutoff)` is a NO-OP when `cutoff+1 >= current_lsn` (returns early, does NOT touch committed_lsn). Only when it actually cuts does it set committed_lsn=cutoff + clamp durable. So clamping the cutoff to >= durable tip makes it inert on a recovered leader (current_lsn == durable_tip+1).
+
+## Key Learning (2026-06-24, bug-1067) — AppendEntries has TWO terms; never conflate them
+- Raft's AppendEntries carries (a) the HEADER/leader term — `term` arg of raft_on_append_entries_request, used for UpdateTerm + the stale-reject `if term < s->term → ae_reject_stale` — and (b) each ENTRY's STORED term (`entry_term` arg), used for logOk + on_persist_entry. The XCE library `raft_on_append_entries_request` already takes them as SEPARATE args; the controller adapter `ha_consume_append_entries` was setting `ae_term = entry_term` (the stored term). Correct for a LIVE append (leader appends at its current term ⇒ equal), but for CATCH-UP of a HISTORICAL entry written by a PRIOR leader, entry_term < current leader term ⇒ stale-reject of every historical entry ⇒ cross-term catch-up wedges forever (standby stuck, streamer retransmits). FIX: carry the leader's current term out-of-band per entry (`wal_catchup_entry_t.leader_current_term`, wire `WalCatchupEntry.leaderCurrentTerm @8`), stamped by `ha_stream_wal_to_standby` = ha->term, used as ae_term with fallback to entry_term when 0.
+- DO NOT try to fix this receiver-side by using `s->term` as the AE term: a genuinely STALE leader (leader term < s->term) could then push divergent entries through (the stale-check is the ONLY thing rejecting it). The leader's current term MUST travel on the wire and be checked. Test both: cross-term ACCEPT + stale-leader REJECT.
+- `wal_catchup_entry_t` is HAND-DEFINED in src/include/typemap.h, NOT generated — but `capnp compile` GENERATES encode/decode (controller.capnp.c) AGAINST it via `$C.mapname`. So adding a schema field requires editing BOTH controller.capnp (wire) AND typemap.h (the C struct) or the build fails "no member". All scalar mapname fields auto-decode (request.c xce_decode only hand-extracts the Data payload); the leader term rides on each sub-entry and reaches ha_consume via ae.entries[0]=e by reference (no extra plumbing through ha_apply_term_entry_via_library).
+- `ha_stream_wal_to_standby` has MANY callers incl. several in test_config_repl.c (2589/2630/2664/2740) and test_coldstart_lsn0.c (339/387) — grep ALL of them before changing the signature, not just the production caller.
+
 ## Do-Not-Repeat (2026-06-21)
 - Don't trust a green "baseline" reading blindly: the lib suite was actually red on HEAD via a stale test_config.c assertion (DEFAULT_SYNC_TIMEOUT_MS 50 vs committed 500 from commit ca1d9ae). Verify with the exact meson test command, not assumptions.
 
@@ -257,3 +272,28 @@ Pure library core (no I/O/threads) + controller callback adapters keeps it simul
 - conf_save_config bumps myconf->config_version_tag (the advertised tag) WITHOUT decoding/activating (cluster_config unchanged). Only load_primary_config decodes into mc->cluster_config. So the activation witness in tests is the cluster_config POINTER (load_primary_config frees+reallocates it), NOT config_version_tag.
 - In the single-member commit fixture (commit_engine_init n_peers=1), majority_candidate_locked returns persist_durable_lsn => committed_lsn auto-advances to the durable WAL tip on any commit re-eval (incl. conf_save_config). To hold a flushed marker "uncommitted" in a test, SAVE first then atomic_store committed_lsn back below the marker LSN (models a standby whose committed prefix lags its local WAL tip).
 - Off-iomgr scheduling pattern: ha_convergence.c is the canonical example — delay_task_ms(&c->delaymgr, 1, 1, entry, ctx) enqueues (ordq_put only, non-blocking) and the delaymgr worker runs the body off the iomgr ACK thread. Tests drive the queued task by calling its entry fn directly (the fixture leaves delaymgr unstarted). Guard scheduling on c->delaymgr.running to avoid leaking tasks into an undrained queue in the unit-test fixture.
+
+## Key Learnings (2026-06-23, /xce-test-ha HA soak — 3 failover bugs)
+- (bug-854 dead-first residual) The client `client`/`xcecli_start` initial connect was hardcoded to `endpoint_connect(masters[0])`; if masters[0] is a KILLED node (connection-refused), the whole session fails BEFORE the failover loop (which only runs on an already-connected ctx) can rotate. FIX: loop `xcecli_start` over ALL masters, re-init the endpoint between attempts (endpoint_fini + endpoint_init + endpoint_set_peer + rebind ctx.iomgr — same as xce_ctx_reconnect), set failover.cur_master to the connected index. Verified: dead-first masters list discovers primary in 46ms (was: hang). Distinct from the earlier bug-854 fix (xcecli_ctrlstatus_primary force-rotation) which only covered the all-live standby-first case.
+- (bug-1042 rejoin never converges) ha_init_raft (myapp.c, BEFORE election) sources the raft library `initial_log_len` from persist_durable_floor — but on a standby-REJOIN boot persist_recover runs LATER (cluster_init_phase2). So a restarted node's library comes up log_len=0 and its logOk rejects EVERY AppendEntries (prevLogIndex>=1 vs empty log) → permanently stuck non-converging standby; primary logs "standby[i] lsn=N < committed_lsn=M -> stream WAL batches" forever. FIX: ha_sync_raft_log_from_recovery(ha,p) after persist_recover (cluster.c), raising raft.log_len/commit_index/term to recovered values (monotonic, clamp commit<=log_len). Mirrors bug-848's applied_lsn sync. Diagnostic tell: `ae_reject_logmismatch` at prevIdx==recovered_floor on the rejoined node.
+- (bug-1043 catch-up apply gap) The term-safe library catch-up route (ha_apply_term_entry_via_library → ha_consume_append_entries → on_persist_entry → persist_wal_write_replicated) writes the WAL but NEVER calls apply_wal_entry/restore_job — so a member that catches up the missing suffix by REPLICATION holds entries on disk but NOT in the queryable job table. On promotion joblist/expect-job miss them → looks like committed-job loss but walcompare is byte-identical (it's an apply gap, not data loss). The LEGACY ha_apply_single_entry path called apply_wal_entry after persist; the term-safe route (ha-term-safety 3.4) dropped it. FIX: re-add apply_wal_entry on the accepted path in ha_apply_term_entry_via_library.
+- (HA soak harness design) /xce-test-ha convergence gate must NOT key on "standby[i] converged" log (only re-emitted for a standby that was NOT-converged under the current primary term — a standby restarted already-caught-up never logs it). Use "no NEW 'stream WAL batches' line on the primary for ~6s" == all standbys converged (HA_REPL_CONVERGED is a silent no-op). Detect new-primary election via the INFO line "HA FSM (raft): → CTRL_PRIMARY (term=N)" with term STRICTLY > pre-kill term (user guidance). Only SINGLE-failover is guaranteed — the soak MUST wait full 3-of-3 reconvergence before the next kill, else it tests an unsupported double-fault window and reports false loss.
+
+## Do-Not-Repeat (2026-06-23, shell portability)
+- This deploy env's `realpath` is BUSYBOX (`/sbin/realpath`): it does NOT support `-m` (treats it as a filename, errors to stderr, and on a relative path whose parent exists still prints a resolved line to stdout → with `|| echo "$TOP"` you get MULTI-LINE garbage in the var). `readlink -f` also fails on nonexistent paths here. Don't rely on `realpath -m`/`readlink -f` for path normalization in scripts that run under nix/this host. Use pure-bash absolutization instead: `case "$P" in /*) : ;; *) P="$PWD/$P" ;; esac`. (Caught while adding the [dir] arg to run-ha-soak.sh.)
+
+## Key Learnings (2026-06-23, /xce-test-ha [dir] arg)
+- run-ha-soak.sh deploy dir now resolves: positional `$1` > `$XCE_HA_TOP` env > default `/home/claude/tmp/xce-ha-deploy`. The skill maps `/xce-test-ha [rounds] [dir]` → rounds to `XCE_HA_ROUNDS` env, dir to the positional arg. Because the deploy is `rm -rf`-destructive and the dir is now caller-chosen, a sentinel `.xce-ha-deploy` marker is written at deploy time and a guard refuses to wipe a non-empty dir lacking it (fail-loud, protects a typo'd home/source path). Empty/nonexistent dir is always safe.
+
+## Key Learnings (2026-06-23, bug-845 WAL double-append fixed)
+- The hook system (src/lib/hook.c) had NO dedup: xce_add_hook appended a new slot for a repeat (fn,data); xce_run_hooks fires ALL slots; xce_disable_hook NULLs only ONE. So registering the same hook twice = fires twice + survives one disable. Every WAL_JOB_SUBMIT was double-appended (byte-identical, adjacent LSN) because persist_start() (boot, unconditional) AND ha_register_persist_hooks() (PRIMARY promotion) both register persist_job_hook_fun with the same persist_t. The hooks_registered guard only stops ha_register from re-adding ITSELF — it's blind to persist_start's registration.
+- FIX = make xce_add_hook idempotent on exact (fn,data): re-adding an active (fn,data) is a no-op; same fn + DISTINCT data still creates a 2nd slot (legitimate multi-closure, preserved by test). Chosen over "remove persist_start's registration" because MANY controller unit tests (test_e2e_checkpoint, test_ha_role_lifecycle, test_commit_barrier_reply:387 "persist_start registers the production hooks") DEPEND on persist_start registering the hooks and do NOT call ha_register — removing it would break that contract.
+- ack gating: handle_jobsubmit (job.c) acks the client iff xce_run_hooks(AFTER_EVENT JOB_NEW) returns 0, and xce_run_hooks returns the LAST hook's rc. With the double-register the ack was (accidentally) gated on the 2nd append's repl_wait_majority_ack (LSN+1) — an extra majority round-trip that MASKED a latent failover durability race. Single-append (correct) removed that margin: one post-fix soak then hit the KNOWN intermittent committed-job-loss (bug-795/828 class — LSN-N cross-node divergence: killed primary had the entry, the 2 survivors did not). Confirmed pre-existing/timing (baseline also passes-or-fails on timing; 3/3 post-fix soaks then passed), NOT caused by the dedup. Validation recipe for HA soak: rebuild controller/xce_controller+xce_test, then run .claude/skills/xce-test-ha/run-ha-soak.sh; judge by waldump single-vs-double entries, not the leaky ASAN exit code.
+
+## Do-Not-Repeat (2026-06-23, isolate before blaming your own change)
+- When a soak regresses right after your edit, DO NOT assume your edit caused it: the working tree here had pre-existing UNCOMMITTED HA changes (cluster.c/ha.c/persist.c/config_repl.c) that the stale prebuilt binaries (dated 15:01) had NEVER compiled. The first rebuild pulls BOTH your change AND those deltas. Isolate by `git stash push -- <your files>`, rebuild, re-run. Here it proved the WAL-dup fix was clean and the job-loss was the known intermittent durability race.
+
+## Key Learnings (2026-06-23, /xce-test-ha leadership coverage — ctrl-3 never primary)
+- WHY ctrl-3 never led: HA election timeout is PRIORITY-STAGGERED. my_priority = index in --peers list (ctrl-1=0, ctrl-2=1, ctrl-3=2; lower = higher priority), and the election delay = my_priority * election_priority_step_ms (ha.h:138, default 500ms → p0 immediate / p1 +500 / p2 +1000). The soak only ever kills ONE node (single-failover), so the surviving pair always contains a lower-index node that out-races ctrl-3. Confirmed: ctrl-3 had 0 CTRL_PRIMARY announcements but DID start 6 elections (it tries, always loses the race). No leadership-transfer/stepdown command exists in xce_test or ha.c, so leadership can't be handed over directly.
+- HOW to force a specific (lowest-priority) node T to lead, DETERMINISTICALLY and loss-free (run-ha-soak.sh force_elect_node): exploit the RequestVote freshness gate (a behind node cannot win). P=current primary, S=third node. (1) kill S so it stops receiving entries; (2) submit a batch → commits on {P,T}, S now provably behind; (3) kill P → only T alive briefly (T holds ALL commits — they were on {P,T} — so nothing is lost); (4) restart the STILL-BEHIND S → quorum {T fresh, S stale}; S cannot catch up while there is no primary, so it stays freshness-fenced for the whole election → T wins deterministically (NOT by timing); (5) verify no loss, restart P, reconverge. KEY INSIGHT: a behind node can't catch up without a primary, and there's no primary during the election, so the spoiler is guaranteed fenced — this removes all timing-race flakiness.
+- The soak now tracks SEEN_PRIMARY per node and the verdict ASSERTS all 3 led (PASS line: "all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)"); a node that never leads is a FAIL. force_elect_node is generic (works for any never-led node), run post-loop for each missing node after restoring 3-of-3.

.wolf/memory.md

@@ -2579,3 +2579,221 @@
 | 21:47 | Edited .kiro/steering/persistence.md | inline fix | ~126 |
 | 21:48 | Session end: 294 writes across 37 files (test_term_safety_adapter.c, commit.h, commit.c, ha.c, ha.h) | 66 reads | ~607176 tok |
 | 00:04 | Session end: 294 writes across 37 files (test_term_safety_adapter.c, commit.h, commit.c, ha.c, ha.h) | 66 reads | ~607176 tok |
+
+## Session: 2026-06-23 00:14
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 00:29 | Created ../../../../../claude/.claude/plans/piped-mixing-glade.md | — | ~2049 |
+| 00:34 | Edited src/controller/config_repl.h | expanded (+10 lines) | ~391 |
+| 00:34 | Edited src/controller/config_repl.h | modified struct() | ~91 |
+| 00:35 | Edited src/controller/config_repl.c | modified ha_stream_wal_to_standby() | ~64 |
+| 00:35 | Edited src/controller/config_repl.c | added error handling | ~285 |
+| 00:35 | Edited src/controller/config_repl.c | modified persist_entry_term() | ~140 |
+| 00:36 | Edited src/controller/config_repl.c | modified persist_entry_term() | ~110 |
+| 00:36 | Edited src/controller/config_repl.c | 3→3 lines | ~47 |
+| 00:36 | Edited src/controller/ha.c | modified if() | ~202 |
+| 00:37 | Edited src/controller/testsuite/test_config_repl.c | 4→4 lines | ~66 |
+| 00:37 | Edited src/controller/testsuite/test_config_repl.c | 4→4 lines | ~66 |
+| 00:37 | Edited src/controller/testsuite/test_config_repl.c | 4→4 lines | ~67 |
+| 00:38 | Edited src/controller/testsuite/test_coldstart_lsn0.c | 3→3 lines | ~50 |
+| 00:38 | Edited src/controller/testsuite/test_coldstart_lsn0.c | 3→3 lines | ~50 |
+| 00:39 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~1034 |
+| 00:40 | Edited src/controller/testsuite/test_config_repl.c | 4→5 lines | ~76 |
+| 00:44 | Edited src/controller/config_repl.c | 3→3 lines | ~48 |
+| 00:45 | Edited src/controller/config_repl.c | modified persist_entry_term() | ~49 |
+| 00:46 | Edited src/lib/client.c | added 1 condition(s) | ~188 |
+| 00:47 | Edited src/lib/client.c | added 1 condition(s) | ~487 |
+| 01:03 | Created repro854.sh | — | ~416 |
+| 01:05 | Edited repro854.sh | 3→3 lines | ~45 |
+| 01:22 | Created repro_joblist.sh | — | ~201 |
+| 08:30 | bug-1028: streamer term-source FIXED (persist_entry_term); bug-854: client rotation FIXED (xce_failover_rotate on non-auth reply); both RED→GREEN, full build 0, unit :FAIL:=0, live committed-prefix identical + wait-primary 92ms | config_repl.c/.h, ha.c, client.c, test_config_repl.c | bugs CLOSED | ~28k |
+| 01:29 | Session end: 23 writes across 9 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 16 reads | ~250791 tok |
+| 01:33 | Session end: 23 writes across 9 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 16 reads | ~250791 tok |
+| 01:44 | Edited src/lib/testsuite/test_wal_wire.c | modified heartbeat() | ~150 |
+| 01:46 | Session end: 24 writes across 10 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 18 reads | ~266661 tok |
+| 02:15 | Session end: 24 writes across 10 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 19 reads | ~275693 tok |
+| 02:44 | Edited src/controller/cluster.c | added 1 condition(s) | ~660 |
+| 02:53 | Edited src/lib/xcectx.c | 1→2 lines | ~26 |
+| 02:54 | Edited src/lib/xcectx.c | added 2 condition(s) | ~286 |
+| 02:58 | Session end: 27 writes across 12 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 22 reads | ~281425 tok |
+| 03:07 | Session end: 27 writes across 12 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 22 reads | ~281425 tok |
+| 03:25 | Edited src/lib/netio.c | modified iomgr_monitor() | ~240 |
+| 03:27 | Session end: 28 writes across 13 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 22 reads | ~282052 tok |
+| 03:31 | Created ../../../../../../tmp/xce-ha-verify/test.sh | — | ~291 |
+| 03:37 | Edited src/lib/netio.c | modified handle_ssl_handshake() | ~276 |
+| 03:40 | Edited src/controller/cluster.c | expanded (+13 lines) | ~282 |
+| 03:43 | Edited src/lib/netio.c | inline fix | ~12 |
+| 03:44 | Edited src/lib/netio.c | inline fix | ~7 |
+| 03:46 | Edited src/include/netio.h | 2→3 lines | ~60 |
+| 03:48 | Edited src/controller/cluster.c | inline fix | ~12 |
+| 03:49 | Edited src/controller/cluster.c | modified if() | ~290 |
+| 03:51 | Edited src/lib/netio.c | modified iomgr_monitor() | ~161 |
+| 03:52 | Edited src/lib/netio.c | modified handle_ssl_handshake() | ~202 |
+| 03:53 | Edited src/include/netio.h | 3→2 lines | ~37 |
+| 03:54 | Edited src/lib/netio.c | inline fix | ~6 |
+| 03:58 | Session end: 40 writes across 15 files (piped-mixing-glade.md, config_repl.h, config_repl.c, ha.c, test_config_repl.c) | 23 reads | ~290775 tok |
+
+## Session: 2026-06-23 06:20
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 06:24 | Created .claude/skills/xce-test-ha/run-ha-soak.sh | — | ~2292 |
+| 06:25 | Created .claude/skills/xce-test-ha/SKILL.md | — | ~1430 |
+| 06:27 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified xtest() | ~112 |
+| 06:27 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified code() | ~263 |
+| 06:28 | Edited .claude/skills/xce-test-ha/SKILL.md | 7→12 lines | ~194 |
+| 06:30 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified discover_primary() | ~199 |
+| 06:37 | Edited src/controller/testsuite/waldump.c | 3→4 lines | ~69 |
+| 06:37 | Edited src/controller/testsuite/waldump.c | added 1 condition(s) | ~227 |
+| 06:37 | Edited src/controller/testsuite/waldump.c | 2→3 lines | ~50 |
+| 06:42 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 7→10 lines | ~158 |
+| 06:59 | Edited src/lib/client.c | added 1 condition(s) | ~550 |
+| 07:01 | Edited src/lib/client.c | added 2 condition(s) | ~451 |
+| 07:04 | Edited .claude/skills/xce-test-ha/SKILL.md | expanded (+33 lines) | ~516 |
+| 07:08 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified wait_for_primary_log() | ~503 |
+| 07:08 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified 10() | ~770 |
+| 07:15 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified logs() | ~790 |
+| 07:15 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+8 lines) | ~467 |
+| 07:23 | Edited src/controller/ha.c | added error handling | ~746 |
+| 07:23 | Edited src/controller/ha.h | expanded (+11 lines) | ~214 |
+| 07:24 | Edited src/controller/cluster.c | 2→7 lines | ~135 |
+| 07:25 | Edited src/controller/testsuite/test_persist_recovery.c | modified test_recovered_member_syncs_raft_log_len() | ~955 |
+| 07:39 | Edited src/controller/ha.c | 3→3 lines | ~37 |
+| 07:43 | Edited src/controller/ha.c | modified if() | ~516 |
+| 07:48 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified wait_for_full_convergence() | ~537 |
+| 14:52 | /xce-test-ha skill + 3 failover bug fixes (bug-854 dead-first, bug-1042 rejoin log_len, bug-1043 catch-up apply); 5-round soak PASS zero loss | client.c, ha.c, cluster.c, ha.h, waldump.c, skill | 3 bugs FIXED + skill created | ~60k |
+| 07:53 | Edited .claude/skills/xce-test-ha/SKILL.md | expanded (+8 lines) | ~175 |
+| 07:54 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified submit_jobs() | ~426 |
+| 08:00 | Session end: 26 writes across 8 files (run-ha-soak.sh, SKILL.md, waldump.c, client.c, ha.c) | 13 reads | ~186275 tok |
+
+## Session: 2026-06-23 08:01
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+
+## Session: 2026-06-23 08:15
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 08:31 | Created ../../../../../claude/.claude/plans/task-notification-task-id-bkkk2llpr-tas-zany-locket.md | — | ~366 |
+| 08:31 | Session end: 1 writes across 1 files (task-notification-task-id-bkkk2llpr-tas-zany-locket.md) | 1 reads | ~392 tok |
+| 08:47 | Session end: 1 writes across 1 files (task-notification-task-id-bkkk2llpr-tas-zany-locket.md) | 1 reads | ~392 tok |
+| 09:01 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+13 lines) | ~379 |
+| 09:02 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+9 lines) | ~286 |
+| 09:02 | Edited .claude/skills/xce-test-ha/SKILL.md | "End-to-end HA failover so" → "End-to-end HA failover so" | ~62 |
+| 09:02 | Edited .claude/skills/xce-test-ha/SKILL.md | expanded (+8 lines) | ~213 |
+| 09:03 | Edited .claude/skills/xce-test-ha/SKILL.md | expanded (+9 lines) | ~153 |
+| 09:04 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 5→9 lines | ~136 |
+| 22:11 | enhance /xce-test-ha: add [dir] arg (positional $1 > XCE_HA_TOP > default) + rm-rf safety guard (.xce-ha-deploy marker, fail-loud) | run-ha-soak.sh, SKILL.md | verified: guard refuses unmarked non-empty dir; custom-dir 5-round soak PASS 18 jobs zero loss | ~3k |
+| 15:11 | Session end: 7 writes across 3 files (task-notification-task-id-bkkk2llpr-tas-zany-locket.md, run-ha-soak.sh, SKILL.md) | 3 reads | ~7892 tok |
+| 15:13 | Session end: 7 writes across 3 files (task-notification-task-id-bkkk2llpr-tas-zany-locket.md, run-ha-soak.sh, SKILL.md) | 3 reads | ~7892 tok |
+| 15:50 | Edited src/lib/testsuite/test_hooks.c | modified AND() | ~682 |
+| 15:52 | Edited src/lib/hook.c | added 4 condition(s) | ~377 |
+| 23:10 | fix bug-845 WAL double-append: xce_add_hook idempotent on (fn,data) | src/lib/hook.c, test_hooks.c | duplicates gone (waldump single-entry), 5/5 hook tests, 343 ctrl tests 0 new fails, 3/3 soaks pass; 1 earlier soak hit known intermittent bug-795/828 loss (pre-existing) | ~9k |
+| 16:10 | Session end: 9 writes across 5 files (task-notification-task-id-bkkk2llpr-tas-zany-locket.md, run-ha-soak.sh, SKILL.md, test_hooks.c, hook.c) | 10 reads | ~120437 tok |
+
+## Session: 2026-06-24 10:37
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 19:58 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified log() | ~227 |
+| 10:58 | Session end: 1 writes across 1 files (run-ha-soak.sh) | 3 reads | ~5079 tok |
+| 19:58 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 3→4 lines | ~44 |
+| 19:58 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 4→5 lines | ~91 |
+| 20:00 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | added error handling | ~890 |
+| 20:00 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+28 lines) | ~369 |
+| 20:01 | Edited .claude/skills/xce-test-ha/SKILL.md | modified nodes() | ~447 |
+| 03:04 | enhance /xce-test-ha: force lowest-priority ctrl-3 to lead (freshness-fence maneuver) + assert all 3 nodes led | run-ha-soak.sh, SKILL.md | verified: ctrl-3 elected term 8, zero loss, PASS "all 3 nodes led" | ~7k |
+| 20:21 | Session end: 6 writes across 2 files (run-ha-soak.sh, SKILL.md) | 4 reads | ~9263 tok |
+
+## Session: 2026-06-24 11:28
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 11:35 | /xce-test-ha 10 /tmp/xce-test-ha: 10 rounds, 36 jobs, zero job loss, all 3 nodes led | xce-ha-soak.log | PASS (exit 0) | ~50 |
+
+## Session: 2026-06-24 11:50
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 12:05 | Diagnosed prevote deadlock after soak test: ctrl-3 incorrectly truncated committed entries 45-47 (term=12) on becoming primary (safe_committed_lsn=44 from voter quorum, not honoring own ack knowledge); ctrl-2 received term=13 NOOP at LSN=45 (conflicts with ctrl-1 term=12 job); 2-node deadlock permanent — restart ctrl-3 to resolve | ctrl-*.log, WALs | ANALYZED: bug-795 class | ~1200 |
+
+## Session: 2026-06-24 21:15
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 21:20 | Edited src/controller/testsuite/test_ha_election.c | added error handling | ~841 |
+| 21:20 | Edited src/controller/testsuite/test_ha_election.c | 3→6 lines | ~78 |
+| 21:21 | Edited src/controller/ha.h | expanded (+20 lines) | ~362 |
+| 21:21 | Edited src/controller/ha.c | modified ha_promotion_truncate_floor() | ~195 |
+| 21:21 | Edited src/controller/ha.c | modified 1060() | ~356 |
+| 21:30 | Edited src/controller/ha.c | expanded (+15 lines) | ~308 |
+| 21:30 | Edited src/controller/ha.c | 6→10 lines | ~188 |
+| 21:31 | Edited src/controller/ha.c | added error handling | ~738 |
+| 21:31 | Edited src/controller/ha.c | modified ha_promotion_truncate_floor() | ~195 |
+| 21:31 | Edited src/controller/ha.c | modified 1060() | ~356 |
+| 21:32 | Edited src/controller/ha.h | expanded (+11 lines) | ~235 |
+| 21:32 | Edited src/controller/ha.h | expanded (+19 lines) | ~383 |
+| 21:33 | Edited src/controller/testsuite/test_ha_election.c | added error handling | ~841 |
+| 21:33 | Edited src/controller/testsuite/test_ha_election.c | 3→6 lines | ~78 |
+| 21:36 | Edited src/controller/CLAUDE.md | inline fix | ~239 |
+| 21:37 | Edited src/controller/CLAUDE.md | inline fix | ~147 |
+| 21:37 | Edited src/controller/CLAUDE.md | inline fix | ~63 |
+
+## Session: 2026-06-24 12:38
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 18:05 | fix bug-1060: new leader truncated its own committed entries (Raft Leader Completeness violation) → divergent WALs + prevote deadlock + job loss | src/controller/ha.c, ha.h, testsuite/test_ha_election.c | added ha_promotion_truncate_floor=max(safe_lsn,durable_tip); raft_cb_become_leader truncates at clamped floor (no-op on own log); unit test GREEN, 0 failures | ~60k |
+| 18:06 | steering + bookkeeping for bug-1060 | src/controller/CLAUDE.md, .wolf/buglog.json, .wolf/cerebrum.md | updated Leader Completeness/Post-Election/checklist; bug-1060 marked FIXED; Key Learning + Do-Not-Repeat (clang-format whole-file recurrence) | ~8k |
+| 21:39 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_bug1060_leader_completeness.md | — | ~553 |
+| 21:39 | Session end: 1 writes across 1 files (project_bug1060_leader_completeness.md) | 0 reads | ~593 tok |
+
+## Session: 2026-06-24 14:50
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+
+## Session: 2026-06-24 15:03
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+
+## Session: 2026-06-24 15:36
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 01:17 | Edited src/controller/testsuite/test_term_safety_adapter.c | modified test_intake_crossterm_catchup_uses_leader_current_term() | ~1730 |
+| 01:17 | Edited src/controller/testsuite/test_term_safety_adapter.c | 2→4 lines | ~68 |
+| 01:17 | Edited src/schema/controller.capnp | added error handling | ~260 |
+| 01:17 | Edited src/lib/request.c | added error handling | ~139 |
+| 01:18 | Edited src/lib/request.c | 4→7 lines | ~112 |
+| 01:18 | Edited src/controller/config_repl.c | modified ha_stream_wal_to_standby() | ~70 |
+| 01:18 | Edited src/controller/config_repl.c | added error handling | ~190 |
+| 01:18 | Edited src/controller/config_repl.c | modified if() | ~121 |
+| 01:18 | Edited src/controller/ha.c | modified if() | ~292 |
+| 01:19 | Edited src/controller/config_repl.h | added error handling | ~166 |
+| 01:19 | Edited src/controller/testsuite/test_config_repl.c | 3→5 lines | ~95 |
+| 01:20 | Edited src/controller/ha.c | added error handling | ~256 |
+| 01:21 | Edited src/include/typemap.h | added error handling | ~222 |
+| 01:22 | Edited src/controller/testsuite/test_config_repl.c | 3→4 lines | ~63 |
+| 01:22 | Edited src/controller/testsuite/test_config_repl.c | 3→4 lines | ~62 |
+| 01:22 | Edited src/controller/testsuite/test_config_repl.c | 3→4 lines | ~67 |
+| 01:23 | Edited src/controller/testsuite/test_coldstart_lsn0.c | 2→3 lines | ~48 |
+| 01:32 | Edited src/controller/CLAUDE.md | added error handling | ~136 |
+| 20:10 | fix bug-1067: cross-term WAL catch-up stalled — AE header term conflated with entry's stored term → ae_reject_stale on every historical entry | src/schema/controller.capnp, src/include/typemap.h, src/lib/request.c, src/controller/config_repl.c, ha.c, config_repl.h | added leader_current_term field (wire @8 + C struct); streamer stamps ha->term; receiver uses it as AE header term w/ fallback. 2 new tests GREEN (cross-term accept + stale-leader reject); 185+80+6 PASS, 0 fail | ~95k |
+| 20:11 | bookkeeping for bug-1067 | .wolf/buglog.json, .wolf/cerebrum.md, src/controller/CLAUDE.md | bug-1067 FIXED; Key Learning (AE two-term distinction; capnp struct hand-defined in typemap.h); steering checklist item added | ~6k |
+| 01:33 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_bug1067_ae_two_terms.md | — | ~743 |
+| 01:34 | Session end: 19 writes across 11 files (test_term_safety_adapter.c, controller.capnp, request.c, config_repl.c, ha.c) | 13 reads | ~244235 tok |
+
+## Session: 2026-06-24 16:35
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|
+| 16:45 | xce-test-ha soak regression: 10 rounds, 36 jobs, zero loss, all 3 nodes led; bug-1060 (leader-completeness truncation fix) + bug-1067 (cross-term AE header fix) both GREEN | src/controller/ha.c, config_repl.c, request.c, typemap.h | PASS | ~50 |
+
+## Session: 2026-06-24 02:00
+
+| Time | Action | File(s) | Outcome | ~Tokens |
+|------|--------|---------|---------|--------|