lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)

Browse source 
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
Rongsong Shen 2026-07-07 03:03:40 +00:00
parent 5f98d1b0ef
commit a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

59
.claude/skills/xce-test-ha/SKILL.md
View file

@ -99,6 +99,57 @@ regardless of the excluded bug-854/bug-924 counts. The driver also prints the
usual terminal `PASS:` / `FAIL:` oracle line (exit 0 / non-zero) so a goal-loop
can branch on it unchanged.
### `--factstream` mode: kill-mid-fact-stream (raft-apply-decoupling task 7.2)
Drives a continuous **worker-fact stream** — `jstart`/`jfinish`/`jfail`, real
wire-protocol worker-report notifications (`xcecli_notify_jobstart`/
`_jobfinish`/`_jobfail` in libxce), sent from `xce_test`'s `worker` REPL
context — against a batch of pre-submitted jobs, kills the primary partway
through the stream, lets a new primary take over, and asserts:
- **zero committed-state divergence** (same `walcompare --committed`
mechanism chaotic mode uses), and
- **zero phantom fact reactions** — a job whose live materialized state
(`jobinfo`) implies a fact was applied (and so its reactions — worker slot
release, scheduler dispatch trigger, client/apiserver notify — already
fired, since `apply_committed_entry()` is the single seam that both
mutates state and fires the reaction hook) with **no committed WAL fact
entry backing it** on some node (`waldump`, extended by this task to
decode `jobid`/`state` for every worker-fact WAL type, not just
`WAL_JOB_SUBMIT`).
This is a **different traffic shape** than every other mode: `jobsub`
(existing modes) is a client *command* (job-submission); `jstart`/`jfinish`/
`jfail` are worker *facts* (asynchronous job-lifecycle reports) — the two
travel different code paths in the controller (`handle_jobsubmit` vs.
`on_job_start`/`on_job_finish`/`on_job_fail`) and this mode is the only one
that exercises the fact path under a kill.
```bash
SKILL_DIR=.claude/skills/xce-test-ha
# 2-round kill-mid-fact-stream soak, scratch dir
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --factstream ~/tmp/xce-ha-factstream
# tune job count / kill timing / round count
XCE_HA_FACTSTREAM_JOBS=40 XCE_HA_FACTSTREAM_KILL_DELAY=2 XCE_HA_FACTSTREAM_ROUNDS=3 \
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --factstream ~/tmp/xce-ha-factstream
```
Verdict line (`soak-verdict.sh`'s `verdict_emit_factstream`):
```
[xce-test-ha] VERDICT mode=factstream rounds=2 jobs=3 facts_sent=40 committed_divergence=0 phantom_facts=0 excluded_bug924_fdatasync=0 result=PASS
```
`result=PASS` iff `committed_divergence == 0` **and** `phantom_facts == 0`.
`jobs` is the `jobsub`-tracked count from the shared initial-submit step;
`facts_sent` is the number of fact-stream jobs (each gets a `jstart` +
`jfinish`/`jfail` pair, so the actual fact-message count is `2 × facts_sent`).
A fact genuinely lost in the kill window (fire-and-forget `notify`, no reply)
is **not** a phantom and is **not** counted — that is the accepted
block-but-durable outcome (Requirement 6) for a fact that never reached
majority commit; only the reverse (a reaction fired with no committed WAL
entry backing it) counts.
### Verdict self-test (no cluster)
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
@ -167,12 +218,18 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|---|---|---|
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
| `XCE_HA_MODE` | `sequential` or `chaotic` (`--chaotic` flag wins) | `sequential` |
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, or `factstream` (its `--flag` wins) | `sequential` |
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
| `XCE_HA_FACTSTREAM_ROUNDS` | `--factstream` kill-mid-fact-stream rounds | `2` |
| `XCE_HA_FACTSTREAM_JOBS` | `--factstream` jobs per round (each = 1 jstart + 1 jfinish/jfail) | `30` |
| `XCE_HA_FACTSTREAM_KILL_DELAY` | `--factstream` seconds before killing the primary after the stream starts | `3` |
| `XCE_HA_FACTSTREAM_WAVE_SIZE` | `--factstream` jobs per paced wave (own xce_test connection) | `3` |
| `XCE_HA_FACTSTREAM_WAVE_GAP` | `--factstream` seconds slept between waves | `0.5` |
| `XCE_HA_FACTSTREAM_SETTLE` | `--factstream` seconds slept after convergence, before traffic starts (bug-1416 mitigation) | `10` |
## What the driver does (maps to the requested steps)

358
.claude/skills/xce-test-ha/run-ha-soak.sh
View file

@ -32,6 +32,16 @@
# own log that a snapshot install actually happened (not
# just a claim), then asserts zero walcompare divergence
# and zero job loss. Single dedicated round, not a loop.
# --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a
# continuous WORKER-FACT stream (jstart/jfinish/jfail — real
# wire-protocol worker reports, distinct from job-submission
# commands) against a batch of jobs, kills the primary
# mid-stream, lets a new primary take over, and asserts zero
# committed-state divergence and zero PHANTOM fact reactions
# (a job whose live state implies a fact applied — and so its
# reactions already fired — with no committed WAL fact entry
# backing it on some node). Repeats XCE_HA_FACTSTREAM_ROUNDS
# times (default 2), then emits one VERDICT line.
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
# and exit — no cluster, no binaries needed.
#
@ -50,6 +60,27 @@
# out, so the 64 MB production default never rotates at
# soak job counts and force-snapshot could never observe
# a real snapshot-triggered catch-up without this)
# XCE_HA_FACTSTREAM_ROUNDS kill-mid-fact-stream rounds (default 2)
# XCE_HA_FACTSTREAM_JOBS jobs per fact-stream round (default 30) —
# each job gets a jstart + jfinish/jfail pair,
# so this is HALF the fact-message count
# XCE_HA_FACTSTREAM_KILL_DELAY seconds to sleep before killing the primary
# after starting the background fact stream
# (default 3) — tunes how far into the stream
# the kill lands
# XCE_HA_FACTSTREAM_WAVE_SIZE jobs per fact-stream "wave" (default 3) —
# each wave is its own xce_test connection;
# smaller waves = finer-grained pacing
# XCE_HA_FACTSTREAM_WAVE_GAP seconds slept between waves (default 0.5) —
# paces the stream over several real seconds
# instead of a sub-second burst, so the kill
# reliably lands mid-stream
# XCE_HA_FACTSTREAM_SETTLE seconds slept after full convergence,
# before any fact-stream traffic starts
# (default 10) — bug-1416 mitigation: gives
# every node's boot-time recovery vtask a
# realistic window to finish before this
# scenario's traffic begins
set -u
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
@ -67,6 +98,7 @@ while [ $# -gt 0 ]; do
--chaotic) MODE="chaotic" ;;
--sequential) MODE="sequential" ;;
--force-snapshot) MODE="force-snapshot" ;;
--factstream) MODE="factstream" ;;
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
@ -141,6 +173,10 @@ V_JOBLOSS=0 # tracked committed jobs "not found" on a REACHABLE primary
V_BUG854=0 # client could not locate the primary (libxce discovery fault)
V_BUG924=0 # overlayfs `fdatasync FAILED` log lines (environment defect)
# ── Fact-stream mode counters / tracking (raft-apply-decoupling task 7.2) ────
V_PHANTOM=0 # phantom fact reactions observed (see verify_factstream_no_phantom)
FACTSTREAM_ALL_JOBIDS=() # every jobid ever driven through the fact stream, across rounds
log() { echo "[xce-test-ha] $*"; }
fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; }
@ -774,6 +810,266 @@ run_force_snapshot_round() {
walcheck_count "${#ALL_JOBIDS[@]}"
}
# ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ───
#
# Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail —
# real wire-protocol worker-report notifications; see cmd_jobstart/
# cmd_jobfinish/cmd_jobfail in testsuite/test.c) against a batch of jobs,
# kills the primary partway through, lets a new primary take over, then
# verifies zero committed-state divergence and zero PHANTOM fact reactions.
#
# Why wire-level facts, not a real xcecd_worker process: on_job_start/
# on_job_finish/on_job_fail (job.c) require only that the job already exists
# in jobmgr (i.e. was submitted) — they do not require a real scheduler
# dispatch/host-allocation handshake. jstart/jfinish/jfail therefore drive
# the EXACT SAME production receipt path a genuine worker's report would
# (on_job_* -> record_job_*_fact -> persist_submit_job_fact[_ex] -> durable
# append -> commit engine -> apply pump -> apply_committed_entry()) without
# needing to stand up a real scheduler + worker daemon pair (a much larger,
# more fragile lift that nothing else in this soak harness attempts either —
# the existing scenarios only ever drive job-submission commands). This is
# the same "wire-level synthetic worker" pattern the harness already uses for
# jfinish (pre-existing); jstart/jfail were the missing primitives.
#
# What "phantom" means here, concretely: apply_committed_entry() is the
# SINGLE seam that both mutates job state AND fires the post-mutation
# notification hook (worker slot release / scheduler dispatch trigger /
# client-apiserver notify all fan out from that one firing — raft-apply-
# decoupling task 6.3's verified finding). So a job's live materialized
# state on a given node is a faithful proxy for "did that node's reactions
# already fire" — checking (live state implies X happened) against (a
# committed WAL entry for X exists on that node) directly tests the
# apply-after-commit invariant this whole feature exists to enforce,
# without needing separate instrumentation of workermgr slot counts.
# Cross-check every jobid ever driven through the fact stream: live
# materialized state (jobinfo, queried on the reconverged primary) vs.
# committed WAL fact entries for that jobid on EVERY node (waldump — checking
# all three, not just the primary, catches a same-node apply-without-commit
# defect even if walcompare's coarser prefix comparison somehow missed it).
# A job that simply never shows a fact's effect (e.g. a fact genuinely lost
# in the kill window, block-but-durable per Req 6 / decision D1) is NOT a
# phantom — only the reverse (state implies a fact fired, but no committed
# WAL entry backs it on some node) counts. Increments V_PHANTOM.
verify_factstream_no_phantom() {
local want="${#FACTSTREAM_ALL_JOBIDS[@]}"
[ "$want" -eq 0 ] && return 0
local script jid out
script="set masters $MASTERS
set myname factverify
client
wait-primary $WAITMS"
for jid in "${FACTSTREAM_ALL_JOBIDS[@]}"; do
script+="
jobinfo -j $jid"
done
script+="
quit"
out="$(printf '%s\n' "$script" | xtest)"
# JobID:/state_raw: lines always alternate 1:1 (print_jobinfo, testsuite/
# test.c). Deliberately parse state_raw (the raw hex bitmask), NOT the
# pretty state: string: job_state_str() checks JOB_STAT_RUN before
# JOB_STAT_EXIT, so a job that is simultaneously RUN and EXIT|PERR (e.g. a
# worker-reported FAIL received with no prior FINISH — apply_job_fail_
# transition ORs in EXIT|PERR without clearing RUN, job.c) prints as
# "RUN", silently hiding the EXIT/PERR bits the fact-stream's own 1-in-5
# jfail-without-jfinish jobs deliberately produce. Testing the raw bitmask
# avoids under-detecting exactly the traffic shape this scenario sends.
declare -A JSTATE=()
local cur=""
while IFS= read -r line; do
case "$line" in
"JobID: "*) cur="${line#JobID: }" ;;
"state_raw: "*) [ -n "$cur" ] && JSTATE["$cur"]="${line#state_raw: }"; cur="" ;;
esac
done <<<"$out"
# One waldump per node (not per job) — cheap, offline, read-only.
declare -A DUMPS=()
local n
for n in "${NODES[@]}"; do
DUMPS["$n"]="$(printf 'waldump %s/controller/%s/data\nquit\n' "$TOP" "$n" | xtest)"
done
# JOB_STAT_* bit values (src/include/xce/types.h): RUN=0x2 EXIT=0x4
# DONE=0x8 PERR=0x100.
local phantom_this_check=0 raw started terminal has_start_any d
declare -A HAS_START=() HAS_FINISH=()
for jid in "${FACTSTREAM_ALL_JOBIDS[@]}"; do
raw="${JSTATE[$jid]:-}"
# Job vanished from joblist entirely — a separate, already-covered
# loss class (verify_no_loss/walcheck_count), not this check's job.
[ -z "$raw" ] && continue
terminal=0
if [ $((raw & 0x4)) -ne 0 ] || [ $((raw & 0x8)) -ne 0 ] || [ $((raw & 0x100)) -ne 0 ]; then
terminal=1 # EXIT, DONE, or PERR — a FINISH or FAIL applied
fi
# Whether a START ever committed for this job can NOT be inferred
# from a TERMINAL state: apply_job_finish_transition (job.c ~1033)
# does an ABSOLUTE overwrite to DONE/EXIT, and apply_job_fail_
# transition's default: branch (job.c ~1139, non-POST_EXEC stages)
# sets JOB_STAT_EXIT unconditionally — NEITHER requires JOB_STAT_RUN
# to have been set first. The fact-stream harness's own reconnect-
# mid-wave design can legitimately lose a jstart message (fire-and-
# forget, dropped at the kill instant) while its paired jfinish/jfail
# still lands and commits — a tolerated "block-but-durable" outcome
# (Req 6 / decision D1), not a phantom. So "terminal implies a prior
# START" is FALSE and must not gate the JOB_START check. Instead,
# scan every node's WAL directly for a committed JOB_START entry —
# the same presence check already used for JOB_FINISH/JOB_FAIL below
# — and treat "committed on at least one node" as ground truth for
# "a START really happened," independent of the job's terminal
# state. The live RUN bit (0x2) is kept as an additional, INDEPENDENT
# positive signal: it is reliable exactly when the job is NOT
# terminal (a job can only be actively RUN via a real applied START),
# and catches the more severe "state shows RUN with no commit
# anywhere" reaction-before-commit phantom this whole feature exists
# to prevent.
has_start_any=0
HAS_START=(); HAS_FINISH=()
for n in "${NODES[@]}"; do
d="${DUMPS[$n]}"
HAS_START["$n"]=0; HAS_FINISH["$n"]=0
printf '%s\n' "$d" | grep -F "JOB_START" | grep -qF "jobid=$jid " \
&& HAS_START["$n"]=1 && has_start_any=1
printf '%s\n' "$d" | grep -E "JOB_(FINISH|FAIL)" | grep -qF "jobid=$jid " \
&& HAS_FINISH["$n"]=1
done
started=0
[ $((raw & 0x2)) -ne 0 ] && started=1
[ "$has_start_any" -eq 1 ] && started=1
[ "$started" -eq 0 ] && [ "$terminal" -eq 0 ] && continue
for n in "${NODES[@]}"; do
if [ "$started" -eq 1 ] && [ "${HAS_START[$n]}" -eq 0 ]; then
log "PHANTOM: job $jid state_raw=$raw (JOB_START committed on at least one node) but $n's WAL has NO committed JOB_START"
phantom_this_check=$((phantom_this_check + 1))
fi
if [ "$terminal" -eq 1 ] && [ "${HAS_FINISH[$n]}" -eq 0 ]; then
log "PHANTOM: job $jid state_raw=$raw (terminal — worker slot would be freed) but $n's WAL has NO committed JOB_FINISH/JOB_FAIL"
phantom_this_check=$((phantom_this_check + 1))
fi
done
done
V_PHANTOM=$((V_PHANTOM + phantom_this_check))
log "factstream: cross-checked $want jobid(s) across 3 nodes — $phantom_this_check phantom fact reaction(s) this check"
return 0
}
# One kill-mid-fact-stream round: submit a batch of jobs, stream jstart/
# jfinish/jfail for all of them in the background, kill the primary partway
# through, let a new primary take over, restart the downed node, reconverge,
# then verify no committed loss / divergence / phantom reactions.
run_factstream_round() {
local round="$1"
local nfacts="${XCE_HA_FACTSTREAM_JOBS:-30}"
local kill_delay="${XCE_HA_FACTSTREAM_KILL_DELAY:-3}"
local before=${#ALL_JOBIDS[@]}
log "factstream round $round: submitting $nfacts jobs to drive the fact stream against"
submit_jobs "$nfacts" || fail "factstream round $round: could not submit $nfacts jobs"
verify_no_loss || fail "factstream round $round: job loss right after submit, before the fact stream started"
local round_jobids=("${ALL_JOBIDS[@]:$before}")
FACTSTREAM_ALL_JOBIDS+=("${round_jobids[@]}")
# Paced worker-fact stream: jstart followed by jfinish (4-in-5) or jfail
# (1-in-5) per job — a realistic worker lifecycle mix exercising both
# terminal paths (and the _pad-discriminated FAIL apply-time recompute,
# raft-apply-decoupling task 6.1 round 2). Sent as several small WAVES
# (each its own xtest invocation — a fresh worker connection, join
# included) with a short sleep between waves, NOT one giant script: a
# single unpaced script sends all facts back-to-back over one connection
# in well under a second (loopback round trips), so a 1s kill_delay
# landed AFTER the whole stream had already finished, not mid-stream —
# and the resulting instantaneous burst tripped repl_broadcast's
# pending_ack eviction threshold (an unrelated backpressure mechanism),
# observed live to cascade into a slow reconverge. Pacing into waves (a)
# makes "kill mid-stream" land reliably mid-stream over several real
# seconds, (b) avoids the artificial burst, and (c) doubles as a
# realistic "worker reconnects after primary failover" exercise, since
# each wave re-resolves the primary via `set masters` + libxce failover
# exactly like a real worker's next report after a failover would.
# `set continue-on-error` is essential: a wave landing at/after the kill
# instant is EXPECTED to fail transport-side (fire-and-forget notify —
# XCECLI_FF_CALL, client_internal.h) and must not abort that wave.
local wave_size="${XCE_HA_FACTSTREAM_WAVE_SIZE:-3}"
local wave_gap="${XCE_HA_FACTSTREAM_WAVE_GAP:-0.5}"
local stream_log="$TOP/logs/factstream-round$round.out"
: >"$stream_log"
(
local start_i=0 total=${#round_jobids[@]} wi wjid wscript
while [ "$start_i" -lt "$total" ]; do
wscript="set masters $MASTERS
set myname factstream
set continue-on-error
worker
join"
wi=0
while [ "$wi" -lt "$wave_size" ] && [ "$start_i" -lt "$total" ]; do
wjid="${round_jobids[$start_i]}"
wscript+="
jstart -j $wjid"
if [ $((start_i % 5)) -eq 4 ]; then
wscript+="
jfail -j $wjid -c 1"
else
wscript+="
jfinish -j $wjid -c 0"
fi
start_i=$((start_i + 1))
wi=$((wi + 1))
done
wscript+="
..
quit"
printf '%s\n' "$wscript" | xtest >>"$stream_log" 2>&1
sleep "$wave_gap"
done
) &
local stream_pid=$!
log "factstream round $round: streaming $((${#round_jobids[@]} * 2)) fact messages in $wave_size-job waves in the background (log: $stream_log)"
sleep "$kill_delay"
PRIMARY="$(discover_primary)"
[ -n "$PRIMARY" ] || fail "factstream round $round: no serving primary right before the kill"
local prev_term; prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
log "factstream round $round: killing primary $PRIMARY mid-stream (term=$prev_term)"
kill_node "$PRIMARY" || fail "factstream round $round: could not kill $PRIMARY"
DOWN="$PRIMARY"
local newp; newp="$(wait_for_new_primary "$prev_term")"
[ -n "$newp" ] || fail "factstream round $round: no new primary elected after the mid-stream kill"
log "factstream round $round: new primary $newp (term=$(latest_primary_term))"
record_primary "$newp"
# Let the background stream finish. Best-effort reconnect resumes any
# remaining facts against whichever primary is reachable; facts already
# in flight at the kill instant may be silently lost (fire-and-forget,
# no reply) — the accepted D1 block-but-durable outcome (requirements.md
# Requirement 6), not a bug, and exactly what verify_factstream_no_
# phantom above tolerates (a lost fact ⇒ the job simply never shows the
# corresponding state — not flagged).
wait "$stream_pid" 2>/dev/null
log "factstream round $round: fact-stream script finished"
start_node "$DOWN"
wait_for_full_convergence \
|| fail "factstream round $round: cluster did not reconverge to 3-of-3 after restarting $DOWN"
DOWN=""
verify_no_loss || fail "factstream round $round: a committed JOB_SUBMIT was lost after the mid-stream kill"
walcheck_count "${#ALL_JOBIDS[@]}"
verify_factstream_no_phantom
}
# Drive a SPECIFIC node to primary — used to exercise leadership on the
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
@ -894,6 +1190,68 @@ if [ "$MODE" = "force-snapshot" ]; then
exit 1
fi
# ── Fact-stream mode: kill-mid-fact-stream regression ─────────────────────────
# Opt-in (--factstream / XCE_HA_MODE=factstream). raft-apply-decoupling task
# 7.2 (Req 10.2/10.4). See run_factstream_round / verify_factstream_no_phantom
# above for the scenario + verification. Repeats XCE_HA_FACTSTREAM_ROUNDS times
# (default 2 — independent of the general --chaotic ROUNDS knob, since this
# scenario's own repeatability bar is "reliably lands mid-stream and produces a
# correct verdict", not a long soak) so a single lucky/unlucky kill-timing
# window does not decide the verdict, then emits one VERDICT line and exits —
# like force-snapshot/chaotic, it does NOT run the sequential loop or
# leadership-coverage maneuver.
if [ "$MODE" = "factstream" ]; then
FS_ROUNDS="${XCE_HA_FACTSTREAM_ROUNDS:-2}"
log "MODE=factstream — kill-mid-fact-stream soak ($FS_ROUNDS rounds)"
# Wait for full 3-of-3 convergence, THEN an extra settle delay, before
# introducing any fact-stream load or kills. Step 4 above only confirms
# ONE primary is reachable, and ctrlstatus answers as soon as a node's
# listener is open — BEFORE that node's own cluster_init_phase2()/
# persist_recover() (the boot-time apply-correctness verification, Req
# 8.1/8.2) has actually run, since cluster_open_listener() precedes
# persist_recover() in phase2 and persist_recover() itself fires from a
# vtask, not synchronously (bug-1416, found live by this scenario): if a
# primary is elected fast and starts replicating aggressive early job/fact
# traffic while a standby's OWN recovery vtask is still queued behind a
# burst of incoming replication-write vtasks on the shared pool, that
# standby's recovery can be delayed 30s+ (observed) instead of its normal
# ~5s — by the time it finally runs, the entity/position counts it
# verifies against no longer match its own single-pass replay model,
# producing a false APPLY_DIVERGENT fence that (correctly, per the fence's
# own fail-loud design) can NEVER self-clear without a further process
# restart. This is a real, reproducible finding (see bug-1416), out of
# this task's boundary to fix (persist_recover/cluster_init_phase2
# sequencing, not the regression-scenario itself). The settle delay below
# is a scenario-side mitigation only: give every node's recovery vtask a
# realistic window to run BEFORE this scenario's traffic begins, so the
# kill this scenario asserts on lands "mid-fact-stream on a settled
# cluster" (what it is chartered to test) rather than racing an unrelated
# boot-sequencing issue every run.
wait_for_full_convergence \
|| fail "factstream: cluster did not reach full 3-of-3 convergence before starting the fact stream"
settle="${XCE_HA_FACTSTREAM_SETTLE:-10}"
log "factstream: cluster converged — settling ${settle}s before starting fact-stream traffic (bug-1416 mitigation)"
sleep "$settle"
for fs_round in $(seq 1 "$FS_ROUNDS"); do
log "=== factstream round $fs_round/$FS_ROUNDS ==="
run_factstream_round "$fs_round"
done
scan_bug924
verdict_emit_factstream "$FS_ROUNDS" "${#ALL_JOBIDS[@]}" \
"${#FACTSTREAM_ALL_JOBIDS[@]}" "$V_DIVERGENCE" "$V_PHANTOM" "$V_BUG924"
VERDICT_RC=$?
if [ "$VERDICT_RC" -eq 0 ]; then
log "PASS (factstream): $FS_ROUNDS rounds, ${#FACTSTREAM_ALL_JOBIDS[@]} fact-stream jobs, zero committed divergence, zero phantom fact reactions (bug924=$V_BUG924 excluded)"
cleanup
trap - EXIT INT TERM
exit 0
fi
echo "[xce-test-ha] FAIL: factstream verdict — committed_divergence=$V_DIVERGENCE phantom_facts=$V_PHANTOM (bug924=$V_BUG924 excluded)" >&2
cleanup
trap - EXIT INT TERM
exit 1
fi
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
# windows; outcomes are COUNTED (committed divergence + committed job loss) with

20
.claude/skills/xce-test-ha/soak-verdict.sh
View file

@ -89,3 +89,23 @@ verdict_emit() {
"result=$result"
[ "$result" = "PASS" ]
}
# Emit the machine-parseable verdict line for the factstream (task 7.2)
# scenario. PASS iff committed_divergence == 0 AND phantom_facts == 0 (a
# fact-driven notification/reaction observed on a node with no matching
# committed WAL fact entry backing it — see run_factstream_round's
# verify_factstream_no_phantom in run-ha-soak.sh for the concrete check).
# excluded_bug924 is reported for visibility only, same convention as
# verdict_emit above. Returns 0 on PASS, 1 on FAIL.
# Args: rounds jobs facts_sent committed_divergence phantom_facts bug924
verdict_emit_factstream() {
local rounds="$1" jobs="$2" facts="$3" div="$4" phantom="$5" b924="$6"
local result="PASS"
if [ "$div" -ne 0 ] || [ "$phantom" -ne 0 ]; then
result="FAIL"
fi
echo "[xce-test-ha] VERDICT mode=factstream rounds=$rounds jobs=$jobs" \
"facts_sent=$facts committed_divergence=$div phantom_facts=$phantom" \
"excluded_bug924_fdatasync=$b924 result=$result"
[ "$result" = "PASS" ]
}

3
.kiro/specs/raft-apply-decoupling/tasks.md
View file

File diff suppressed because one or more lines are too long

69
.wolf/anatomy.md
View file

@ -1,7 +1,13 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-07-03T03:23:30.967Z
> Files: 569 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-07-05T07:39:59.974Z
> Files: 600 tracked | Anatomy hits: 0 | Misses: 0
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/
- `append_bug_1414.py` (~1114 tok)
- `append_bugs.py` (~1412 tok)
- `run_regression.sh` (~181 tok)
## ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/c9cdeb23-6930-42d0-adab-2651b711793f/scratchpad/
@ -219,10 +225,10 @@
## .claude/skills/xce-test-ha/
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~13692 tok)
- `SKILL.md` — XCE HA Failover Soak Test (~3703 tok)
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~18920 tok)
- `SKILL.md` — XCE HA Failover Soak Test (~4553 tok)
- `soak-verdict-selftest.sh` — soak-verdict-selftest.sh — deterministic unit test for the verdict parser in (~1857 tok)
- `soak-verdict.sh` — soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. (~1194 tok)
- `soak-verdict.sh` — soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. (~1383 tok)
## .codegraph/
@ -462,7 +468,12 @@
## .kiro/specs/raft-apply-decoupling/
- `brief.md` — Brief: raft-apply-decoupling (~2239 tok)
- `design.md` — Design Document: raft-apply-decoupling (~9210 tok)
- `facts-in-raft.md` — Design Note: Routing Facts Through Raft (apply-after-commit for RETAIN events) (~5569 tok)
- `requirements.md` — Requirements Document (~4629 tok)
- `research.md` — Research & Design Decisions (~4600 tok)
- `spec.json` (~129 tok)
- `tasks.md` — Implementation Plan (~20719 tok)
## .kiro/specs/raft-commit-safety/
@ -614,6 +625,10 @@
- `#raft.tla#` — ------------------------------- MODULE raft --------------------------------- (~5162 tok)
- `raft.tla` — ------------------------------- MODULE raft --------------------------------- (~5350 tok)
## reports/
- `jun29-jul04-2026.md` — Work Summary: June 30 – July 4, 2026 (~857 tok)
## src/
- `flake.nix` (~571 tok)
@ -889,24 +904,42 @@
- `CLAUDE.md` — XCE Controller (~3373 tok)
- `cluster.c` — include "config.h" (~18098 tok)
- `commit.c` — Declares is (~12825 tok)
- `config_repl.c` — define _GNU_SOURCE (~28945 tok)
- `config_repl.h` — pragma once (~6155 tok)
- `ha.c` — define _GNU_SOURCE (~100992 tok)
- `ha.c` — define _GNU_SOURCE (~102306 tok)
- `ha.h` — ifndef HA_H (~26001 tok)
- `job.c` — include <errno.h> (~9864 tok)
- `job.c` — include <errno.h> (~16424 tok)
- `main.c` — include <argp.h> (~3035 tok)
- `meson.build` (~1491 tok)
- `myapp.c` — define _GNU_SOURCE (~4920 tok)
- `persist.c` — define _GNU_SOURCE (~43339 tok)
- `meson.build` (~1580 tok)
- `myapp.c` — define _GNU_SOURCE (~5215 tok)
- `persist.c` — define _GNU_SOURCE (~60786 tok)
## src/controller/include/
- `cluster.h` — if !defined(_CLUSTER_H_) (~2610 tok)
- `commit.h` — pragma once (~3635 tok)
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1817 tok)
- `controller_test.c` — include <argp.h> (~1986 tok)
- `snapdiff.c` — define _GNU_SOURCE (~833 tok)
- `snapdiff.h` — pragma once (~569 tok)
- `snapverify.c` — define _GNU_SOURCE (~2505 tok)
- `snapverify.h` — pragma once (~645 tok)
- `test_apply_committed_entry.c` — Declares directly (~4141 tok)
- `test_apply_correctness_verification.c` — Task 5.1-5.3: post-replay apply verdict (CLEAN/DIVERGENT), routes through existing loud-fence; 11 tests incl. explicit stays-alive pin + fence-clear-via-restart + direct persist_clear_recovery_unrecoverable coverage (~17000 tok)
- `test_apply_pump_integration.c` — Declares hook (~16913 tok)
- `test_apply_pump.c` — Task 6.1 ordered apply pump tests (4, pool=NULL) + task 6.2 round-2 bug-1415 regression `test_apply_pump_late_pending_apply_registration_self_heals` (5th test, own tpr_* fixture with a REAL multi-vcpu vtaskmgr_t via commit_set_pool, forces apply-pump-wins-race via a barrier) (~17700 tok)
- `test_apply_reapply_idempotency.c` — Declares xce_job_t (~4001 tok)
- `test_apply_seam.c` (~2936 tok)
- `test_apply_seam.c` — raft-apply-decoupling task 2.1: calls apply_job_start/finish/fail/update_transition() directly on a bare jobmgr+job_t (no cluster_t), bypassing on_job_start/finish/fail, to prove the extracted guard cores work outside the receipt path (~2900 tok)
- `test_commit_barrier_reply.c` — Declares uint8_t (~38815 tok)
- `test_commit_durability.c` — define _GNU_SOURCE (~31376 tok)
- `test_commit_resolve.c` (~14018 tok)
- `test_config_repl.c` — define _GNU_SOURCE (~42756 tok)
- `test_convergence_verification.c` — define _GNU_SOURCE (~16987 tok)
- `test_demotion_convergence.c` — define _GNU_SOURCE (~12767 tok)
- `test_e2e_hooks.c` — define _GNU_SOURCE (~2184 tok)
- `test_egress_filter.c` — Declares xce_job_t (~5072 tok)
- `test_ha_core_safety.c` — Declares char (~36309 tok)
@ -917,12 +950,15 @@
- `test_ha_vote_gate.c` — define _GNU_SOURCE (~14165 tok)
- `test_persist_hooks.c` — define _GNU_SOURCE (~4305 tok)
- `test_persist_snapshot_entities.c` — unsigned: queue (~14897 tok)
- `test_persist_truncate.c` — define _GNU_SOURCE (~5031 tok)
- `test_replay_apply_seam.c` — Declares through (~3707 tok)
- `test_snapdiff.c` — define _GNU_SOURCE (~1033 tok)
- `test_snapverify.c` — define _GNU_SOURCE (~2008 tok)
- `test_walcompare.c` — define _GNU_SOURCE (~6845 tok)
- `test.c` — include <argp.h> (~28321 tok)
- `test.c` — include <argp.h> (~29726 tok)
- `walcompare.c` — define _GNU_SOURCE (~5176 tok)
- `walcompare.h` — pragma once (~1474 tok)
- `waldump.c` — define _GNU_SOURCE (~5404 tok)
## src/include/
@ -931,8 +967,9 @@
- `delay.h` — if !defined(_DELAY_H_) (~414 tok)
- `myconf.h` — if !defined(_MYCONF_H_) (~560 tok)
- `netio.h` — if !defined(_NETIO_H_) (~2313 tok)
- `persist.h` — pragma once (~11003 tok)
- `persist.h` — pragma once (~20431 tok)
- `typemap.h` — if !defined(_TYPEMAP_H_) (~4717 tok)
- `wal_types.h` — pragma once (~1996 tok)
## src/lib/
@ -942,9 +979,15 @@
- `request.c` — include <errno.h> (~15919 tok)
- `vtaskmgr.c` — include <errno.h> (~3822 tok)
## src/lib/raft/
- `raft_sim.c` — ifdef XCE_TESTING (~28490 tok)
- `raft.h` — ifndef RAFT_H (~17581 tok)
## src/lib/testsuite/
- `test_ctrlstatus.c` (~4821 tok)
- `test_raft.c` — include <errno.h> (~98108 tok)
- `test_wal_wire.c` — include <errno.h> (~4238 tok)
## src/schema/

174
.wolf/buglog.json
View file

@ -12388,5 +12388,179 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-03T21:00:00.000Z"
},
{
"id": "bug-1409",
"timestamp": "2026-07-04T17:00:00.000Z",
"error_message": "test_demotion_convergence hung/spun (18.5M+ log lines, timeout kill) once commit.c's commit_resolve_pending unconditionally called the new persist_apply_pump_kick",
"file": "src/controller/persist.c",
"root_cause": "apply_pump_drain()'s recheck-then-clear retry loop ignored persist_apply_pump_run()'s return code: on a genuine read failure (e.g. committed_lsn advanced in a test fixture with no real WAL bytes on disk for that LSN, rc=-ENOENT), last_applied_lsn never advances, so the very next re-check inside the SAME drain loop sees identical unresolved work and immediately reclaims the scheduled flag and retries forever -- a tight busy loop with no backoff, hammering disk reads and the logger.",
"fix": "apply_pump_drain() now checks persist_apply_pump_run()'s rc immediately after clearing apply_pump_scheduled; a negative rc returns immediately instead of re-checking for more work and reclaiming the flag. The failure is already logged loudly inside persist_apply_pump_run itself. A future, genuinely NEW kick (a later real commit-advance event) retries then, at that event's natural cadence, never in a busy spin.",
"tags": [
"raft-apply-decoupling",
"task-6.1",
"apply-pump",
"busy-loop",
"infinite-loop",
"commit-resolve-pending",
"regression-sweep"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-04T17:00:00.000Z"
},
{
"id": "bug-1410",
"timestamp": "2026-07-04T17:00:00.000Z",
"error_message": "test_apply_pump's spy hook recorded jobid=0 for HOOK_DETAIL_JOB_UPDATE/HOOK_DETAIL_JOB_NEW instead of the real jobid",
"file": "src/controller/testsuite/test_apply_pump.c",
"root_cause": "The hook's per-subscriber `data`/cls pointer (bound at xce_add_hook registration time) is separate from the va_list's own variadic arguments. apply_committed_entry's notify calls pass (cluster_t*, payload) as the actual varargs -- the spy hook read the FIRST va_arg as the payload pointer directly, skipping the leading cluster_t* argument every real dispatch call (job.c, persist.c) always sends first.",
"fix": "Added `(void)va_arg(ap, cluster_t*);` before extracting the real payload (xlist_t*/job_t*) in both HOOK_DETAIL_JOB_NEW and HOOK_DETAIL_JOB_UPDATE cases, mirroring persist_job_hook_fun's own established va_arg consumption order.",
"tags": [
"raft-apply-decoupling",
"task-6.1",
"test-only",
"va_arg",
"hook-system"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-04T17:00:00.000Z"
},
{
"id": "bug-1411",
"timestamp": "2026-07-04T17:00:00.000Z",
"error_message": "AddressSanitizer global-buffer-overflow in dataq_enq when test_apply_pump.c called persist_submit_job_fact with a real 3-member repl setup",
"file": "src/controller/testsuite/test_apply_pump.c",
"root_cause": "repl_broadcast tried to enqueue onto the dummy standby `int` stand-ins (g_tap_hep_a/b, opaque identity handles only) as if they were real endpoint_t* objects, overflowing past the 4-byte int into adjacent globals -- the fixture omitted the repl_broadcast_send_hook interception test_commit_barrier_reply.c's cbrp_setup already uses to keep repl_broadcast from ever dereferencing those dummy pointers.",
"fix": "Added the identical repl_broadcast_send_hook = tap_drop_send_hook (a no-op stub returning success) in tap_setup/tap_teardown, mirroring test_commit_barrier_reply.c's cbrp_drop_send_hook exactly.",
"tags": [
"raft-apply-decoupling",
"task-6.1",
"test-only",
"asan",
"repl-broadcast",
"fixture"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-04T17:00:00.000Z"
},
{
"id": "bug-1412",
"timestamp": "2026-07-04T17:00:00.000Z",
"error_message": "test_apply_pump.c failed to compile: 'unknown type name RETAIN', stray bytes, __gnuc_va_list errors cascading from an unrelated location",
"file": "src/controller/testsuite/test_apply_pump.c",
"root_cause": "A doc comment contained the literal substring 'WAL_JOB_*/RETAIN' -- the '*' immediately followed by '/' forms a real C block-comment terminator, closing the file's top doc comment many lines early and turning the rest of the comment text into stray code tokens.",
"fix": "Reworded to 'a WAL_JOB_SUBMIT WAIT-mode entry mixed with WAL_JOB_{START,FINISH} RETAIN-mode facts', removing the accidental */ sequence. Lesson: never write '<TOKEN>*/<TOKEN>' inside a C block comment, even when '*' is meant as a wildcard/glob character.",
"tags": [
"raft-apply-decoupling",
"task-6.1",
"test-only",
"comment-terminator",
"build-break"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-07-04T17:00:00.000Z"
},
{
"id": "bug-1413",
"timestamp": "2026-07-04T17:00:00.000Z",
"error_message": "test_apply_pump's burst-ordering test hung the NEXT test (longjmp-past-teardown) after a wrong off-by-one assertion (expected lsn_before+3, actual watermark is lsn_before+2 for 3 appended entries) failed",
"file": "src/controller/testsuite/test_apply_pump.c",
"root_cause": "A TEST_ASSERT failure mid-test longjmps past the rest of the function body, including the explicit tap_teardown() call at the end -- leaving the fsync thread/tmpdir/hooksys from the failed test alive when the next test's tap_setup() ran, hanging it. Root trigger was a genuine off-by-one in the test's own math (current_lsn before N appends is the FIRST assigned lsn, not one less than it; the watermark after N appends is first_lsn + N - 1, not first_lsn + N).",
"fix": "Fixed the off-by-one (first_lsn + 2 == watermark for 3 appends starting at first_lsn). Restructured all three tests in the file to capture every assertion-relevant observable into local variables BEFORE any TEST_ASSERT call, and to call tap_teardown() unconditionally before running any assertion -- the exact pattern already established in this codebase for task 3.1's identical hazard (tasks.md Implementation Notes, task 3.1 round 2/3).",
"tags": [
"raft-apply-decoupling",
"task-6.1",
"test-only",
"longjmp-past-teardown",
"off-by-one",
"test-hygiene"
],
"related_bugs": [
"bug-1408"
],
"occurrences": 1,
"last_seen": "2026-07-04T17:00:00.000Z"
},
{
"id": "bug-1414",
"timestamp": "2026-07-04T18:00:00.000Z",
"error_message": "raft-apply-decoupling task 6.1 round 1 REJECTED: FINISH (LSN N) then a POST_EXEC-stage FAIL (LSN N+1) for the same job, with FAIL received before FINISH applies, corrupts the job to RUN|EXIT|PERR (0x106) instead of the correct DONE|PERR (0x108) once both apply in LSN order -- DONE is silently lost and a stray RUN bit appears.",
"file": "src/controller/job.c, src/controller/persist.c, src/include/wal_types.h, src/include/persist.h, src/controller/include/cluster.h",
"root_cause": "record_job_fail_fact (job.c) and record_job_update_fact (postdone) precomputed a FINAL absolute state at RECEIPT time (scratch-copy merge for FAIL, inline backstop+PDONE merge for postdone) and baked it into wal_jobstate_t.state -- correct only when receipt and apply are the same event. Once task 6.1 deferred apply past the commit barrier, a second in-flight fact for the same job (e.g. FINISH) could apply BETWEEN this fact's receipt and its own apply, making the receipt-time snapshot stale by the time apply_committed_entry's generic apply_job_update_transition() did its blind absolute overwrite -- silently clobbering whatever the other fact had just applied.",
"fix": "Repurposed wal_jobstate_t._pad (previously always-zero, unused) as a discriminator: _pad==0 still means legacy absolute state (job_ctrl's mark_job_fail/mark_job_update live paths, which mutate+capture under one lock atomically -- no staleness risk, unchanged); a nonzero _pad (WAL_JOB_FAIL_STAGE_FLAG|stage for FAIL, WAL_JOB_UPDATE_POSTDONE_MARK for postdone) marks the entry for apply-time recomputation. apply_committed_entry (persist.c) now dispatches WAL_JOB_FAIL/WAL_JOB_UPDATE on this bit, calling the REAL apply_job_fail_transition() / new apply_job_postdone_transition() (job.c) against the job's ACTUAL current state at apply time instead of a stale baked value. New persist_submit_job_fact_ex(p,type,jobid,state,pad) primitive added; persist_submit_job_fact becomes a pad=0 thin wrapper so zero existing call sites needed changes. Also fixed 2 pre-existing test_commit_barrier_reply.c tests that asserted the old wire format directly (cbrp_count_jobstate_entries reading state bits) -- extended the helper to a _ex variant exposing _pad, updated assertions to check the new marker plus a direct call to the guard core on a scratch job_t. New regression test test_apply_pump_fail_racing_finish_merges_against_current_state in test_apply_pump.c drives FINISH then FAIL through the real receipt handlers with both entries durably appended but unapplied, then a single covering ACK; RED-confirmed byte-for-byte (Expected 264 Was 262) against the reverted round-1 code, GREEN after the fix.",
"tags": [
"raft-apply-decoupling",
"task-6.1",
"round-2",
"job-state-corruption",
"merge-staleness",
"deferred-apply",
"wal-wire-format",
"wal_jobstate_t",
"_pad-discriminator"
],
"related_bugs": [
"bug-1413"
],
"occurrences": 1,
"last_seen": "2026-07-04T18:00:00.000Z"
},
{
"id": "bug-1415",
"timestamp": "2026-07-04T19:30:00.000Z",
"error_message": "raft-apply-decoupling task 6.2 review finding (NOT reproduced under the test suite's pool-less fixtures, NOT fixed): a real vtaskmgr pool (n_vcpus > 1) can run the commit continuation's dispatch (job_commit_resolve_adapter -> jobsubmit_commit_continuation -> persist_register_pending_apply) and the apply pump's kick (persist_apply_pump_kick -> apply_pump_drain -> persist_apply_pump_run) on two DIFFERENT worker threads with no completion-order guarantee, even though commit_resolve_pending() enqueues them in that order. If the apply-pump task happens to finish first, persist_take_pending_apply() finds nothing registered yet for that lsn (0, correct no-op per its own contract), applies the entry, and advances last_applied_lsn past it -- then when the continuation task finally runs and calls persist_register_pending_apply() for that now-already-applied lsn, the registration is permanently orphaned: the walk only ever revisits [last_applied_lsn+1..committed_lsn], so nothing will ever call persist_take_pending_apply() for that lsn again. Net effect: the job is correctly constructed and visible in c->jobmgr, but the client that submitted it never receives a reply -- silently, forever.",
"file": "src/controller/commit.c (commit_resolve_pending, commit_resolve/dispatch_continuation), src/controller/persist.c (persist_apply_pump_kick, persist_apply_pump_run)",
"root_cause": "commit_resolve_pending() (commit.c ~793-835, landed by task 1.2/6.1) resolves every ready continuation via commit_resolve() -> dispatch_continuation(), which for a real pool (ce->pool != NULL) enqueues the continuation onto vtaskmgr via vtask_enq_classified(VTASK_LIVE_BLOCK) and returns WITHOUT waiting for it to run; commit_resolve_pending() then calls persist_apply_pump_kick(ce->p, ce->pool), which enqueues a SECOND, independent VTASK_LIVE_BLOCK task onto the SAME pool. vtaskmgr.c's dispatcher (vdispatcher_entry) hands queued tasks to whichever vcpu worker thread is next idle (pick_idle_vcpu/pick_eligible_task) -- with n_vcpus > 1 (the only realistic production configuration; every test fixture in this spec deliberately never calls commit_set_pool, so ce->pool stays NULL and both dispatches run synchronously inline instead, which is why no test in this suite reproduces the race), two tasks enqueued back-to-back can be picked up by two different idle vcpus at nearly the same time, and their COMPLETION order depends on each task's own runtime (a disk-reading WAL-scan-and-apply walk vs. a calloc+mutex-guarded list insert) -- not their enqueue order. This is a pre-existing structural gap from task 1.2 (async continuation dispatch) + task 6.1 (apply-pump-kick wired into the same choke point, same pool) that had no observable consequence until task 6.2 added a real consumer (persist_take_pending_apply) whose registration can now race the very entry it targets.",
"fix": "FIXED round 2 (task 6.2): persist_register_pending_apply() (persist.c) now self-heals option (b) from this entry's own proposed fixes -- before linking a new node onto pending_applies, it checks `lsn <= persist_last_applied_lsn(p)`; if the apply pump's walk has already applied `lsn` (last_applied_lsn already at or past it), `on_applied(ctx, lsn)` fires immediately and synchronously on the CALLING thread instead of being linked for a walk pass that will never revisit that lsn. Race-safe in both interleavings: when the walk has not reached `lsn` yet, the check is false and registration proceeds exactly as before (unchanged behavior, verified against the full existing regression suite); when the walk already passed `lsn`, its own persist_take_pending_apply() call for that lsn already ran and found nothing (that is precisely how we know registration is late), so the immediate inline fire can never double-fire alongside it. New deterministic regression test `test_apply_pump_late_pending_apply_registration_self_heals` (test_apply_pump.c) wires a REAL multi-vcpu vtaskmgr_t (n_vcpus=4) via commit_set_pool -- the one test in this spec to do so -- and forces the exact bad interleaving via a barrier (the test's stand-in commit continuation polls persist_last_applied_lsn() until the apply pump has already applied the lsn, THEN registers), rather than relying on natural timing luck. Verified genuine RED (fails with 'Expected 1 Was 0' against the unfixed check, confirming the hang is real and reproducible) -> GREEN (fires exactly once, resource released exactly once, no orphaned registration) after the fix. Full regression sweep (test_apply_pump, test_commit_barrier_reply, test_apply_reapply_idempotency, test_commit_resolve, test_commit_admission_pool, test_demotion_convergence) shows zero new failures -- the only 2 failures present (test_jobsubmit_returns_before_commit_and_registers_pending_apply, test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements, both in test_commit_barrier_reply.c) are pre-existing/expected per task 6.1's own Implementation Notes (their assertion text predates task 6's apply pump landing; task 7.3's job to update), confirmed identical before and after this fix by reverting to the pre-round-2 baseline and re-running.",
"tags": [
"raft-apply-decoupling",
"task-6.2",
"vtaskmgr",
"race-condition",
"orphaned-registration",
"lost-reply",
"commit_resolve_pending",
"persist_apply_pump_kick",
"fixed",
"self-healing"
],
"related_bugs": [
"bug-1409"
],
"occurrences": 1,
"last_seen": "2026-07-04T23:30:00.000Z"
},
{
"id": "bug-1416",
"timestamp": "2026-07-05T06:15:00.000Z",
"error_message": "[xce-test-ha] FAIL: factstream round 1: cluster did not reconverge to 3-of-3 after restarting ctrl-1 -- root cause: persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=72 new_job_count=23); persist_recover_loud_fence_if_unrecoverable: UNRECOVERABLE, staying alive/non-promotable",
"file": "src/controller/persist.c (persist_verify_apply_correctness / persist_recover, cluster.c cluster_init_phase2)",
"root_cause": "Found live via the new raft-apply-decoupling task 7.2 kill-mid-fact-stream soak scenario (--factstream), reproduced 4/4 runs (ASAN debug build, paced-wave ASAN build, release build, and a cleanup-suppressed debug postmortem run) in this sandboxed environment. cluster_init_phase2() calls cluster_open_listener() (which starts accepting network I/O, including replicated/catch-up WAL traffic from an already-serving primary) BEFORE persist_recover() runs its boot-time apply-correctness verification (task 5.1, Req 8.1/8.2). Under a slow/contended boot (observed: persist_recover firing ~36s after process start instead of the normal ~5s, plausibly due to heavy concurrent nix-develop/test-process CPU contention in this sandbox), a standby's listener can already be live and processing real traffic for tens of seconds before its own persist_recover() call finally executes. persist_verify_apply_correctness() (persist.c ~3597) reads p->last_applied_lsn and live-queries count_jobmgr_entries(p->cluster) AFTER replay_wal completes -- both of these are also mutated by the LIVE apply pump (raft-apply-decoupling task 6.1, wired through commit_resolve_pending) reacting to replicated commit-advance events. If live traffic is already flowing and being applied through the live pump DURING the extended pre-persist_recover window, the live-read 'actual' values used by persist_verify_apply_correctness can reflect state beyond what persist_recover's own single WAL-scan pass computed as 'expected', producing a DIVERGENT verdict even though no real apply defect exists -- confirmed via ctrlstatus probes showing committedLsn identical (79) and no actual data loss across all 3 nodes; the fence engaged correctly per its own fail-loud design (stayed alive, non-promotable, kept retrying) but two of three nodes (including the node that went on to win the next election, term=5, first_boot=1) were fenced at the exact moment persist_recover ran. This matches task 5.1's own documented residual-risk note (tasks.md Implementation Notes, task 5.1 entry): 'the position check is diagnostically tautological for THIS SPECIFIC call site today... this will change once task 6's live apply pump gives expectation and actual state a real chance to diverge across threads/time' -- that predicted future divergence has now materialized under a slow-boot + concurrent-live-traffic combination that only a scenario driving traffic this early/aggressively after cluster boot (the factstream scenario) exercises; existing soak scenarios (chaotic/sequential/force-snapshot) submit far fewer jobs, more slowly, and do not reliably land traffic inside a slow node's pre-recovery window. REFINED after further reproduction (10/10 runs total, across ASAN/release builds and job counts from 8 to 20): the trigger is specifically the RESTART/catch-up path, not the initial multi-node cold boot. A standalone control test (3 nodes, no kill, 20s settle before any traffic) submitted jobs and drove jstart/jfinish/jfail cleanly with zero APPLY_DIVERGENT -- confirming the race needs a node that was DOWN while the cluster kept moving (i.e. an actual failover), not merely a fast cold boot: the returning node's persist_recover()/cluster_init_phase2 vtask must complete while it is ALSO receiving a compressed catch-up replication burst covering everything it missed while down, and that burst's volume is inherent to 'kill mid-fact-stream' (the whole point of the scenario) -- a run-ha-soak.sh-side pacing fix cannot eliminate the trigger the way it did for the (separately real but distinct) initial-boot instance the settle-delay mitigation addresses. Reproduced identically down to 8 fact-stream jobs with 2-job/1s-gap waves, so the trigger threshold is low. This plausibly explains why no prior soak scenario in this spec's history (chaotic/sequential/force-snapshot, all using NJOBS=3 by default, un-paced but small) ever surfaced it: none combine a real kill-during-traffic failover with this much job+fact volume until raft-apply-decoupling task 7.2's factstream scenario. Also independently added test.c's print_jobinfo state_raw hex field (task 7.2) and confirmed via a live probe that apply_job_fail_transition correctly applies EXIT|PERR bits even when job_state_str's RUN-checked-first priority ordering hides them in the pretty-printed state string (job showed 'state: RUN' / 'state_raw: 0x106' after a jstart+jfail-without-jfinish sequence) -- a separate, minor, likely pre-existing (not introduced by raft-apply-decoupling) display-only quirk in job_state_str, also not fixed here (out of boundary), but which the factstream phantom-check was updated to work around by reading state_raw instead of the ambiguous pretty string.",
"fix": "FIXED (round 2, raft-apply-decoupling task 7.2 review remediation). Root cause confirmed exactly as hypothesized in the prior entry's candidate #2: persist_verify_apply_correctness() compared a WAL-scan-derived 'expected' snapshot against a LIVE re-read of last_applied_lsn/count_jobmgr_entries(), while TWO live mutation paths could write those same values concurrently -- the live apply pump (persist_apply_pump_run, driven by persist_apply_pump_kick off any of commit.c's 6 committed_lsn-advance sites) and the standby live-replication apply path (ha_apply_single_entry / ha_apply_term_entry_via_library in ha.c). Fixed via a new _Atomic bool persist_t.recovery_complete gate (persist.h/persist.c): persist_apply_pump_kick() now no-ops while the gate is closed; ha_apply_single_entry/ha_apply_term_entry_via_library now skip ONLY the in-memory apply_wal_entry() call while closed (durability write + replication ACK + applied_lsn advance are never gated, preserving the existing persist-before-apply-before-ACK contract and bug-1043's durability/materialization decoupling precedent). persist_recover() closes the gate at entry and reopens it (plus issues one deliberate persist_apply_pump_kick catch-up pass) right before returning, regardless of verdict -- covering any direct unit-test caller. ROUND 2 FINDING during live verification: this alone was insufficient -- a node stuck retrying its own election as a losing/timing-out candidate can receive and durably apply live-replicated entries via the ungated (persist_init default=true) path for tens of seconds BEFORE cluster_init_phase2() ever calls persist_recover() for the first time, reproducing the identical false-DIVERGENT symptom via an earlier entry point that persist_recover()'s own bracket cannot see. Closed by adding persist_mark_recovery_pending(&c.persist) in myapp_run() (myapp.c), called immediately after persist_init()/persist_start() succeed and BEFORE ha_start_election_async()/cluster_run() can process any network traffic -- closing the gate from the earliest safe moment in the real boot sequence. persist_init()'s default stays TRUE (ungated) so none of this spec's ~20 existing unit-test fixtures that construct a persist_t directly (bypassing myapp_run entirely) are affected; persist_mark_recovery_pending is never called outside myapp.c. Verified: live 3-node factstream soak (2 rounds, kill+restart both rounds) now PASSes clean with zero APPLY_DIVERGENT/UNRECOVERABLE across 5 persist_recover() calls (2 initial-boot delayed by the SAME pre-existing election-instability condition that originally exposed the round-2 gap, 2 post-kill restarts, all APPLY_CLEAN); broader controller_unit_test sweep (23 groups incl. test_apply_correctness_verification's 3 real-divergence tests, test_apply_pump/_integration, test_ha_core_safety, test_demotion_convergence) shows zero new failures (only the 2 pre-existing, documented task-6.1/6.2 test_commit_barrier_reply.c failures remain, task 7.3's job). Real-divergence detection independently re-confirmed intact (test_apply_divergent_decision_forces_unconditional_fence, test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive, test_real_replay_divergence_sets_unrecoverable_fence all still PASS).",
"tags": [
"raft-apply-decoupling",
"task-7.2",
"xce-test-ha",
"factstream",
"persist_recover",
"APPLY_DIVERGENT",
"false-positive-candidate",
"boot-race",
"concurrency",
"live-regression-found",
"fixed",
"recovery-complete-gate",
"myapp-boot-sequencing"
],
"related_bugs": [
"bug-1409",
"bug-1415"
],
"occurrences": 11,
"last_seen": "2026-07-05T07:45:00.000Z"
}
]

54
.wolf/cerebrum.md
View file

@ -9,6 +9,11 @@
<!-- How the user likes things done. Code style, tools, patterns, communication. -->
## Key Learnings
- (2026-07-04, raft-apply-decoupling task 6.1 round 2 — job-state merge-staleness fix) Round 1's `apply_committed_entry` wired the live pump but left a real corruption: `record_job_fail_fact`/`record_job_update_fact` (job.c) precomputed a FINAL absolute state at RECEIPT time (a scratch-copy merge for FAIL, an inline backstop+PDONE merge for postdone) and baked it into `wal_jobstate_t.state`; under deferred apply, a second fact for the same job (e.g. FINISH) could apply BETWEEN receipt and this fact's own apply, making the baked snapshot stale — a POST_EXEC-stage FAIL received while a job is still `RUN` (FINISH not yet applied) bakes `RUN|EXIT|PERR`, and when FINISH applies first (`DONE`) then FAIL's blind absolute overwrite clobbers it back to `RUN|EXIT|PERR` — DONE silently lost. FIX PATTERN: `wal_jobstate_t._pad` (previously always-zero, unused) is repurposed as a discriminator — `_pad==0` still means "legacy absolute state, safe" (job_ctrl's mark_job_fail/mark_job_update paths, which mutate+capture state synchronously under one lock, no staleness window); a nonzero `_pad` (`WAL_JOB_FAIL_STAGE_FLAG|stage` for FAIL, `WAL_JOB_UPDATE_POSTDONE_MARK` for postdone) marks "recompute at apply time against ACTUAL current state" — `apply_committed_entry` (persist.c) dispatches on this bit per WAL type, calling the REAL `apply_job_fail_transition`/new `apply_job_postdone_transition` (job.c, mirrors the existing guard-core family) instead of `apply_job_update_transition`'s blind overwrite. New low-level `persist_submit_job_fact_ex(p, type, jobid, state, pad)` added as the primitive; `persist_submit_job_fact` becomes a pad=0 thin wrapper — ZERO existing call sites (including ~20 in test_commit_barrier_reply.c) needed touching. GENERAL PATTERN for "wire format has 2 producers, one safe one not": don't change the safe producer's shape — add a discriminator field/flag so the consumer can tell them apart, and make "0"/"absent" mean exactly the old/safe behavior so every existing writer keeps working unchanged.
- (2026-07-04, same task) Fixing a wire-format bug INSIDE a task's own scope can break pre-existing tests that assert the wire format directly (not just behavior) — 2 tests in test_commit_barrier_reply.c (`test_guard_forward_update_on_terminal_preserved`, `test_guard_fail_after_finish_preserves_perr_merge`) read the raw on-disk `wal_jobstate_t.state` via a helper (`cbrp_count_jobstate_entries`) and asserted DONE/PDONE/PERR bits directly on it — exactly the field this fix stops populating for FAIL/postdone. These are NOT the "old apply-before-commit ordering" tests task 7.3 owns (a different axis — ordering vs wire-format) — they are a DIRECT, in-scope, mechanical consequence of the wire-format fix itself, so I fixed them here rather than deferring: extended the helper to a `_ex` variant returning `_pad` too (existing helper becomes a thin wrapper, zero other call sites touched — same pattern as persist_submit_job_fact_ex), changed the wire-level assertions to check the new `_pad` marker, and added a direct call to the guard core (`apply_job_fail_transition`/`apply_job_postdone_transition`) on a scratch `job_t` to keep pinning the actual merge semantics the test was written to protect. A 3rd test (`test_fact_state_unchanged_throughout_quorum_loss_outage`) had the identical `st_upd` bit-check for postdone; fixed the same way (already had a `(void)st_fail` precedent in the same file for exactly this "order-dependent, can't check at receipt time" reasoning — extended it to `st_upd`/`pad_upd`). Lesson: when a task's fix legitimately changes what a durable record contains, grep for every DIRECT byte/field-level test assertion on that record (not just behavioral tests) before declaring the regression sweep clean — `grep cbrp_count_jobstate_entries` found them, `--run test_apply_pump` alone would not have.
- (2026-07-04, same task) RED-phase verification for a bugfix already merged into the same commit as its own regression test (no separate "before" commit to check out) — reverted the exact 2 production hunks in-place (record_job_fail_fact's scratch-copy computation + apply_committed_entry's WAL_JOB_FAIL case ignoring `_pad`) using Edit, confirmed the new test fails with the EXACT predicted corrupted value (`Expected 264 Was 262` = DONE|PERR vs RUN|EXIT|PERR, byte-for-byte matching the hand-derived bug trace), then restored from a pre-saved `cp` snapshot of the fixed files and `diff`-verified byte-identical restoration before rebuilding GREEN. Combined with the existing longjmp-past-teardown lesson (isolate the new test alone in RUN_TEST via `#if 0` during the revert-and-check step, restore full list after) — both techniques compose cleanly.
- (2026-07-04, same task) **Do-not-repeat**: this repo has TWO independently-built `controller_unit_test` binaries — `src/build/controller/controller_unit_test` (correct, incremental, what `ninja -C build` from `src/` updates) and `../build/controller/controller_unit_test` i.e. repo-root `build/` (a STALE separate build tree from an earlier/different session — NOT updated by the `src/`-relative ninja command). Running `../build/controller/controller_unit_test` after rebuilding via `ninja -C build` silently executes stale, unrelated test binaries — symptom observed: a `--run test_apply_pump` invocation printed exactly 3 tests (the pre-round-2 test count) with `PASS` for all, at line numbers that didn't match the current file, right after a successful rebuild that should have produced 4 tests. Always verify with `strings build/controller/controller_unit_test | grep <new_test_name>` (or check `ls -la --full-time` / mtimes) when a test run's output looks suspiciously like an older version of the suite — don't trust "the build succeeded" alone to mean "the binary I'm about to run reflects it." ALWAYS run `./build/controller/controller_unit_test` (relative, from `src/`), never `../build/...`.
- (2026-07-04, same task) Observed a rare (~1/50 runs) flake in `test_apply_pump_fail_racing_finish_merges_against_current_state` where the final job state read back as the untouched SEEDED value (job never mutated at all) despite every prior assertion (including `committed_lsn == watermark`) passing — i.e. the covering ACK genuinely advanced `committed_lsn` but the inline (NULL-pool fallback) apply-pump walk apparently didn't mutate the job that pass. Reproduced 1 time in ~55 total runs across 3 batches; the identical ACK+inline-pump trigger mechanism is shared with round-1's already-reviewer-verified `test_apply_pump_kick_applies_committed_burst_in_ascending_lsn_order`, which never flaked in the same sampling — inconclusive whether this is environmental (system load during rapid repeated `nix develop` shell spin-up) or a genuine rare race in `persist_apply_pump_run`/`commit_resolve_pending`'s kick path (both explicitly out of this task's remit, verified-correct by the round-1 reviewer). Flagged in the status report as a CONCERN, not fixed — the fix logic itself (`apply_job_fail_transition`/`apply_job_postdone_transition`/the `_pad` dispatch) is pure and time-independent, so this does not implicate the round-2 correctness fix specifically.
- (2026-07-03, raft-apply-decoupling task 1.5) Wrote the concurrency unit tests closing task-group 1: `test_commit_resolve.c` gained 3 new tests (a genuine multi-threaded admission-cap burst using a sampling monitor thread; a mixed majority-ack+timeout burst using a `bounded_pool_setup` helper that DECOUPLES the continuation registry's capacity from the vtaskmgr pool's real worker-thread count so "bounded worker usage" is a meaningful, non-trivial assertion via `pthread_equal`-based distinct-thread counting; and a nice-to-have genuine two-real-thread majority-wins-over-timeout race relying on `commit_advance_locked`'s `pthread_cond_broadcast` to wake the sweep thread early). The demotion- and truncation-triggered burst cases were placed in `test_demotion_convergence.c` and `test_persist_truncate.c` INSTEAD of `test_commit_resolve.c`, reusing those files' existing `dc_setup`/`make_fresh_persist` fixtures rather than duplicating a full `cluster_t` fixture — keeps each file's fixture cohesive; design.md's file-plan wording ("test_commit_resolve.c covers demotion/truncation admission behavior") is better read as describing task-group 1's aggregate coverage, not a literal single-file mandate. IMPORTANT DISCOVERY: bug-1407 (`test_commit_advance_retry_clears_push_before_commit_fence`, test_config_repl.c), which task 1.4's notes closed as "70/70 passing, 3/3 runs", is REOPENED — it now fails 100% of the time (10/10), confirmed on BOTH the task-1.5 working tree AND a `git stash`-clean checkout of the task-1.4 commit with zero task-1.5 changes applied (i.e. genuinely pre-existing on the current tree, not caused by this task's test-only diff). Do not trust a prior task's "CLOSED, N/N runs" bug verdict at face value when validating a later task in the same task-group — always re-run the specific regression yourself before citing it as settled, especially for a concurrency-adjacent flake with no isolated root cause.
- (2026-07-03, raft-apply-decoupling task 1.4) Wired demotion + log-truncation to resolve pending commit continuations as FAILED. TWO call sites, TWO different idioms, deliberately: (1) **demotion** — `raft_cb_become_follower`'s existing `if (prior_role == CTRL_PRIMARY && c != NULL)` hook-disable block (ha.c) now also calls the new `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` directly — ha.c is already a direct consumer of commit.c's public API elsewhere (commit_set_writable_lsn etc.), so a direct call is idiomatic there; lsn_floor=0 means "all pending" (a demoted node can never commit ANYTHING it had pending — there is no partial-floor case, unlike truncation). (2) **truncation** — used the SAME inversion-of-control pattern as the existing durable-advance wake (`persist_set_durable_cb`/`on_durable_advance`) rather than calling commit.c from ha.c/persist.c's 3 real truncate call sites (`raft_cb_become_leader`'s post-election truncate, `ha_cb_truncate_from_stub`'s conflict-reconcile, `ha_demotion_convergence_fn`'s post-demotion truncate) individually: added `persist_set_truncate_cb`/`on_truncate`/`truncate_cb_ctx` to persist_t, registered by `commit_engine_init` (`commit_on_truncate` → `commit_resolve_pending_above(ce, cutoff_lsn, -ENOENT)`), fired ONCE from the single shared `persist_truncate_wal_above` wrapper (never `truncate_wal_above_locked`, which runs under `p->lock` — the callback must fire AFTER `pthread_mutex_unlock(&p->lock)`, checked via `after < before` on `current_lsn`, mirroring the flusher's `after > before` durable-advance guard EXACTLY) — this is DRY (one wiring point catches all 3 real+future truncate callers) and preserves the documented "persist.c never depends on commit.c" direction. `-XCE_ERR_STANDBY` (demotion) and `-ENOENT` (truncation) reuse the exact rc vocabulary `test_commit_resolve.c`'s task-1.2 author had already picked in `test_commit_resolve_delivers_failed_outcome_for_{demotion,truncation}` — do not invent new codes when an existing test file has already established the vocabulary for a not-yet-wired scenario. New commit.c/h function `commit_resolve_pending_above(ce, lsn_floor, rc)`: snapshot-then-resolve under `ce->lock` (mirrors `commit_resolve_pending`'s shape exactly), resolves every `lsn > lsn_floor`, exactly-once via `commit_resolve`'s own idempotent slot-clear.
- (2026-07-03, RED-verification hazard with Unity + threaded fixtures — reusable gotcha) Proving a genuinely-RED test in THIS harness by temporarily reverting production code and running the WHOLE test group (`--run <group>`) can HANG the process, even though the identical group passes cleanly once GREEN. Root cause: Unity's `TEST_ASSERT_*` macros `longjmp` out of the failing test function on the FIRST failed assertion, which SKIPS any teardown code written after that assertion in the same function (e.g. `dc_teardown()` at the end of a test in test_demotion_convergence.c) — so a thread-owning fixture (a `commit_engine_t` with its task-1.3 sweep thread, a persist flusher thread) is never joined/freed. The NEXT `RUN_TEST` in the same process then calls the SAME group's `setup()` which does `memset(&fixture, 0, sizeof(fixture))` on a struct a live orphaned thread still references — corrupting a mutex/cond mid-use and hanging (observed: 0% CPU, process alive, no output, indefinitely). FIX for RED verification (not a product bug — never manifests in GREEN, since teardown runs normally when assertions pass): temporarily edit the suite's `RUN_TEST` list down to JUST the one new test (wrap the rest in `#if 0`/`#endif`), rebuild, confirm the single isolated failure, then revert the RUN_TEST list back before restoring the production fix. Reusable rule: whenever a fixture under test owns a background thread AND you need to prove RED via a real (not calling-the-helper-directly) test, isolate that ONE test in the group during the revert-and-check step — do not run the full group against reverted production code.
@ -88,6 +93,7 @@
- (2026-06-13) OPERATIONAL: controller meson test wrapper ROUTINELY hits its 300s timeout under load (barrier commit-deadline waits) — `Timeout: 1` is NOT a failure. Authoritative green = direct binary `src/xce.build/controller/controller_unit_test` (EXIT 0, every group "N Tests 0 Failures"). A pre-existing ~153KB suite-wide LSan leak is SUPPRESSED by `ASAN_OPTIONS=detect_leaks=0` at controller/meson.build:163 — LSan exit=1 only appears on a RAW direct-binary run without that env; baseline, reproduces at pre-spec commits, not a per-change regression.
- (2026-06-14, reserve-lsn-0 / bug-543 FIX) LSN 0 is now RESERVED as the none/empty sentinel — `persist_init()` sets `current_lsn = 1` on a fresh store (persist.c, the ONLY fresh-init site; recovery overwrites from snapshot/WAL tip). First-ever WAL entry (promotion NOOP) is lsn 1. This was a ONE-LINE production change + a comment block; everything downstream ALREADY treated 0 as "none/empty" and lsn>=1 as real, so NO change was needed in commit.c (highest_appended_lsn `current_lsn>0?-1:0`, commit_submit rejects lsn 0, wait_done writable_lsn==0=gate-off), the streamer (`hdr.lsn>standby_lsn`: 1>0=true now streams the first entry → no hole), ha.c own-term gate (noop_lsn=current_lsn-1=1, writable_lsn=1 effective), or the wire/capnp format. The pre-alloc zero-region sentinel `hdr.lsn==0 && type==0 && payload_len==0` (persist.c:874, config_repl.c:1580, truncate:1666) is STRENGTHENED — no real entry is ever lsn 0. Segment naming follows current_lsn at open time, so the first segment is `seg-...001.wal` (not 000); recovery/truncate scan by sscanf, no hard-coded seg-000 assumption.
- (2026-06-14, reserve-lsn-0 TEST FALLOUT) Base-shift broke tests that did a "priming append so the entry lands at LSN >= 1" — that priming was ONLY to escape the old lsn-0 special-case and is now redundant. Fix = REMOVE the priming append so the real entry lands at lsn 1 (test_commit_engine tests 32/34, test_commit_barrier_rollout fact-hook, test_commit_barrier_reply cbrp_setup, test_p2_integration single_node) OR bump a hard-coded `current_lsn==1` assertion to `==2` if the priming must stay (test_egress_filter ef_setup). The ENGINE-LESS walAck path only "worked" via `repl_wait_majority_ack`'s `entry_lsn==0 → return 0` hatch (repl.c:437); with the first entry now lsn 1 it routes to `commit_submit(ce=NULL)` → -EINVAL, so engine-less commit tests MUST wire a commit engine (test_p2_integration test 9: added commit_engine_init(n_peers=3) + repl_set_commit_engine + commit_engine_fini). Tests asserting RELATIVE lsn (lsn_before+1, current_lsn>0, h1.lsn>h0.lsn) or byte-offset truncation math are base-shift-immune.
- (2026-07-03, raft-apply-decoupling task 3.3) `test_commit_barrier_reply.c`'s `cbrp_setup` fixture wires job submission's commit continuations through a SEPARATE standalone `commit_engine_t g_hce` (via `commit_engine_init(&g_hce, &g_hc.persist, ...)` + `repl_set_commit_engine`), NOT through `cluster_t`'s own embedded `c->commit` field. This matters because `raft_cb_become_follower`'s demotion branch (ha.c:1395) calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` — i.e. it operates on `c->commit`, the field `g_hce` deliberately bypasses. So `raft_cb_become_follower_with_cluster_for_test(&g_hc, term)` CANNOT be used to force a job-submission continuation FAILED in this file's harness — it would touch the wrong (never-initialized, all-zero) engine. `persist_truncate_wal_above(&g_hc.persist, cutoff)` has no such problem: `persist_set_truncate_cb` is wired directly by `commit_engine_init` onto whichever engine you pass it (here, `g_hce`), so truncation is the ONLY viable way to force a deterministic, explicit non-timeout FAILED outcome (`-ENOENT`) for a job-submit continuation in this specific fixture. Added 4 tests exercising this (single + repeated truncation-forced FAILED, repeated timeout-forced FAILED, array mixed-outcome) — all GREEN on first run (no bug found; task 3.2's failure-path guarantee holds). Generic demotion-resolve-continuations-FAILED behavior already has its own dedicated coverage in `test_demotion_convergence.c` (task 1.4/1.5), which uses `ha_promote_to_primary` + the cluster-aware `_for_test` seam against a REAL `c->commit`-shaped fixture — reuse THAT file, not this one, for anything demotion-specific.
- (2026-06-13, kiro-impl process) A RED-only test pin is only valid if its OWN impl task can turn it green via the SAME seam the test drives. Misfires caught in review: boot-replay pin RED on an lsn-0 `replay_wal` skip artifact; hooks pin via the `c==NULL` raft shim where `ha_register_persist_hooks(&c->hooksys)` can never run (re-point to the real-cluster `ha_promote_to_primary` seam). Deleting machinery → DELETE the machinery-subject tests (design "no shims, rewritten or deleted not stubbed"), never add file-local no-op stubs. Unity runner prints the PREVIOUS passing test's file:line as a stale prefix on FAIL lines → attribute failures by test NAME.
## Do-Not-Repeat
@ -369,3 +375,51 @@ Swept 7 old "not yet fixed" buglog entries (846, 847, 855, 1096, 1147, 1198, 130
## Key Learning (2026-07-03, bug-847 investigation) — trace the actual library mechanism before trusting a plausible-sounding root-cause hypothesis
A prior investigation (and a fresh subagent re-investigation) both converged on "the encoder captures a capnp segment pointer once outside a loop and reuses it stale across iterations" as bug-847's root cause — plausible-sounding, cites real code (`cs = capn_root(&c).seg;` before a `for` loop in `xce_wal_batch_encode_raw`, src/lib/request.c). Tracing the ACTUAL c-capnproto library (`new_object`/`new_data`/`write_ptr`/`capn_setp` in the vendored `capn.c`) showed this is a non-issue by design: `new_object` always re-resolves and re-tags `.seg` on segment overflow regardless of what stale segment pointer was passed in, and `write_ptr`/`capn_setp` correctly emit far/double-far pointers whenever the source and target objects land in different segments. The theory was never tested against the library source, only against the observed symptom — and the observed symptom (RED test failure) turned out to no longer reproduce at all once actually run. Lesson: a hypothesis that "sounds mechanistically right" and cites real line numbers is not the same as a verified root cause — for a data-loss bug specifically, always run the existing/write a NEW regression test before accepting a hypothesis, even one two independent investigations agreed on.
## Key Learning (2026-07-04, raft-apply-decoupling task 5.3) — `recovery_unrecoverable`'s clear path is LIVE-catch-up-only, not recovery-driven; apply_divergent has no live self-heal yet
`persist_clear_recovery_unrecoverable()` (persist.c) has ZERO call sites inside persist.c itself — its only two callers are in ha.c (`promotion_fence_timeout_fn` ~ha.c:867, and the promotion-eligibility gate ~ha.c:1115), both gated on STRUCTURAL getters (`persist_recovery_truncated`/`persist_durable_floor` vs cluster HWM) that can genuinely change while a process stays up (a fenced standby keeps receiving replicated WAL from a healthy peer). `persist_recover()`'s own APPLY_CLEAN branch never calls the clear function — a second, later `persist_recover()` call reaching APPLY_CLEAN does NOT auto-clear a sticky flag set by an earlier APPLY_DIVERGENT verdict on the same object. This is correct by design (task 5.2's decision table fences apply_divergent unconditionally specifically because more WAL bytes/catch-up can't prove a local apply-dispatch defect is fixed) but means: today, the apply_divergent-triggered fence has no LIVE self-heal analogue to the structural fence's catch-up-timer release — it only "clears" via a full process restart into a fresh `persist_t` (persist_init defaults `recovery_unrecoverable=false`) that then replays cleanly. Task 6 (ordered apply pump) is where a live re-verification analogue would need to be built if ever wanted. Don't assume "the fence auto-clears like the structural one" for this trigger without checking which mechanism (live re-check vs restart) actually applies.
## Key Learning (2026-07-04, raft-apply-decoupling task 5.3 round-2, CLOSED) — the trigger-agnostic sticky-fence bug: fix the guard on BOTH live-clear call sites, not just one
Round-1 review found a CRITICAL gap: `persist_recovery_unrecoverable` is a bare boolean with no provenance about WHY it was set — task 5.2 added `apply_divergent` as a new trigger reusing the existing sticky flag, but the flag's two pre-existing LIVE clear call sites (`promotion_fence_timeout_fn` ha.c:~867, and `raft_cb_become_leader`'s promotion-eligibility gate ha.c:~1109) both only ever checked STRUCTURAL conditions (`persist_recovery_truncated`/`persist_durable_floor` vs HWM) — never the apply verdict. Since a WAL can be structurally perfectly healthy (not truncated, durable_floor>=HWM) while the local apply-dispatch path is still defective (APPLY_DIVERGENT — the two are orthogonal by construction), the structural timer would eventually observe "healthy" and silently clear an apply-divergence fence, letting a genuinely divergent node get promoted on its NEXT election. FIX (additive, ~30 lines, both call sites in ha.c): added `bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;` at each site and ANDed `!apply_divergent` into the existing enclosing `if` condition that gates the clear (`if (!truncated && !below_hwm && !apply_divergent)` in the timer; `if (gate==ELIGIBLE && !unrecoverable && !apply_divergent)` in the promotion gate). **Key design insight**: gating just the bare `persist_clear_recovery_unrecoverable(...)` call-statement itself is NOT sufficient at the become_leader site — the safety-critical action there is falling through to `CTRL_PRIMARY` promotion, not the clear call (which in production can only be reached when `unrecoverable` already reads false); you must extend the ENCLOSING branch condition that also gates promotion, or the fix is a no-op that still promotes a divergent node while merely skipping a redundant flag-clear. Added to `promotion_fence_timeout_fn`'s guard anyway (timer site) since it DOES flip true→false there and is the actually-exploitable half. Since `apply_verdict` is only ever recomputed by `persist_recover()` (boot-time-once, single call site `cluster.c:1451`), this guard means an apply-divergent fence can now ONLY clear via a full process restart that replays cleanly — never a live timer/gate. Regression test `test_promotion_fence_apply_divergent_survives_structural_catchup` (test_commit_durability.c) mirrors `test_promotion_fence_released_on_catchup`'s exact harness/timing, differing only in also forcing `c->persist.apply_verdict = APPLY_DIVERGENT` + `recovery_unrecoverable=true` up front — confirmed RED (fence wrongly cleared) against the pre-fix code, GREEN after. All 3 required suites pass with 0 regressions: test_commit_durability 39/39, test_apply_correctness_verification 11/11, test_ha_core_safety 28/28. [[project_bug1060_leader_completeness]]
## Do-Not-Repeat (2026-07-04, sticky-flag-with-multiple-triggers pattern)
When a shared sticky/latching flag gains a NEW trigger reason (reusing an existing mechanism, as instructed by a design doc's "don't build a new mechanism" guidance), always audit EVERY existing call site that CLEARS or CHECKS that flag for whether it silently assumes the OLD trigger's semantics (e.g. "clearable once structural conditions are healthy"). A flag with no provenance field is only as safe as the least-restrictive clear site — adding a new, stricter-to-clear trigger without updating every clearer creates exactly this class of bug. Grep for all callers of the clear function FIRST (`grep -n "persist_clear_recovery_unrecoverable" src/controller/ha.c` found exactly 2, matching the reviewer's claim precisely) rather than trusting a blast-radius/paraphrased caller count.
## Key Learning (2026-07-04, raft-apply-decoupling task 5.3 round-3, CLOSED) — a logically-correct guard with a green suite can still have zero coverage; prove it by reverting the guard, not by reading the code
Round-2's `!apply_divergent` conjunct at the `raft_cb_become_leader` promotion-eligibility gate (ha.c:1130) was independently verified LOGICALLY correct twice, yet reverting ONLY that conjunct left all 298 pre-existing tests green — no test ever drove a node that was simultaneously (a) structurally ELIGIBLE (`ha_promotion_gate_decision` genuinely returns `RECOVER_DECISION_ELIGIBLE`), (b) NOT sticky-fenced (`persist_recovery_unrecoverable()==false`), AND (c) `apply_verdict==APPLY_DIVERGENT`. Closed by adding `test_promotion_refused_when_apply_divergent_even_if_eligible` to `test_convergence_verification.c`, reusing `test_promotion_registers_standbys_when_peer_set`'s exact `cv_setup` fixture (n_peers=2, n_standbys=0, appended_lsn=0 — this fixture is PROVEN to reach genuine ELIGIBLE because that sibling test's own assertion, `repl_active_standby_count==2` after promotion, is unreachable unless the gate let promotion through) and forcing only `c->persist.apply_verdict = APPLY_DIVERGENT` before calling `raft_cb_become_leader_with_cluster_for_test`. RED→GREEN confirmed by temporarily deleting the `&& !apply_divergent` conjunct in-place, rebuilding, observing exactly 1 failure (the new test, on the promotion-refused assertion), then restoring and reconfirming 0 failures. Full 13-group / 299-test regression (all files touching `raft_cb_become_leader`) stayed green throughout. **Lesson**: "the reviewer traced the boolean logic and it's correct" and "a regression test exists that would catch a revert of this exact line" are different claims — always close a coverage gap by demonstrating the RED failure, not by re-reading the code once more. **Environment note**: contrary to a round-2 flag about a possible openssl/flake.nix issue in this worktree, `nix develop --command bash -c "ninja -C ../build ..."` from `src/` built cleanly with no header-path problems — that concern did not reproduce here.
## Key Learning (2026-07-04, raft-apply-decoupling task 6.1, CLOSED) — the "detail-filter" fix for a dual-origin hook detail is a trap; use call-provenance instead
Hazard 1 (task 4.1's double-append warning) named `HOOK_DETAIL_JOB_UPDATE` as the risky detail once `apply_committed_entry()` goes live, and the obvious-sounding fix is "early-return `persist_job_hook_fun` when `detail == HOOK_DETAIL_JOB_UPDATE`". That fix is WRONG: `job_ctrl`'s SIGCONT/SIGSTOP path (`mark_job_update`, job.c:498/508) fires the IDENTICAL detail for its own still-legitimate live append, and a detail-based filter would silently break job-suspend/resume durability with zero test failure to catch it (no existing test exercises job_ctrl's WAL append in isolation). The correct fix is a call-PROVENANCE guard: a `_Thread_local bool t_apply_seam_notifying` set only around `apply_committed_entry()`'s own `xce_run_hooks()` calls, checked by `persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun`. Thread-local (not a shared `persist_t` field) because a shared flag races across the apply-pump thread and any concurrent live command thread touching a DIFFERENT job. Lesson: when a hook detail is dual-origin (one AFTER_EVENT firing point that both a legacy live-command path and a new apply-seam path share), filtering on the detail value is never safe — filter on which CALL STACK is asking.
## Key Learning (2026-07-04, raft-apply-decoupling task 6.1) — commit_resolve_pending is the ONE choke point for ALL SIX committed_lsn-advance paths; wiring a new trigger only into ha.c's on_commit_advanced silently starves single-node/self-quorum clusters
design.md's File Structure Plan says to wire the live apply pump from ha.c's `on_commit_advanced` callback. That callback is fed by exactly ONE of the six commit.c-internal paths that can move `committed_lsn` (durable-advance, self-quorum `commit_submit`, ack-driven `commit_on_ack`, the writable-lsn gate, the raft-library commit-index feed, membership change) — task 1.2 already discovered and consolidated all six into `commit_resolve_pending()`. Wiring the apply-pump kick ONLY into ha.c would leave 5 of 6 paths never triggering it — concretely, most unit-test fixtures (n_peers<=1 self-quorum, no raft library wired) would NEVER see the pump fire. Wired the kick into `commit_resolve_pending()` itself instead (one new line, `persist_apply_pump_kick(ce->p, ce->pool)`, after its existing resolve loop) — a documented, deliberate deviation from design.md's literal text, justified by this discovery. Lesson: when a design doc names ONE integration point for "whenever X changes," check whether X actually has multiple internal advance paths before trusting that single named point is complete — grep for every writer of the field.
## Do-Not-Repeat (2026-07-04, raft-apply-decoupling task 6.1) — a retry-until-caught-up drain loop MUST distinguish "no more work" from "the same failed work is still there"
`apply_pump_drain()`'s first version cleared the scheduled flag, rechecked `last_applied_lsn+1 <= committed_lsn`, and reclaimed+looped if true — without checking whether the PRECEDING `persist_apply_pump_run()` call actually made progress. When a WAL read genuinely fails (e.g. a test fixture advances `committed_lsn` without real WAL bytes on disk for that LSN), `last_applied_lsn` never advances, so the recheck ALWAYS sees identical unresolved work and reclaims the flag forever — an unbounded tight busy-loop (18.5M+ log lines before a 90s timeout killed the process in `test_demotion_convergence`). Fixed by checking the walk's return code immediately: a negative rc returns without rechecking or reclaiming, so a genuinely new future kick (a distinct commit-advance event) is what retries, never the drain loop's own tail. Lesson: any "loop until caught up, recheck before exiting to avoid a lost wakeup" idiom (task 1.3's precedent) MUST also handle "the loop body failed and made zero progress" as a THIRD outcome, distinct from both "caught up" and "genuinely more work" — otherwise the recheck-for-lost-wakeup safety mechanism itself becomes an infinite-retry hazard the moment the underlying operation can fail.
## Key Learning (2026-07-04, raft-apply-decoupling task 6.1) — Hazard 2 (merge-staleness) is real, immediately exploitable under ordinary traffic, and requires a wire-format change; it was escalated as BLOCKED, not silently shipped
Traced `apply_job_start_transition` (`job->job.state = JOB_STAT_RUN`, a full overwrite to a FIXED constant, discarding any other bits) and confirmed different job-state fact types need genuinely different apply semantics: START overwrites-to-fixed-value, FINISH/FAIL/UPDATE are guard-then-set, RESUME needs to CLEAR a bit (not just OR one in). A naive fix (change the shared `apply_job_update_transition` core from overwrite to OR-merge) would silently break RESUME's clear semantics on both replay and any live re-apply — trading one correctness bug for another. Since the wire format (`wal_jobstate_t{jobid,state}`) carries ONLY the final computed value, not which bits a given fact intends to set vs. clear vs. leave alone, no in-boundary fix exists without either (a) a wire-format change (delta/mask encoding) or (b) a receipt-time per-job serialization gate (a much bigger architectural change, arguably violating Req 11.4's "no per-fact synchronous barrier"). Escalated as a BLOCKED-severity, carried-forward finding in task 6.1's status report rather than either ignoring it or shipping a wrong compensating fix — mirrors task 4.1's own precedent of carrying forward a documented, unfixed hazard to the next task rather than guessing at scope it doesn't own.
## Key Learning (2026-07-04, raft-apply-decoupling task 6.2 round 2, bug-1415 FIXED) — self-heal a late registration by checking the already-applied watermark; reproduce the multi-vcpu race in a test by forcing the interleaving with a poll-barrier, not a sleep
Fixed via the self-healing option this same entry's original write-up proposed: `persist_register_pending_apply()` (persist.c) now checks `lsn <= persist_last_applied_lsn(p)` BEFORE linking onto `pending_applies`; if the walk already passed `lsn`, it fires `on_applied(ctx, lsn)` immediately, synchronously, on the CALLER's thread instead of orphaning a registration nothing will ever revisit. Proving this deterministically required a genuinely new test-fixture pattern for this codebase: `vtaskmgr_init()` alone starts NOTHING (no dispatcher, no vcpu threads) — only `vtaskmgr_start()` spawns them, so `test_commit_admission_pool.c`'s existing pool fixture (admission-accounting only) is NOT sufficient for a test that needs real concurrent vtask execution; you must call both. To force the specific bad interleaving (apply-pump wins) deterministically rather than relying on a timing-guessed `usleep()`, the test's stand-in commit-continuation callback blocks in a poll loop on `persist_last_applied_lsn(p) < lsn` BEFORE calling `persist_register_pending_apply()` — this is a genuine barrier tied to the actual race predicate (not a sleep guess), so it reproduces the exact interleaving on every run once a real multi-vcpu pool is wired via `commit_set_pool`. New test `test_apply_pump_late_pending_apply_registration_self_heals` (test_apply_pump.c) is the ONE test in this entire spec that wires a real pool; every other fixture deliberately keeps `ce->pool == NULL` and must stay that way (verified via full regression sweep: zero new failures, only the 2 pre-existing task-7.3-owned `test_commit_barrier_reply.c` failures persist unchanged). [[project_raft_commit_safety_done]]
## Key Learning (2026-07-04, raft-apply-decoupling task 6.2, historical — original bug-1415 finding, now fixed above) — two vtask dispatches enqueued back-to-back from the same function have NO completion-order guarantee once a real (n_vcpus>1) pool is wired
Wired `persist_apply_pump_run()` to consume task 3.2's hand-off (`persist_take_pending_apply()` immediately after each entry's `apply_committed_entry()` + `last_applied_lsn` advance, then invoke the returned continuation) and made `jobsubmit_apply_complete` actually send the batch's success reply (`RequestReply_data_jobid`, gated on `actx->replied` the same way `jobsubmit_reply_failure_once` already gates the failure reply — added a `jobsubmit_reply_success_once` sibling and a `jobid` field on `jobsubmit_async_ctx_t`). While tracing correctness end-to-end, found (not fixed — out of this task's boundary, which is scoped to `persist_apply_pump_run`'s loop only) that `commit_resolve_pending()` (commit.c) enqueues TWO independent `VTASK_LIVE_BLOCK` tasks back-to-back onto the SAME `vtaskmgr_t` pool: first each ready continuation's dispatch (which, for a job submit, calls `persist_register_pending_apply()`), then `persist_apply_pump_kick()`'s own walk task. `vtaskmgr.c`'s dispatcher (`vdispatcher_entry`/`pick_idle_vcpu`) hands queued tasks to whichever of `n_vcpus` worker threads is next idle — with more than one idle vcpu, two tasks enqueued in that order can run on two DIFFERENT threads with no guarantee the first-enqueued one finishes first. If the apply-pump task wins, it applies the entry and advances `last_applied_lsn` past it BEFORE the registration exists; the registration then lands for an lsn the walk will never revisit (it only ever walks `[last_applied_lsn+1..]` forward) — a silently, permanently orphaned reply. **This is invisible in every test in this entire spec** because every fixture (`tap_setup`, `cbrp_setup`, etc.) deliberately never calls `commit_set_pool`, so `ce->pool` stays NULL and `dispatch_continuation`/`persist_apply_pump_kick` both fall back to running INLINE, synchronously, in enqueue order — a single-threaded test can never observe a multi-worker race that only exists when a pool is actually wired. Logged as bug-1415 (not fixed) with two candidate fixes (make the pump-kick wait for its batch's continuation dispatches to finish first, or make `persist_register_pending_apply()` self-heal by checking `lsn <= persist_last_applied_lsn()` and firing inline immediately if so) — flag for whoever builds a real multi-vcpu regression test (task 6.3/6.4 or a fast-follow) to close it. **Lesson**: "the test fixture never wires a pool" is a load-bearing fact worth stating explicitly whenever reasoning about ordering between two `vtask_enq_classified` calls — pool-less inline fallback silently converts an async multi-threaded dispatch into a same-thread synchronous one, which can hide a real race for an entire task's worth of otherwise-thorough tests. [[project_raft_commit_safety_done]]
## Key Learning (2026-07-05, raft-apply-decoupling task 6.3) — a test spy for a hook-fan-out reaction must mirror the REAL routing table, not the idealized one
`apply_committed_entry()` fires every job-state fact via the generic `HOOK_DETAIL_JOB_UPDATE` only (never the type-specific `HOOK_DETAIL_JOB_START/_EXIT/_FAIL/_RUSAGE_UPDATE` — a documented wire-format deviation since task 2.2, persist.h's doc comment). Consequence for test-writing: `sched_job_start_fun`/`sched_job_exit_fun`/`sched_job_fail_fun` (sched.c) and their apiserver.c equivalents are structurally UNREACHABLE for any live-applied fact. A reaction spy registered for the type-specific details would silently test a code path that can never fire in production — passing vacuously, catching nothing. The correct spy listens ONLY for `HOOK_DETAIL_JOB_NEW` (submit) and `HOOK_DETAIL_JOB_UPDATE` (every fact type), since that is the only pair actually reachable. Verified this file's tests are NOT vacuous by temporarily reintroducing the pre-task-4.1 eager receipt-time mutation+notify bug in `on_job_start` (job.c) — both new integration tests failed immediately (double-append from the reintroduced live-notify racing `persist_job_hook_fun`'s durable-append guard), confirmed clean revert (`git diff` empty), then GREEN again, 30/30 stable reruns.
## Do-Not-Repeat (2026-07-05) — `TEST_ASSERT_EQUAL_INT_MESSAGE` requires 3 args; a bare 2-arg call is a compile error, not a silent no-message pass
Caught immediately by the build (gcc: "macro requires 3 arguments, but only 2 given"), not by review — worth remembering `TEST_ASSERT_EQUAL_INT` (no `_MESSAGE` suffix) is the correct macro when no custom message is needed, in this Unity-based C test harness.
## Key Learning (2026-07-05, raft-apply-decoupling task 7.2) — xce_test's `worker` REPL context was missing jstart/jfail; libxce already had the full wire API
`src/lib/client.c` already implements the ENTIRE worker-fact wire API (`xcecli_notify_jobstart/_jobfail/_jobstop/_jobresume/_jobchkpt/_jobpostdone`, alongside the pre-existing `_jobfinish`) — but `testsuite/test.c`'s `worker_commands[]` table only ever exposed `jfinish` as a REPL command. Added `jstart`/`jfail` (mirroring `jfinish`'s exact getopt/parse_jobid/SET_ERROR shape) to unlock a synthetic-but-real-wire-protocol worker-fact stream for the new `--factstream` HA soak scenario, without needing a real `xcecd_worker` + scheduler dispatch handshake — `on_job_start`/`on_job_fail` (job.c) only require the job to already exist in jobmgr (i.e. be submitted), not to have been dispatched, so this is genuinely the SAME production receipt→commit→apply path a real worker would drive, just skipping actual OS-process execution (irrelevant to raft-apply-decoupling's apply-ordering concern). **Do-not-repeat**: `cmd_jobfinish` never reset `optind = 1` — harmless for every EXISTING caller (always invoked at most once per xce_test process) but a REAL bug once a script calls `jfinish` repeatedly in one session (as the factstream scenario does): getopt's static `optind` carries over from whatever position the PRIOR command's parse left it at, silently causing the 2nd+ call to skip its own `-j`/`-c` flags. Fixed by adding `optind = 1` to `cmd_jobfinish` (matching the majority convention most other `cmd_` handlers already follow) — any new REPL command that might be invoked repeatedly in one script MUST reset `optind` itself; don't assume the previous command's handler already did it for you.
## Key Learning (2026-07-05, raft-apply-decoupling task 7.2) — waldump only decoded jobid for WAL_JOB_SUBMIT; extended it to the flat wal_jobstate_t fact types
`testsuite/waldump.c` only ever decoded `jobid`/`state` for `WAL_JOB_SUBMIT` (capnp Job schema) — every fact type (`WAL_JOB_START/FINISH/FAIL/STOP/RESUME/UPDATE/RUSAGE/CKPT/PREASON`, all sharing the flat native-endian `wal_jobstate_t{jobid,state,_pad}` struct, wal_types.h) printed as `jobid=?`. Added `wal_type_is_jobstate()`/`decode_jobstate_info()` (a bounds-checked `memcpy`, not a parser — same triviality as the struct itself) so these types now print `jobid=<fmt> state=0x%x pad=0x%x` too, letting a caller grep waldump output per-jobid per-fact-type across nodes. This is what makes the factstream scenario's phantom-fact cross-check possible (grep -F "JOB_START" + "jobid=$jid " per node). Generically useful beyond this task: any future "did fact X actually commit for job Y" question can now be answered offline via waldump alone.
## Key Learning (2026-07-05, raft-apply-decoupling task 7.2) — `job_state_str`'s if/else-if priority ordering (RUN checked before EXIT) HIDES bits from the pretty-printed state string
`apply_job_fail_transition` (job.c) merges `JOB_STAT_EXIT|JOB_STAT_PERR` onto the job's CURRENT state via `|=` — it does NOT clear `JOB_STAT_RUN`. So a job that receives a POST_EXEC-stage FAIL WITHOUT a prior FINISH (a legitimate worker-report ordering — "died without ever reporting finish") ends up with `state_raw = RUN(0x2)|EXIT(0x4)|PERR(0x100) = 0x106`, genuinely correct per the merge semantics — but `job_state_str()`'s `if (RUN) return "RUN"` check fires FIRST and never reaches the `EXIT` branch, so `jobinfo`/`joblist` print "RUN", silently hiding that the fail was actually applied. Confirmed via a live standalone probe (3-node cluster, `jstart`+`jfail` with no `jfinish`): `state: RUN` / raw `0x106`. This is a pre-existing, minor, likely-harmless-until-now display quirk (not introduced by raft-apply-decoupling — `apply_job_fail_transition` is an unmodified lift from the original `mark_job_fail`), NOT fixed (out of task 7.2's boundary), but it DOES matter for any script-level check that infers "did a terminal fact apply" from the pretty string — the factstream phantom-check was written against `state:` first and produced a false negative (silently failed to check for a committed JOB_FAIL backing a RUN-displayed job) until switched to a new `state_raw: 0x%x` field added to `print_jobinfo()` (test.c) and bitwise-AND'd against the real `JOB_STAT_*` constants. **Do-not-repeat**: never infer "which facts applied" from a job's pretty-printed state string in this codebase — always use `state_raw` (or the raw bits directly) since the string's priority-ordering can mask co-set bits.
## Key Learning (2026-07-05, bug-1416, raft-apply-decoupling task 7.2) — cluster_init_phase2's persist_recover() apply-correctness verification can false-positive APPLY_DIVERGENT when a returning node's recovery vtask races a compressed catch-up replication burst — FIXED round 2
Found live, reproduced 10/10 times, building the `--factstream` kill-mid-fact-stream HA soak scenario. `cluster_init_phase2()` (cluster.c) calls `cluster_open_listener()` (accepts network I/O, including replicated/catch-up WAL writes) BEFORE `persist_recover()` (the task-5.1 boot-time apply-correctness verification, Req 8.1/8.2) — and `persist_recover()` itself fires from a vtask, not synchronously. `persist_verify_apply_correctness()`'s LIVE reads of `p->last_applied_lsn` and `count_jobmgr_entries()` can race two live mutation paths (the apply pump, `persist_apply_pump_run`/`persist_apply_pump_kick`; and the standby live-replication apply path, `ha_apply_single_entry`/`ha_apply_term_entry_via_library` in ha.c) that write those SAME values concurrently, producing a DIVERGENT verdict on a node that has NOT actually diverged. **FIXED (round 2, review remediation)**: new `_Atomic bool persist_t.recovery_complete` gate (persist.h/persist.c). `persist_apply_pump_kick()` no-ops while closed; `ha_apply_single_entry`/`ha_apply_term_entry_via_library` skip ONLY the in-memory `apply_wal_entry()` call while closed (durability write + ACK + applied_lsn advance stay ungated — mirrors bug-1043's durability/materialization decoupling). `persist_recover()` closes it at entry, reopens it + issues one deliberate catch-up kick before returning. **Round-2-of-the-fix gap, found via live verification, not just code review**: this alone was NOT enough — a node stuck retrying its own election as a losing/timing-out candidate can receive and durably APPLY live-replicated entries via the ungated (`persist_init` default=true) path for tens of seconds BEFORE `cluster_init_phase2()` ever calls `persist_recover()` for the first time, reproducing the identical false-DIVERGENT symptom through an earlier entry point `persist_recover()`'s own bracket can't see. Closed with `persist_mark_recovery_pending(&c.persist)` in `myapp_run()` (myapp.c), called right after `persist_init()`/`persist_start()` succeed and BEFORE `ha_start_election_async()`/`cluster_run()` can process any network traffic — closing the gate from the true earliest safe moment. `persist_init()`'s default stays TRUE so none of this spec's ~20 unit-test fixtures that construct a `persist_t` directly (bypassing `myapp_run`) are affected. Verified: live 3-node `--factstream` soak (2 rounds, kill+restart both) now PASSes clean, zero `APPLY_DIVERGENT` across 5 `persist_recover()` calls; broader `controller_unit_test` sweep (23 groups) shows zero new failures; real-divergence detection re-confirmed intact via the 3 dedicated `test_apply_correctness_verification.c` divergence tests. **Do-not-repeat lesson**: when gating a live path behind a "recovery in progress" flag, the flag's OWN bracket around the recovery function itself is not sufficient if OTHER code paths (election retry, live replication) can mutate the same state BEFORE that function is ever first called — the gate must close at the EARLIEST point in the real process lifecycle traffic could possibly begin, not just around the specific function whose output it protects.

513
.wolf/hooks/_session.json
View file

@ -1,217 +1,470 @@
{
"session_id": "session-2026-07-02-1655",
"started": "2026-07-02T23:55:20.704Z",
"session_id": "session-2026-07-05-2050",
"started": "2026-07-05T03:50:02.712Z",
"files_read": {
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/requirements.md": {
"count": 3,
"tokens": 4629,
"first_read": "2026-07-05T05:13:49.214Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md": {
"count": 12,
"tokens": 20719,
"first_read": "2026-07-05T05:14:01.912Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/design.md": {
"count": 1,
"tokens": 9210,
"first_read": "2026-07-05T05:14:02.728Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c": {
"count": 4,
"tokens": 28490,
"first_read": "2026-07-05T05:14:31.226Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft.h": {
"count": 1,
"tokens": 17581,
"first_read": "2026-07-05T05:15:27.928Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T05:20:15.107Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md": {
"count": 4,
"tokens": 4553,
"first_read": "2026-07-05T05:28:34.593Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh": {
"count": 18,
"tokens": 18920,
"first_read": "2026-07-05T05:28:47.075Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c": {
"count": 8,
"tokens": 28321,
"first_read": "2026-07-02T23:58:08.847Z"
"count": 12,
"tokens": 29726,
"first_read": "2026-07-05T05:29:07.259Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/job.c": {
"count": 5,
"tokens": 16424,
"first_read": "2026-07-05T05:30:09.491Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client.c": {
"count": 2,
"tokens": 0,
"first_read": "2026-07-05T05:30:41.791Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/cluster.c": {
"count": 4,
"tokens": 18098,
"first_read": "2026-07-05T05:30:54.661Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client_internal.h": {
"count": 2,
"tokens": 0,
"first_read": "2026-07-05T05:31:49.690Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/sched.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T05:32:31.884Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c": {
"count": 6,
"tokens": 5404,
"first_read": "2026-07-05T05:33:41.172Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/wal_types.h": {
"count": 2,
"tokens": 1996,
"first_read": "2026-07-05T05:33:51.925Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xce/types.h": {
"count": 3,
"tokens": 0,
"first_read": "2026-07-05T05:34:30.060Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/soak-verdict.sh": {
"count": 3,
"tokens": 1383,
"first_read": "2026-07-05T05:37:16.616Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c": {
"count": 20,
"tokens": 60786,
"first_read": "2026-07-05T06:14:23.860Z"
},
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/full.diff": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-05T06:47:57.965Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/utils.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-02T23:58:46.397Z"
"first_read": "2026-07-05T06:51:04.731Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/build/lib/controller.capnp.c": {
"count": 8,
"tokens": 0,
"first_read": "2026-07-02T23:58:53.910Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/xcectx.c": {
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/soak.diff": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-02T23:59:35.188Z"
"first_read": "2026-07-05T06:59:21.281Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/utils.h": {
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
"count": 9,
"tokens": 102306,
"first_read": "2026-07-05T07:06:14.209Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h": {
"count": 5,
"tokens": 19990,
"first_read": "2026-07-05T07:09:10.020Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/xcectx.h": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-02T23:59:56.995Z"
"first_read": "2026-07-05T07:12:14.838Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/client.h": {
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/vtaskmgr.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-03T00:00:50.948Z"
"tokens": 3822,
"first_read": "2026-07-05T07:19:54.733Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/client.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-03T00:01:50.797Z"
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/controller_test.c": {
"count": 2,
"tokens": 1986,
"first_read": "2026-07-05T07:25:28.359Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/tools/test-commit-durability.sh": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-03T00:03:51.694Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_ctrlstatus.c": {
"count": 3,
"tokens": 4804,
"first_read": "2026-07-03T00:07:50.578Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_wal_wire.c": {
"count": 3,
"tokens": 4273,
"first_read": "2026-07-03T02:58:53.775Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/request.c": {
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/myapp.c": {
"count": 4,
"tokens": 15919,
"first_read": "2026-07-03T02:59:46.887Z"
"tokens": 5215,
"first_read": "2026-07-05T07:31:53.169Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/src/schema/controller.capnp": {
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_apply_correctness_verification.c": {
"count": 4,
"tokens": 17000,
"first_read": "2026-07-05T08:08:11.151Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md": {
"count": 1,
"tokens": 9344,
"first_read": "2026-07-03T03:00:35.357Z"
"tokens": 380,
"first_read": "2026-07-05T08:09:36.337Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/build/lib/controller.capnp.h": {
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/tacv_out.log": {
"count": 1,
"tokens": 0,
"first_read": "2026-07-03T03:00:48.039Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-single-leader-role/spec.json": {
"count": 1,
"tokens": 142,
"first_read": "2026-07-03T03:19:29.123Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-convergence-recovery/spec.json": {
"count": 1,
"tokens": 142,
"first_read": "2026-07-03T03:19:43.313Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-consistency-invariants/spec.json": {
"count": 1,
"tokens": 259,
"first_read": "2026-07-03T03:19:58.314Z"
},
"/home/rshen/projects/xce/worktrees/raft_tla/.gitignore": {
"count": 1,
"tokens": 131,
"first_read": "2026-07-03T03:23:24.225Z"
"first_read": "2026-07-05T12:13:07.549Z"
}
},
"files_written": [
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/client.h",
"action": "edit",
"tokens": 136,
"at": "2026-07-03T00:02:30.169Z"
"file": "/home/rshen/projects/xce/worktrees/raft_tla/reports/jun29-jul04-2026.md",
"action": "create",
"tokens": 979,
"at": "2026-07-05T03:57:01.756Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 214,
"at": "2026-07-03T00:03:30.228Z"
"tokens": 40,
"at": "2026-07-05T05:20:38.611Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 38,
"at": "2026-07-03T00:03:35.821Z"
"at": "2026-07-05T05:20:58.465Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/tools/test-commit-durability.sh",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 288,
"at": "2026-07-03T00:04:15.847Z"
"tokens": 26,
"at": "2026-07-05T05:21:03.400Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_ctrlstatus.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/raft/raft_sim.c",
"action": "edit",
"tokens": 25,
"at": "2026-07-05T05:21:17.748Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md",
"action": "edit",
"tokens": 25,
"at": "2026-07-05T05:25:03.542Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-apply-decoupling/tasks.md",
"action": "edit",
"tokens": 896,
"at": "2026-07-05T05:25:21.590Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 46,
"at": "2026-07-05T05:41:04.648Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 97,
"at": "2026-07-05T05:41:12.545Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 825,
"at": "2026-07-05T05:41:30.455Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 566,
"at": "2026-07-03T00:08:14.285Z"
"at": "2026-07-05T05:41:44.220Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_ctrlstatus.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c",
"action": "edit",
"tokens": 52,
"at": "2026-07-03T00:08:25.314Z"
"tokens": 415,
"at": "2026-07-05T05:42:04.666Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/waldump.c",
"action": "edit",
"tokens": 376,
"at": "2026-07-05T05:42:14.873Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/soak-verdict.sh",
"action": "edit",
"tokens": 363,
"at": "2026-07-05T05:42:57.027Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 450,
"at": "2026-07-05T05:44:13.870Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 344,
"at": "2026-07-05T05:44:41.636Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 54,
"at": "2026-07-05T05:44:48.489Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 166,
"at": "2026-07-05T05:45:10.893Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 2683,
"at": "2026-07-05T05:46:02.987Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 568,
"at": "2026-07-05T05:46:28.157Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 871,
"at": "2026-07-05T05:48:15.351Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 228,
"at": "2026-07-05T05:48:37.033Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 776,
"at": "2026-07-05T05:53:15.669Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 17,
"at": "2026-07-05T05:53:37.328Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 318,
"at": "2026-07-05T05:53:47.430Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 141,
"at": "2026-07-05T05:54:00.332Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 374,
"at": "2026-07-05T06:16:57.614Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 613,
"at": "2026-07-05T06:22:34.527Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 13,
"at": "2026-07-05T06:23:51.836Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 210,
"at": "2026-07-05T06:24:37.656Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md",
"action": "edit",
"tokens": 92,
"at": "2026-07-05T06:24:53.835Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"action": "edit",
"tokens": 90,
"at": "2026-07-03T02:39:13.696Z"
"tokens": 245,
"at": "2026-07-05T06:33:11.363Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 57,
"at": "2026-07-03T02:39:21.789Z"
"tokens": 603,
"at": "2026-07-05T06:34:12.298Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 100,
"at": "2026-07-03T02:39:30.562Z"
"tokens": 160,
"at": "2026-07-05T06:34:36.760Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
"action": "edit",
"tokens": 106,
"at": "2026-07-03T02:39:36.862Z"
"tokens": 993,
"at": "2026-07-05T07:02:41.181Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
"action": "edit",
"tokens": 560,
"at": "2026-07-03T02:39:49.258Z"
"tokens": 680,
"at": "2026-07-05T07:16:17.567Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
"action": "edit",
"tokens": 214,
"at": "2026-07-03T02:40:22.884Z"
"tokens": 292,
"at": "2026-07-05T07:16:28.893Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 38,
"at": "2026-07-03T02:40:28.333Z"
"tokens": 160,
"at": "2026-07-05T07:16:43.698Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_wal_wire.c",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 9,
"at": "2026-07-03T03:07:07.144Z"
"tokens": 133,
"at": "2026-07-05T07:16:50.545Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-single-leader-role/spec.json",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 135,
"at": "2026-07-03T03:19:38.097Z"
"tokens": 514,
"at": "2026-07-05T07:17:19.666Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-convergence-recovery/spec.json",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 148,
"at": "2026-07-03T03:19:53.139Z"
"tokens": 362,
"at": "2026-07-05T07:19:06.213Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-consistency-invariants/spec.json",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 416,
"at": "2026-07-03T03:20:11.299Z"
"tokens": 403,
"at": "2026-07-05T07:19:17.398Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.gitignore",
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 88,
"at": "2026-07-03T03:23:30.976Z"
"tokens": 313,
"at": "2026-07-05T07:21:05.529Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
"action": "edit",
"tokens": 504,
"at": "2026-07-05T07:21:38.054Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
"action": "edit",
"tokens": 311,
"at": "2026-07-05T07:21:56.305Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/persist.h",
"action": "edit",
"tokens": 486,
"at": "2026-07-05T07:39:27.113Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c",
"action": "edit",
"tokens": 164,
"at": "2026-07-05T07:39:49.040Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/myapp.c",
"action": "edit",
"tokens": 384,
"at": "2026-07-05T07:39:59.983Z"
}
],
"edit_counts": {
"src/include/client.h": 1,
"src/controller/testsuite/test.c": 9,
"src/tools/test-commit-durability.sh": 1,
"src/lib/testsuite/test_ctrlstatus.c": 2,
"src/lib/testsuite/test_wal_wire.c": 1,
".kiro/specs/ha-single-leader-role/spec.json": 1,
".kiro/specs/ha-convergence-recovery/spec.json": 1,
".kiro/specs/ha-consistency-invariants/spec.json": 1,
".gitignore": 1
"reports/jun29-jul04-2026.md": 1,
"src/lib/raft/raft_sim.c": 4,
".kiro/specs/raft-apply-decoupling/tasks.md": 2,
"src/controller/testsuite/test.c": 5,
"src/controller/testsuite/waldump.c": 2,
".claude/skills/xce-test-ha/soak-verdict.sh": 1,
".claude/skills/xce-test-ha/run-ha-soak.sh": 16,
".claude/skills/xce-test-ha/SKILL.md": 4,
"src/include/persist.h": 3,
"src/controller/persist.c": 7,
"src/controller/ha.c": 2,
"src/controller/myapp.c": 1
},
"anatomy_hits": 8,
"anatomy_misses": 9,
"repeated_reads_warned": 21,
"anatomy_hits": 20,
"anatomy_misses": 11,
"repeated_reads_warned": 104,
"cerebrum_warnings": 0,
"stop_count": 6
"stop_count": 8
}

592
.wolf/memory.md
View file

@ -2,6 +2,7 @@
> Chronological action log. Hooks and AI append to this file automatically.
> Old sessions are consolidated by the daemon weekly.
| 18:25 | Task 6.1 round 2 (raft-apply-decoupling): fixed round-1's job-state corruption bug — FAIL/postdone facts no longer bake a receipt-time state snapshot into WAL; wal_jobstate_t._pad now carries WAL_JOB_FAIL_STAGE_FLAG\|stage or WAL_JOB_UPDATE_POSTDONE_MARK, and apply_committed_entry() re-derives the merge at apply time via apply_job_fail_transition/new apply_job_postdone_transition against ACTUAL current state; new persist_submit_job_fact_ex(pad) primitive; fixed 2 pre-existing test_commit_barrier_reply.c tests whose assertions read the now-changed wire format directly (updated to check _pad + call the guard cores directly, not touching the 2 EXPECTED apply-pump-ordering failures reserved for task 7.3); new RED-then-GREEN regression test_apply_pump_fail_racing_finish_merges_against_current_state proves DONE\|PERR not RUN\|EXIT\|PERR; full suite census 904 PASS / 2 expected FAIL, zero new regressions; observed a rare (~2%) unrelated timing flake in the shared (reviewer-verified, off-limits) apply-pump-kick mechanism, not reproducible as a logic defect | wal_types.h, persist.h, cluster.h, job.c, persist.c, test_apply_pump.c, test_commit_barrier_reply.c | GREEN (2 pre-existing expected FAILs unchanged) | ~2200 |
| 08:35 | Tasks 2.1+2.2+2.3 (ha-wal-replication): repl_add_standby idempotency (scan for dup ep before alloc); register standbys in raft_cb_become_leader (election path, via votes[i].peer->hostname/port) and on_standby_connected (late-joiner); deregister in vote_ep_err_handler and standby_ep_err_handler_primary; TDD RED→GREEN; all controller tests PASS | ha.c, repl.c, test_repl_unit.c | GREEN | ~250 |
| 11:28 | Tasks 5.1-5.3 (ha-wal-replication): added test_repl_registration_lifecycle covering all 3 lifecycle steps (register→ep check, duplicate idempotent, deregister→ep==NULL); tasks 5.1+5.2 already covered by existing tests; 18 Tests 0 Failures in repl unit suite | test_repl_unit.c | GREEN | ~80 |
| 10:30 | Task 4.2-fix (ha-wal-replication): fixed 3 reviewer defects — (1) test_committed_lsn_advances_after_majority_ack: changed assertion from committed>0 to committed_after>committed_before||committed_after==written_lsn (fresh persist has written_lsn=0); (2) added test_single_node_committed_lsn_unchanged documenting -ENODEV degraded behavior (Req 4.4 note); (3) fixed test_wal_nack_does_not_update_acked_lsn: quorum math shows self-count alone satisfies n_active=1 quorum, returns 0 not -ETIMEDOUT | test_p2_integration.c, test_repl_unit.c | GREEN: 308 PASS 0 FAIL | ~180 |
@ -3049,3 +3050,594 @@
| 20:19 | Edited .kiro/specs/ha-convergence-recovery/spec.json | 18→19 lines | ~148 |
| 20:20 | Edited .kiro/specs/ha-consistency-invariants/spec.json | modified spec() | ~416 |
| 20:23 | Edited .gitignore | expanded (+6 lines) | ~82 |
| 20:24 | Session end: 18 writes across 7 files (client.h, test.c, test-commit-durability.sh, test_ctrlstatus.c, test_wal_wire.c) | 17 reads | ~66590 tok |
## Session: 2026-07-03 20:27
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 20:32 | Created .kiro/specs/raft-apply-decoupling/spec.json | — | ~130 |
| 20:33 | Created .kiro/specs/raft-apply-decoupling/requirements.md | — | ~4499 |
| 20:33 | Edited .kiro/specs/raft-apply-decoupling/requirements.md | inline fix | ~46 |
| 20:33 | Edited .kiro/specs/raft-apply-decoupling/spec.json | 8→8 lines | ~55 |
| 03:34 | Generated raft-apply-decoupling requirements.md (10 EARS reqs: hook extraction, apply-after-commit ordering, shared apply_committed_entry seam, facts criterion 2.9, D1 quorum-loss, idempotent replay, restart divergence verdict, keystone commit-drive resource bound, conformance) + spec.json init | .kiro/specs/raft-apply-decoupling/{requirements.md,spec.json} | requirements-generated | ~9k |
| 20:34 | Session end: 4 writes across 2 files (spec.json, requirements.md) | 4 reads | ~12611 tok |
| 20:55 | Edited .kiro/specs/raft-apply-decoupling/requirements.md | 1→2 lines | ~67 |
| 20:55 | Edited .kiro/specs/raft-apply-decoupling/requirements.md | expanded (+11 lines) | ~465 |
| 20:55 | Edited .kiro/specs/raft-apply-decoupling/spec.json | inline fix | ~12 |
| 03:55 | Added Requirement 11 (preserve existing behavior/perf, no-regression bar) to raft-apply-decoupling reqs; user-prompted gap catch | .kiro/specs/raft-apply-decoupling/requirements.md | requirements-generated (11 reqs) | ~2k |
| 20:55 | Session end: 7 writes across 2 files (spec.json, requirements.md) | 4 reads | ~13193 tok |
| 21:00 | Session end: 7 writes across 2 files (spec.json, requirements.md) | 4 reads | ~13193 tok |
| 21:07 | Edited .kiro/specs/raft-apply-decoupling/spec.json | 4→4 lines | ~22 |
| 21:08 | Session end: 8 writes across 2 files (spec.json, requirements.md) | 13 reads | ~21955 tok |
| 21:10 | Session end: 8 writes across 2 files (spec.json, requirements.md) | 27 reads | ~215845 tok |
| 21:10 | Session end: 8 writes across 2 files (spec.json, requirements.md) | 27 reads | ~215845 tok |
| 21:17 | Created .kiro/specs/raft-apply-decoupling/research.md | — | ~4907 |
| 21:20 | Created .kiro/specs/raft-apply-decoupling/design.md | — | ~9297 |
| 21:21 | Edited .kiro/specs/raft-apply-decoupling/design.md | expanded (+10 lines) | ~312 |
| 21:21 | Edited .kiro/specs/raft-apply-decoupling/spec.json | 12→12 lines | ~74 |
| 04:22 | Generated raft-apply-decoupling design.md + research.md: verified brief drift via 4 parallel Explore agents (bug-1306 partial pre-fix, ha-term-safety done, raft-drive-async vtask_enq_classified reusable, replay/online apply divergence in restore_job_state vs mark_job_*), designed apply_committed_entry seam + commit-resolve keystone | .kiro/specs/raft-apply-decoupling/{design.md,research.md,spec.json} | design-generated | ~55k |
| 21:22 | Session end: 12 writes across 4 files (spec.json, requirements.md, research.md, design.md) | 27 reads | ~231471 tok |
| 21:27 | Edited .kiro/specs/raft-apply-decoupling/spec.json | 4→4 lines | ~20 |
| 21:28 | Created .kiro/specs/raft-apply-decoupling/tasks.md | — | ~3340 |
| 21:29 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→3 lines | ~43 |
| 21:29 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→3 lines | ~43 |
| 21:29 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→3 lines | ~58 |
| 21:29 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→3 lines | ~41 |
| 21:30 | Session end: 18 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 33 reads | ~246852 tok |
| 21:41 | Created .kiro/specs/raft-apply-decoupling/tasks.md | — | ~3726 |
| 21:42 | Session end: 19 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 33 reads | ~260108 tok |
| 21:52 | Session end: 19 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 33 reads | ~260108 tok |
| 21:53 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | removed 7 lines | ~25 |
| 21:54 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 4→4 lines | ~36 |
| 21:54 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→3 lines | ~71 |
| 21:55 | Edited .kiro/specs/raft-apply-decoupling/spec.json | 19→19 lines | ~105 |
| 04:55 | Generated + repaired raft-apply-decoupling tasks.md via 2-pass independent sanity review (fixed apply-before-pump contradiction, pump-dependent tests sequenced before pump existed, P markers on container tasks, task1.3/3.2 duplicate-scope bug, missing Req 4.3 coverage); 7 major/23 sub-tasks, all 11 requirement groups covered | .kiro/specs/raft-apply-decoupling/{tasks.md,spec.json} | tasks-generated, ready_for_implementation | ~40k |
| 21:56 | Session end: 23 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 33 reads | ~260282 tok |
| 00:08 | Session end: 23 writes across 5 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 36 reads | ~264105 tok |
| 00:14 | Edited src/controller/include/commit.h | modified enum() | ~455 |
| 00:15 | Edited src/controller/include/commit.h | expanded (+12 lines) | ~255 |
| 00:15 | Edited src/controller/include/commit.h | modified fn() | ~272 |
| 00:16 | Created src/controller/testsuite/test_commit_resolve.c | — | ~3259 |
| 00:16 | Edited src/controller/meson.build | 1→2 lines | ~22 |
| 00:16 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~23 |
| 00:16 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~19 |
| 00:19 | Edited src/controller/commit.c | 4→5 lines | ~27 |
| 00:19 | Edited src/controller/commit.c | added 1 condition(s) | ~312 |
| 00:19 | Edited src/controller/commit.c | modified if() | ~71 |
| 00:19 | Edited src/controller/commit.c | added 4 condition(s) | ~782 |
| 00:23 | Edited src/controller/include/commit.h | 3→3 lines | ~30 |
| 00:27 | Session end: 35 writes across 10 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 44 reads | ~285981 tok |
| 00:50 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/append_bugs.py | — | ~1412 |
| 00:54 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~14 |
| 00:55 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 3→8 lines | ~329 |
| 00:59 | Session end: 38 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 45 reads | ~291752 tok |
| 01:14 | Edited src/controller/include/commit.h | modified fn() | ~462 |
| 01:15 | Edited src/controller/commit.c | added 8 condition(s) | ~1928 |
| 01:15 | Edited src/controller/commit.c | 10→15 lines | ~229 |
| 01:16 | Edited src/controller/commit.c | expanded (+6 lines) | ~136 |
| 01:17 | Edited src/controller/commit.c | modified commit_set_writable_lsn() | ~242 |
| 01:17 | Edited src/controller/commit.c | modified commit_on_membership_change() | ~212 |
| 01:19 | Edited src/controller/commit.c | modified if() | ~142 |
| 01:23 | Edited src/controller/testsuite/test_commit_resolve.c | modified half() | ~919 |
| 01:24 | Edited src/controller/testsuite/test_commit_resolve.c | added 1 condition(s) | ~674 |
| 01:26 | Edited src/controller/testsuite/test_commit_resolve.c | modified commit_resolve() | ~2788 |
| 01:33 | Session end: 48 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 46 reads | ~303205 tok |
| 01:55 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | expanded (+16 lines) | ~789 |
| 01:57 | Edited .kiro/specs/raft-apply-decoupling/design.md | inline fix | ~176 |
| 01:57 | Edited .kiro/specs/raft-apply-decoupling/design.md | 1→2 lines | ~310 |
| 02:00 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 3→3 lines | ~89 |
| 02:01 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~531 |
| 02:03 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 6→7 lines | ~208 |
| 02:05 | Session end: 54 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 48 reads | ~327872 tok |
| 02:15 | Edited src/controller/include/commit.h | 5→6 lines | ~80 |
| 02:15 | Edited src/controller/include/commit.h | modified struct() | ~288 |
| 02:16 | Edited src/controller/include/commit.h | expanded (+11 lines) | ~222 |
| 02:16 | Edited src/controller/include/commit.h | modified fn() | ~358 |
| 02:16 | Edited src/controller/testsuite/test_commit_resolve.c | modified Requirements() | ~257 |
| 02:17 | Edited src/controller/testsuite/test_commit_resolve.c | added 1 condition(s) | ~1579 |
| 02:18 | Edited src/controller/commit.c | expanded (+23 lines) | ~587 |
| 02:19 | Edited src/controller/commit.c | added 2 condition(s) | ~272 |
| 02:19 | Edited src/controller/commit.c | expanded (+11 lines) | ~235 |
| 02:19 | Edited src/controller/commit.c | added 6 condition(s) | ~1092 |
| 02:20 | Edited src/controller/commit.c | modified is() | ~175 |
| 02:20 | Edited src/controller/commit.c | added 1 condition(s) | ~348 |
| 02:21 | Edited src/controller/commit.c | added 1 condition(s) | ~315 |
| 02:21 | Edited src/controller/commit.c | 3→6 lines | ~92 |
| 02:35 | Session end: 68 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 50 reads | ~336331 tok |
| 03:00 | Session end: 68 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 51 reads | ~340779 tok |
| 03:04 | Edited src/controller/commit.c | added 1 condition(s) | ~1497 |
| 03:06 | Edited src/controller/commit.c | modified is() | ~87 |
| 03:07 | Edited src/controller/commit.c | modified never() | ~221 |
| 03:09 | Edited src/controller/testsuite/test_commit_resolve.c | added 1 condition(s) | ~252 |
| 03:10 | Edited src/controller/testsuite/test_commit_resolve.c | modified runs() | ~1546 |
| 03:11 | Edited src/controller/commit.c | modified warning() | ~532 |
| 03:12 | Edited src/controller/commit.c | added 2 condition(s) | ~401 |
| 03:19 | Session end: 75 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 51 reads | ~346515 tok |
| 03:23 | Session end: 75 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 52 reads | ~346681 tok |
| 03:26 | Edited src/controller/commit.c | 3→5 lines | ~56 |
| 03:33 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~16 |
| 03:35 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~744 |
| 03:36 | Session end: 78 writes across 11 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 52 reads | ~347934 tok |
| 03:47 | Edited src/controller/include/commit.h | modified demotion() | ~505 |
| 03:48 | Edited src/controller/commit.c | modified is() | ~183 |
| 03:48 | Edited src/controller/commit.c | expanded (+8 lines) | ~232 |
| 03:48 | Edited src/controller/commit.c | modified if() | ~66 |
| 03:48 | Edited src/controller/commit.c | modified if() | ~44 |
| 03:48 | Edited src/controller/commit.c | added 3 condition(s) | ~726 |
| 03:49 | Edited src/include/persist.h | modified pointer() | ~301 |
| 03:49 | Edited src/include/persist.h | modified MUST() | ~196 |
| 03:49 | Edited src/controller/persist.c | modified unregistered() | ~88 |
| 03:49 | Edited src/controller/persist.c | added 1 condition(s) | ~112 |
| 03:49 | Edited src/controller/persist.c | added 1 condition(s) | ~320 |
| 03:49 | Edited src/controller/ha.c | modified 4() | ~339 |
| 03:50 | Edited src/controller/testsuite/test_demotion_convergence.c | modified pattern() | ~232 |
| 03:50 | Edited src/controller/testsuite/test_demotion_convergence.c | modified 4() | ~860 |
| 03:50 | Edited src/controller/testsuite/test_demotion_convergence.c | 2→3 lines | ~56 |
| 03:50 | Edited src/controller/testsuite/test_demotion_convergence.c | 2→3 lines | ~56 |
| 03:51 | Edited src/controller/testsuite/test_persist_truncate.c | 7→9 lines | ~102 |
| 03:51 | Edited src/controller/testsuite/test_persist_truncate.c | modified struct() | ~1742 |
| 03:51 | Edited src/controller/testsuite/test_persist_truncate.c | 3→5 lines | ~71 |
| 04:22 | Edited src/controller/testsuite/test_demotion_convergence.c | modified TEMP_RED_ISOLATION() | ~337 |
| 04:23 | Edited src/controller/testsuite/test_demotion_convergence.c | reduced (-8 lines) | ~224 |
| 11:27 | Task 1.4 complete: wired demotion (ha.c raft_cb_become_follower) + truncation (persist.c persist_truncate_wal_above via new on_truncate callback) to commit_resolve_pending_above; RED/GREEN verified via stash; all targeted suites green (55 tests) | commit.c,commit.h,persist.c,persist.h,ha.c,test_demotion_convergence.c,test_persist_truncate.c | GREEN | ~9000 |
| 04:31 | Session end: 99 writes across 16 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 56 reads | ~367082 tok |
| 04:51 | Edited src/controller/testsuite/test_demotion_convergence.c | 5→9 lines | ~152 |
| 04:51 | Edited src/controller/testsuite/test_demotion_convergence.c | 5→6 lines | ~51 |
| 04:52 | Edited src/controller/testsuite/test_demotion_convergence.c | 7→3 lines | ~56 |
| 04:52 | Edited src/controller/testsuite/test_demotion_convergence.c | 3→2 lines | ~19 |
| 04:57 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~26 |
| 04:57 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~900 |
| 04:59 | Session end: 105 writes across 16 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 61 reads | ~381822 tok |
| 05:08 | Edited src/controller/testsuite/test_commit_resolve.c | modified Requirements() | ~271 |
| 05:09 | Edited src/controller/testsuite/test_commit_resolve.c | added 7 condition(s) | ~4262 |
| 05:09 | Edited src/controller/testsuite/test_demotion_convergence.c | modified struct() | ~1006 |
| 05:09 | Edited src/controller/testsuite/test_demotion_convergence.c | 3→4 lines | ~77 |
| 05:09 | Edited src/controller/testsuite/test_persist_truncate.c | 7→8 lines | ~45 |
| 05:10 | Edited src/controller/testsuite/test_persist_truncate.c | added 1 condition(s) | ~1256 |
| 05:10 | Edited src/controller/testsuite/test_persist_truncate.c | 3→4 lines | ~66 |
| 05:24 | Task 1.5 done: 6 new concurrency tests (3 in test_commit_resolve.c: admission-cap burst, mixed majority/timeout burst w/ bounded-worker assertion, genuine 2-thread race; 1 in test_demotion_convergence.c: concurrent demotion burst; 1 in test_persist_truncate.c: concurrent truncation burst) + buglog bug-1407 reopened (10/10 fail, confirmed pre-existing on clean task-1.4 checkout too) | test_commit_resolve.c, test_demotion_convergence.c, test_persist_truncate.c, buglog.json | GREEN (all 3 suites stable across 15/13/13 repeated runs); bug-1407 flagged, not fixed (out of task scope) | ~95000 |
| 05:31 | Session end: 112 writes across 16 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 61 reads | ~393931 tok |
| 05:43 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/run_regression.sh | — | ~181 |
| 05:52 | Edited src/controller/commit.c | 4→4 lines | ~55 |
| 05:53 | Edited src/controller/testsuite/test_commit_resolve.c | modified test_commit_resolve() | ~134 |
| 06:04 | Edited src/controller/testsuite/test_commit_resolve.c | modified test_commit_resolve() | ~356 |
| 06:06 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~18 |
| 06:07 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~25 |
| 06:07 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~602 |
| 06:08 | Session end: 119 writes across 17 files (spec.json, requirements.md, research.md, design.md, tasks.md) | 68 reads | ~399408 tok |
## Session: 2026-07-03 06:33
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 08:06 | Edited src/controller/include/cluster.h | modified enum() | ~471 |
| 08:06 | Edited src/controller/job.c | added 1 condition(s) | ~792 |
| 08:06 | Edited src/controller/job.c | added 1 condition(s) | ~738 |
| 08:07 | Edited src/controller/job.c | added 1 condition(s) | ~1188 |
| 08:09 | Created src/controller/testsuite/test_apply_seam.c | — | ~2936 |
| 08:12 | Edited src/controller/meson.build | 2→3 lines | ~27 |
| 08:12 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~23 |
| 08:13 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~25 |
| 08:18 | Session end: 8 writes across 5 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 17 reads | ~109387 tok |
| 08:26 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→2 lines | ~49 |
| 08:26 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~359 |
| 08:27 | Session end: 10 writes across 6 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 18 reads | ~112759 tok |
| 08:42 | Edited src/include/persist.h | modified that() | ~683 |
| 08:43 | Created src/controller/testsuite/test_apply_committed_entry.c | — | ~4003 |
| 08:44 | Edited src/controller/meson.build | 2→3 lines | ~26 |
| 08:44 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~22 |
| 08:44 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~24 |
| 08:45 | Edited src/controller/persist.c | added 15 condition(s) | ~1900 |
| 08:47 | Edited src/controller/testsuite/test_apply_committed_entry.c | 3→7 lines | ~98 |
| 08:47 | Edited src/controller/testsuite/test_apply_committed_entry.c | 1→3 lines | ~39 |
| 08:48 | Edited src/controller/testsuite/test_apply_committed_entry.c | expanded (+9 lines) | ~151 |
| 08:49 | Edited src/controller/testsuite/test_apply_committed_entry.c | 5→1 lines | ~19 |
| 08:49 | Edited src/controller/testsuite/test_apply_committed_entry.c | 3→1 lines | ~9 |
| 08:49 | Edited src/controller/testsuite/test_apply_committed_entry.c | 7→9 lines | ~130 |
| 08:49 | Edited src/controller/testsuite/test_apply_committed_entry.c | added 1 condition(s) | ~83 |
| 08:53 | Session end: 23 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 27 reads | ~141413 tok |
| 09:09 | Session end: 23 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 30 reads | ~149541 tok |
| 09:15 | Edited src/include/persist.h | expanded (+11 lines) | ~432 |
| 09:18 | Session end: 24 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 35 reads | ~184232 tok |
| 09:29 | Session end: 24 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 36 reads | ~184232 tok |
| 09:37 | Session end: 24 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 37 reads | ~184232 tok |
| 09:38 | Edited src/include/persist.h | 4→6 lines | ~113 |
| 09:45 | Session end: 25 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 37 reads | ~184388 tok |
| 09:51 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~12 |
| 09:51 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~890 |
| 09:52 | Session end: 27 writes across 9 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 37 reads | ~186180 tok |
| 10:01 | Created src/controller/testsuite/test_replay_apply_seam.c | — | ~3686 |
| 10:01 | Edited src/controller/testsuite/test_replay_apply_seam.c | 6→7 lines | ~38 |
| 10:01 | Edited src/controller/meson.build | 2→3 lines | ~31 |
| 10:02 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~33 |
| 10:02 | Edited src/controller/testsuite/controller_test.c | 4→5 lines | ~33 |
| 10:03 | Edited src/controller/persist.c | expanded (+8 lines) | ~234 |
| 10:03 | Edited src/controller/persist.c | added error handling | ~295 |
| 10:08 | Edited src/controller/testsuite/test_replay_apply_seam.c | 3→6 lines | ~79 |
| 10:08 | Edited src/controller/testsuite/test_replay_apply_seam.c | 3→4 lines | ~49 |
| 17:22 | Session end: 36 writes across 10 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 40 reads | ~194864 tok |
| 17:31 | Edited src/controller/persist.c | modified ever() | ~484 |
| 17:32 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~23 |
| 17:32 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~705 |
| 17:33 | Session end: 39 writes across 10 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 42 reads | ~297489 tok |
| 17:36 | Created src/controller/testsuite/test_apply_reapply_idempotency.c | — | ~4001 |
| 17:37 | Edited src/controller/meson.build | 3→4 lines | ~44 |
| 17:37 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~46 |
| 17:37 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~38 |
| 17:39 | Session end: 43 writes across 11 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 42 reads | ~302593 tok |
| 17:44 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~13 |
| 17:44 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~22 |
| 17:44 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | modified built() | ~439 |
| 17:45 | Session end: 46 writes across 11 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 44 reads | ~307329 tok |
| 20:02 | Session end: 46 writes across 11 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 44 reads | ~307329 tok |
| 20:17 | Edited src/include/persist.h | modified submission() | ~369 |
| 20:18 | Edited src/controller/persist.c | added 2 condition(s) | ~1097 |
| 20:18 | Edited src/controller/persist.c | modified if() | ~149 |
| 20:18 | Edited src/controller/job.c | added 2 condition(s) | ~698 |
| 20:19 | Edited src/controller/job.c | added 6 condition(s) | ~1743 |
| 20:22 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified test_worker_join_no_hostconfig_error_reply_immediate() | ~1043 |
| 20:22 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→5 lines | ~94 |
| 20:31 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified PROBE() | ~170 |
| 20:32 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 9→4 lines | ~60 |
| 20:36 | Session end: 55 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 47 reads | ~341728 tok |
| 20:54 | Session end: 55 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 49 reads | ~341728 tok |
| 21:01 | Edited src/controller/job.c | modified did() | ~974 |
| 21:01 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 4 condition(s) | ~1707 |
| 21:02 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 3→4 lines | ~77 |
| 21:02 | Edited src/controller/job.c | modified if() | ~527 |
| 21:06 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 19→24 lines | ~308 |
| 21:07 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 39→43 lines | ~627 |
| 21:08 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 4→8 lines | ~146 |
| 21:09 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 43→39 lines | ~620 |
| 21:15 | Fixed bug-1408: handle_jobsubmit array-partial-construct-failure notification stranding (round-2 remediation of raft-apply-decoupling task 3.1) | src/controller/job.c, src/controller/testsuite/test_commit_barrier_reply.c | RED confirmed against round-1 code then GREEN after fix; full controller_unit_test suite 0 failures | ~9800 |
| 21:14 | Session end: 63 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 49 reads | ~348892 tok |
| 21:23 | Edited src/controller/job.c | added 1 condition(s) | ~88 |
| 21:43 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 2 condition(s) | ~1209 |
| 21:44 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~22 |
| 21:44 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~829 |
| 21:45 | Session end: 67 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 53 reads | ~351540 tok |
| 22:05 | Edited src/include/persist.h | expanded (+11 lines) | ~258 |
| 22:05 | Edited src/include/persist.h | modified void() | ~1466 |
| 22:05 | Edited src/controller/persist.c | modified empty() | ~148 |
| 22:06 | Edited src/controller/persist.c | modified struct() | ~320 |
| 22:06 | Edited src/controller/persist.c | modified while() | ~260 |
| 22:07 | Edited src/controller/persist.c | added 13 condition(s) | ~1906 |
| 22:07 | Edited src/controller/job.c | added 5 condition(s) | ~1796 |
| 22:08 | Edited src/controller/job.c | 9→6 lines | ~59 |
| 22:08 | Edited src/controller/job.c | modified already() | ~929 |
| 22:12 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified 1() | ~288 |
| 22:12 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~1761 |
| 22:13 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified REMOVED() | ~436 |
| 22:13 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 3→3 lines | ~56 |
| 22:19 | Session end: 80 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 60 reads | ~399164 tok |
| 22:34 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~33 |
| 22:35 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~1147 |
| 22:36 | Session end: 82 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 62 reads | ~429805 tok |
| 22:48 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~4110 |
| 22:48 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified 3() | ~170 |
| 22:52 | Task 3.3 raft-apply-decoupling: added 4 tests to test_commit_barrier_reply.c (repeated timeout, repeated+single truncation-forced FAILED, array mixed-outcome) covering handle_jobsubmit failure path | src/controller/testsuite/test_commit_barrier_reply.c | 29/29 PASS x3 runs, no regressions | ~700 |
| 22:53 | Session end: 84 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 64 reads | ~453265 tok |
| 23:04 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~12 |
| 23:04 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~19 |
| 23:04 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~635 |
| 23:06 | Session end: 87 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 64 reads | ~458179 tok |
| 23:24 | Edited src/include/persist.h | modified performs() | ~381 |
| 23:24 | Edited src/controller/persist.c | modified 1() | ~368 |
| 23:24 | Edited src/controller/job.c | 4→6 lines | ~104 |
| 23:24 | Edited src/controller/job.c | modified 1() | ~742 |
| 23:25 | Edited src/controller/job.c | modified 1() | ~557 |
| 23:25 | Edited src/controller/job.c | modified mark_job_fail() | ~699 |
| 23:25 | Edited src/controller/job.c | mark_job_fail() → record_job_fail_fact() | ~48 |
| 23:25 | Edited src/controller/job.c | added 1 condition(s) | ~773 |
| 23:26 | Edited src/controller/job.c | modified 1() | ~331 |
| 23:26 | Edited src/controller/job.c | 7→9 lines | ~184 |
| 23:26 | Edited src/controller/job.c | 6→8 lines | ~168 |
| 23:30 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~620 |
| 23:31 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified pin() | ~806 |
| 23:31 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified test_guard_fail_after_finish_preserves_perr_merge() | ~852 |
| 23:33 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 2 condition(s) | ~1744 |
| 23:33 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 4→6 lines | ~78 |
| 23:33 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 4→3 lines | ~59 |
| 23:34 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified arm() | ~939 |
| 23:39 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 3→4 lines | ~67 |
| 23:43 | Session end: 106 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 64 reads | ~473540 tok |
| 23:59 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~23 |
| 23:59 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~1115 |
| 00:00 | Session end: 108 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 65 reads | ~475114 tok |
| 00:07 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified 2() | ~2698 |
| 00:07 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 3→6 lines | ~96 |
| 07:11 | Added test_fact_state_unchanged_throughout_quorum_loss_outage() to test_commit_barrier_reply.c (task 4.2, Req 6.1/6.2): sustained-duration quorum-loss poll (5x20ms) of live job state across all 4 fact types, closing gap between 4.1's single-immediate-check test and pre-existing (ha-commit-barrier-rollout) quorum-loss durability test that never checked live state; 31/31 GREEN first run (guarantee pre-existing, no prod changes) | src/controller/testsuite/test_commit_barrier_reply.c | GREEN | ~9200 |
| 00:12 | Session end: 110 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 65 reads | ~481799 tok |
| 00:21 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~11 |
| 00:21 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~23 |
| 00:22 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~813 |
| 00:23 | Session end: 113 writes across 12 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 65 reads | ~482759 tok |
| 00:33 | Edited src/include/persist.h | added error handling | ~771 |
| 00:34 | Edited src/include/persist.h | expanded (+21 lines) | ~407 |
| 00:34 | Edited src/include/persist.h | modified NULL() | ~533 |
| 00:34 | Edited src/controller/persist.c | modified clear() | ~168 |
| 00:34 | Edited src/controller/persist.c | added 1 condition(s) | ~245 |
| 00:34 | Edited src/controller/persist.c | added 1 condition(s) | ~436 |
| 00:35 | Edited src/controller/persist.c | added 1 condition(s) | ~1380 |
| 00:36 | Edited src/controller/persist.c | added 9 condition(s) | ~772 |
| 00:37 | Created src/controller/testsuite/test_apply_correctness_verification.c | — | ~3600 |
| 00:37 | Edited src/controller/meson.build | 2→3 lines | ~38 |
| 00:37 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~39 |
| 00:37 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~33 |
| 00:54 | Session end: 125 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 66 reads | ~497243 tok |
| 01:04 | Session end: 125 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 68 reads | ~501540 tok |
| 04:11 | Session end: 125 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 68 reads | ~501540 tok |
| 04:17 | Edited src/controller/persist.c | modified 1() | ~704 |
| 04:17 | Edited src/controller/testsuite/test_apply_correctness_verification.c | 5→6 lines | ~28 |
| 04:18 | Edited src/controller/testsuite/test_apply_correctness_verification.c | added 1 condition(s) | ~1623 |
| 04:18 | Edited src/controller/testsuite/test_apply_correctness_verification.c | modified test_apply_correctness_verification() | ~89 |
| 04:19 | Edited src/controller/persist.c | 6→5 lines | ~62 |
| 04:19 | Edited src/controller/persist.c | 5→6 lines | ~67 |
| 11:22 | Task 5.1 round-2 fix: persist_recover apply_exp.expected_last_applied_lsn now reads p->last_applied_lsn (logical cursor) not recovered_tip (physical WAL tip); fixes false APPLY_DIVERGENT on checkpoint-marker-tail crash window | src/controller/persist.c, src/controller/testsuite/test_apply_correctness_verification.c | RED (APPLY_DIVERGENT expected=3)->GREEN (APPLY_CLEAN); full controller_unit_test suite 0 failures | ~1200 |
| 04:23 | Session end: 131 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 68 reads | ~504612 tok |
| 04:28 | Edited src/controller/persist.c | 6→5 lines | ~63 |
| 04:28 | Edited src/controller/persist.c | 5→6 lines | ~67 |
| 04:31 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~31 |
| 04:32 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 1→2 lines | ~984 |
| 04:33 | Session end: 135 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 69 reads | ~509948 tok |
| 04:39 | Edited src/include/persist.h | modified struct() | ~540 |
| 04:39 | Edited src/include/persist.h | 6→8 lines | ~145 |
| 04:39 | Edited src/include/persist.h | 7→8 lines | ~153 |
| 04:41 | Edited src/controller/testsuite/test_apply_correctness_verification.c | added error handling | ~2379 |
| 04:43 | Edited src/controller/persist.c | added 1 condition(s) | ~305 |
| 04:43 | Edited src/controller/persist.c | 3→8 lines | ~116 |
| 04:43 | Edited src/controller/persist.c | added error handling | ~608 |
| 04:46 | Session end: 142 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 70 reads | ~515377 tok |
| 04:53 | Session end: 142 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 70 reads | ~516055 tok |
| 05:02 | Edited src/controller/persist.c | added 1 condition(s) | ~721 |
| 05:02 | Edited src/controller/testsuite/test_apply_correctness_verification.c | modified 2() | ~1599 |
| 05:03 | Edited src/controller/persist.c | reduced (-20 lines) | ~356 |
| 05:08 | Session end: 145 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 70 reads | ~519577 tok |
| 05:17 | Edited src/controller/persist.c | reduced (-26 lines) | ~282 |
| 05:21 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~25 |
| 05:21 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~855 |
| 05:23 | Session end: 148 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 71 reads | ~520835 tok |
| 05:33 | Edited src/controller/testsuite/test_apply_correctness_verification.c | added error handling | ~4582 |
| 05:40 | Task 5.3: added 3 tests to test_apply_correctness_verification.c (explicit stays-alive pin, direct clear-mechanism test, restart-based fence-clear test) | src/controller/testsuite/test_apply_correctness_verification.c | 11/11 GREEN, no production code touched | ~9000 |
| 05:38 | Session end: 149 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 71 reads | ~527121 tok |
| 05:42 | Session end: 149 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 71 reads | ~527912 tok |
| 05:46 | Session end: 149 writes across 13 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 71 reads | ~527912 tok |
| 05:52 | Edited src/controller/testsuite/test_commit_durability.c | modified test_promotion_fence_apply_divergent_survives_structural_catchup() | ~1183 |
| 05:52 | Edited src/controller/testsuite/test_commit_durability.c | modified remediation() | ~99 |
| 05:53 | Edited src/controller/ha.c | modified 3() | ~409 |
| 05:53 | Edited src/controller/ha.c | modified above() | ~269 |
| 05:53 | Edited src/controller/ha.c | modified if() | ~494 |
| 12:55 | Fixed critical safety bug: recovery_unrecoverable fence was trigger-agnostic, both ha.c live-clear sites (promotion_fence_timeout_fn, raft_cb_become_leader gate) could clear an APPLY_DIVERGENT fence based on structural checks alone | src/controller/ha.c, src/controller/testsuite/test_commit_durability.c | RED->GREEN: new regression test_promotion_fence_apply_divergent_survives_structural_catchup confirmed fence wrongly cleared pre-fix, fixed post-fix; guarded both clear sites on persist_apply_verdict()!=APPLY_DIVERGENT; test_commit_durability 39/39, test_apply_correctness_verification 11/11, test_ha_core_safety 28/28 all pass | ~1800 |
| 05:57 | Session end: 154 writes across 15 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 71 reads | ~530540 tok |
| 06:02 | Edited src/controller/ha.c | modified if() | ~88 |
| 06:08 | Edited src/controller/ha.c | modified if() | ~50 |
| 06:11 | Session end: 156 writes across 15 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 71 reads | ~562879 tok |
| 06:15 | Edited src/controller/testsuite/test_convergence_verification.c | added error handling | ~1593 |
| 06:15 | Edited src/controller/testsuite/test_convergence_verification.c | 3→8 lines | ~91 |
| 06:16 | Edited src/controller/ha.c | inline fix | ~31 |
| 06:17 | Edited src/controller/ha.c | inline fix | ~23 |
| 06:20 | Session end: 160 writes across 16 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 72 reads | ~564740 tok |
| 06:25 | Edited src/controller/ha.c | inline fix | ~18 |
| 06:25 | Edited src/controller/ha.c | inline fix | ~23 |
| 06:32 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~21 |
| 06:32 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~13 |
| 06:32 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | modified conditions() | ~1218 |
| 06:34 | Session end: 165 writes across 16 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 72 reads | ~583105 tok |
| 09:34 | Session end: 165 writes across 16 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 72 reads | ~584238 tok |
| 09:52 | Edited src/include/persist.h | modified XCE_TESTING() | ~262 |
| 09:53 | Edited src/include/persist.h | modified unconditionally() | ~1538 |
| 09:53 | Edited src/controller/persist.c | modified idle() | ~102 |
| 09:53 | Edited src/controller/persist.c | expanded (+6 lines) | ~102 |
| 09:53 | Edited src/controller/persist.c | modified makes() | ~780 |
| 09:54 | Edited src/controller/persist.c | removed 19 lines | ~35 |
| 09:54 | Edited src/controller/persist.c | 5→7 lines | ~90 |
| 09:54 | Edited src/controller/persist.c | modified guard() | ~240 |
| 09:54 | Edited src/controller/persist.c | expanded (+9 lines) | ~203 |
| 09:54 | Edited src/controller/persist.c | modified switch() | ~402 |
| 09:55 | Edited src/controller/persist.c | added 1 condition(s) | ~290 |
| 09:55 | Edited src/controller/persist.c | added 1 condition(s) | ~238 |
| 09:55 | Edited src/controller/persist.c | added 1 condition(s) | ~115 |
| 09:57 | Edited src/controller/persist.c | added 5 condition(s) | ~290 |
| 09:58 | Edited src/controller/persist.c | added 24 condition(s) | ~2847 |
| 09:58 | Edited src/controller/commit.c | modified is() | ~328 |
| 10:01 | Edited src/include/persist.h | modified and() | ~68 |
| 10:06 | Created src/controller/testsuite/test_apply_pump.c | — | ~4973 |
| 10:07 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~38 |
| 10:07 | Edited src/controller/testsuite/controller_test.c | 4→5 lines | ~38 |
| 10:07 | Edited src/controller/meson.build | 2→3 lines | ~36 |
| 10:08 | Edited src/controller/testsuite/test_apply_pump.c | 3→4 lines | ~71 |
| 10:10 | Edited src/controller/testsuite/test_apply_pump.c | added error handling | ~200 |
| 10:11 | Edited src/controller/testsuite/test_apply_pump.c | 3→4 lines | ~56 |
| 10:11 | Edited src/controller/testsuite/test_apply_pump.c | modified tap_teardown() | ~28 |
| 10:15 | Edited src/controller/testsuite/test_apply_pump.c | expanded (+6 lines) | ~435 |
| 10:15 | Edited src/controller/testsuite/test_apply_pump.c | 3→5 lines | ~67 |
| 10:17 | Edited src/controller/testsuite/test_apply_pump.c | expanded (+6 lines) | ~248 |
| 10:20 | Created src/controller/testsuite/test_apply_pump.c | — | ~5538 |
| 10:21 | Edited src/controller/testsuite/test_apply_pump.c | 6→7 lines | ~38 |
| 10:21 | Edited src/controller/testsuite/test_apply_pump.c | added 1 condition(s) | ~295 |
| 10:21 | Edited src/controller/testsuite/test_apply_pump.c | 6→10 lines | ~177 |
| 10:22 | Edited src/controller/testsuite/test_apply_pump.c | modified switch() | ~229 |
| 10:29 | Edited src/controller/persist.c | added 1 condition(s) | ~626 |
| 17:37 | Task 6.1 (raft-apply-decoupling) complete: ordered apply pump built (persist_apply_pump_run/kick, take_pending_apply accessor), Hazard 1 fixed via thread-local apply-seam guard (job_ctrl dual-origin precision verified), Hazard 2 investigated+escalated BLOCKED (wire-format needed), trigger wired at commit_resolve_pending (all 6 advance paths, not just ha.c on_commit_advanced), busy-loop bug found+fixed in drain retry, full 78-group regression sweep clean except 2 pre-documented stale tests | src/controller/persist.c, src/include/persist.h, src/controller/commit.c, src/controller/testsuite/test_apply_pump.c | RED confirmed (compile failure on revert) then GREEN, all new+regression tests pass | ~large |
| 10:40 | Session end: 199 writes across 18 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 74 reads | ~619277 tok |
| 10:53 | Session end: 199 writes across 18 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 75 reads | ~619610 tok |
| 11:00 | Edited src/include/wal_types.h | modified struct() | ~657 |
| 11:01 | Edited src/include/persist.h | modified performs() | ~615 |
| 11:01 | Edited src/include/persist.h | modified _pad() | ~494 |
| 11:02 | Edited src/include/persist.h | 6→8 lines | ~149 |
| 11:02 | Edited src/controller/include/cluster.h | modified of() | ~245 |
| 11:03 | Edited src/controller/job.c | modified 1() | ~693 |
| 11:03 | Edited src/controller/job.c | added 1 condition(s) | ~765 |
| 11:04 | Edited src/controller/job.c | modified on_job_postdone() | ~262 |
| 11:04 | Edited src/controller/job.c | modified residual() | ~187 |
| 11:04 | Edited src/controller/persist.c | modified 1() | ~605 |
| 11:05 | Edited src/controller/persist.c | added 7 condition(s) | ~1758 |
| 11:06 | Edited src/controller/job.c | 7→7 lines | ~137 |
| 11:07 | Edited src/controller/testsuite/test_apply_pump.c | modified test_apply_pump_fail_racing_finish_merges_against_current_state() | ~1328 |
| 11:08 | Edited src/controller/testsuite/test_apply_pump.c | modified test_apply_pump() | ~141 |
| 11:09 | Edited src/controller/job.c | modified record_job_fail_fact() | ~202 |
| 11:09 | Edited src/controller/persist.c | reduced (-6 lines) | ~110 |
| 11:11 | Edited src/controller/testsuite/test_apply_pump.c | 9→6 lines | ~92 |
| 11:17 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~715 |
| 11:18 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified 2() | ~1130 |
| 11:19 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified test_guard_fail_after_finish_preserves_perr_merge() | ~1251 |
| 11:19 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→2 lines | ~35 |
| 11:19 | Edited src/controller/testsuite/test_commit_barrier_reply.c | cbrp_count_jobstate_entries() → cbrp_count_jobstate_entries_ex() | ~78 |
| 11:20 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified 2() | ~241 |
| 11:20 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→2 lines | ~41 |
| 11:26 | Created ../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/a13f7866-1ea4-4b63-91a6-4de1393543ae/scratchpad/append_bug_1414.py | — | ~1114 |
| 11:29 | Session end: 224 writes across 20 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 75 reads | ~638494 tok |
| 11:42 | Edited src/controller/testsuite/test_apply_pump.c | added 1 condition(s) | ~639 |
| 11:43 | Edited src/controller/testsuite/test_apply_pump.c | 5→10 lines | ~132 |
| 11:43 | Edited src/controller/testsuite/test_apply_pump.c | expanded (+7 lines) | ~175 |
| 11:43 | Edited src/controller/testsuite/test_apply_pump.c | modified IMPORTANT() | ~378 |
| 11:45 | Edited src/controller/testsuite/test_apply_pump.c | expanded (+26 lines) | ~708 |
| 11:47 | Edited src/controller/testsuite/test_apply_pump.c | 39→41 lines | ~747 |
| 11:48 | Edited src/controller/job.c | modified record_job_fail_fact() | ~150 |
| 11:49 | Edited src/controller/job.c | 3→2 lines | ~36 |
| 11:58 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | 2→2 lines | ~32 |
| 11:59 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~1905 |
| 12:01 | Session end: 234 writes across 20 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 77 reads | ~646803 tok |
| 12:08 | Edited src/controller/persist.c | added 1 condition(s) | ~332 |
| 12:08 | Edited src/controller/job.c | modified struct() | ~256 |
| 12:08 | Edited src/controller/job.c | added 2 condition(s) | ~727 |
| 12:09 | Edited src/controller/job.c | 6→7 lines | ~54 |
| 12:10 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified struct() | ~209 |
| 12:10 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~126 |
| 12:11 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 3 condition(s) | ~2930 |
| 12:11 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→6 lines | ~118 |
| 19:20 | Task 6.2 done: persist_apply_pump_run() consumes persist_take_pending_apply() per-LSN, jobsubmit_apply_complete() now sends the success reply (RequestReply_data_jobid) | persist.c, job.c, test_commit_barrier_reply.c | RED->GREEN via git-stash isolation; 2 new tests pass 5/5 stable; discovered+logged bug-1415 (vtaskmgr pool race, not fixed, out of boundary) | ~95000 |
| 16:53 | Session end: 242 writes across 20 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 77 reads | ~655960 tok |
| 17:02 | Session end: 242 writes across 20 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 78 reads | ~659590 tok |
| 17:08 | Edited src/include/persist.h | modified ever() | ~605 |
| 17:08 | Edited src/controller/persist.c | added 1 condition(s) | ~345 |
| 17:11 | Edited src/controller/testsuite/test_apply_pump.c | expanded (+11 lines) | ~353 |
| 17:12 | Edited src/controller/testsuite/test_apply_pump.c | added 12 condition(s) | ~4160 |
| 17:13 | Edited src/controller/persist.c | 6→6 lines | ~46 |
| 17:13 | Edited src/controller/persist.c | 6→6 lines | ~36 |
| 17:20 | Task 6.2 round 2 (raft-apply-decoupling): fixed bug-1415 (real, code-confirmed race — commit continuation dispatch vs. apply-pump kick are independent vtasks with no completion-order guarantee under a real n_vcpus>1 pool; a late persist_register_pending_apply() call for an already-applied lsn orphaned forever, permanent client hang + leaked jobsubmit_async_ctx_t). Fix: self-healing early-return in persist_register_pending_apply() (persist.c) — fires on_applied immediately if lsn <= persist_last_applied_lsn(p) instead of linking a dead registration. New deterministic regression test_apply_pump_late_pending_apply_registration_self_heals (test_apply_pump.c) wires a REAL multi-vcpu vtaskmgr_t via commit_set_pool (the only test in this spec to do so) and forces the exact bad interleaving via a poll-barrier in its stand-in commit continuation, rather than a timing guess. Verified genuine RED (fails "Expected 1 Was 0" against unfixed code) → GREEN. Full regression sweep (test_apply_pump, test_commit_barrier_reply, test_apply_reapply_idempotency, test_commit_resolve, test_commit_admission_pool, test_demotion_convergence): zero new failures — only the 2 pre-existing/expected task-7.3-owned test_commit_barrier_reply.c failures remain, confirmed identical before/after via baseline diff. Did not touch job.c, persist_take_pending_apply(), or persist_apply_pump_run()'s walk (round 1's verified-correct work, per boundary). | src/controller/persist.c, src/include/persist.h, src/controller/testsuite/test_apply_pump.c, .wolf/buglog.json | GREEN (bug-1415 fixed, RED→GREEN proven, zero regressions) | ~1800 |
| 17:21 | Session end: 248 writes across 20 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 79 reads | ~666106 tok |
| 17:28 | Edited src/controller/persist.c | 4→4 lines | ~28 |
| 17:37 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~23 |
| 17:37 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added 1 condition(s) | ~1390 |
| 17:39 | Session end: 251 writes across 20 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 79 reads | ~668082 tok |
| 17:52 | Created src/controller/testsuite/test_apply_pump_integration.c | — | ~8881 |
| 17:53 | Edited src/controller/meson.build | 2→3 lines | ~26 |
| 17:53 | Edited src/controller/testsuite/controller_test.c | modified setUp() | ~30 |
| 17:53 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~24 |
| 17:53 | Edited src/controller/testsuite/test_apply_pump_integration.c | TEST_ASSERT_EQUAL_INT_MESSAGE() → TEST_ASSERT_EQUAL_INT() | ~138 |
| 17:54 | Edited src/controller/job.c | added error handling | ~244 |
| 17:55 | Edited src/controller/job.c | reduced (-8 lines) | ~38 |
| 17:56 | raft-apply-decoupling task 6.3 complete: new test_apply_pump_integration.c (2 tests: command+fact interleave through same commit, multi-fact quorum-loss-then-restore reactions) | src/controller/testsuite/test_apply_pump_integration.c, meson.build, controller_test.c | GREEN 30/30 stable runs; RED-phase verified via temporary job.c injection (reverted, diff clean); 2 pre-existing test_commit_barrier_reply.c failures confirmed pre-existing/expected (task 7.3's job, not a regression) | ~45000 |
| 18:00 | Session end: 258 writes across 21 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 79 reads | ~688501 tok |
| 18:09 | Edited src/controller/job.c | modified INJECTION() | ~288 |
| 18:09 | Edited src/controller/job.c | reduced (-8 lines) | ~161 |
| 18:12 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~23 |
| 18:12 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~856 |
| 18:14 | Session end: 262 writes across 21 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 80 reads | ~698825 tok |
| 18:25 | Edited src/controller/testsuite/test_apply_pump_integration.c | 3→4 lines | ~23 |
| 18:25 | Edited src/controller/testsuite/test_apply_pump_integration.c | expanded (+10 lines) | ~195 |
| 18:25 | Edited src/controller/testsuite/test_apply_pump_integration.c | added 3 condition(s) | ~727 |
| 18:25 | Edited src/controller/testsuite/test_apply_pump_integration.c | modified tpi_setup() | ~128 |
| 18:26 | Edited src/controller/testsuite/test_apply_pump_integration.c | added 2 condition(s) | ~315 |
| 18:26 | Edited src/controller/testsuite/test_apply_pump_integration.c | inline fix | ~5 |
| 18:26 | Edited src/controller/testsuite/test_apply_pump_integration.c | added 1 condition(s) | ~265 |
| 18:27 | Edited src/controller/testsuite/test_apply_pump_integration.c | added error handling | ~6697 |
| 18:28 | Edited src/controller/testsuite/test_apply_pump_integration.c | expanded (+13 lines) | ~337 |
| 18:28 | Edited src/controller/testsuite/test_apply_pump_integration.c | modified parameter() | ~357 |
| 18:29 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~75 |
| 18:29 | Edited src/controller/testsuite/test_apply_pump_integration.c | inline fix | ~16 |
| 18:29 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~74 |
| 18:30 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~59 |
| 18:32 | Session end: 276 writes across 21 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 80 reads | ~717274 tok |
| 18:40 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~70 |
| 18:41 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~70 |
| 18:41 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~59 |
| 18:41 | Edited src/controller/testsuite/test_apply_pump_integration.c | 4→4 lines | ~59 |
| 18:45 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~11 |
| 18:45 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~25 |
| 18:46 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | added error handling | ~954 |
| 18:47 | Session end: 283 writes across 21 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 81 reads | ~718896 tok |
| 18:57 | Edited src/lib/raft/raft.h | modified RAFT_SIM_BREAK_APPLY_BEFORE_COMMIT() | ~386 |
| 18:57 | Edited src/lib/raft/raft.h | expanded (+9 lines) | ~216 |
| 18:57 | Edited src/lib/raft/raft.h | modified loop() | ~605 |
| 18:57 | Edited src/lib/raft/raft.h | modified raft_sim_replay_commit() | ~251 |
| 18:57 | Edited src/lib/raft/raft_sim.c | 4→9 lines | ~118 |
| 18:58 | Edited src/lib/raft/raft_sim.c | added 1 condition(s) | ~262 |
| 18:58 | Edited src/lib/raft/raft_sim.c | modified sim_conf_on_commit_advanced() | ~438 |
| 18:58 | Edited src/lib/raft/raft_sim.c | added 2 condition(s) | ~191 |
| 18:58 | Edited src/lib/raft/raft_sim.c | added 4 condition(s) | ~435 |
| 18:59 | Edited src/lib/testsuite/test_raft.c | modified CORRECT() | ~2350 |
| 18:59 | Edited src/lib/testsuite/test_raft.c | 3→5 lines | ~73 |
| 18:59 | Session end: 294 writes across 24 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 85 reads | ~823107 tok |
| 19:05 | Edited src/lib/testsuite/test_raft.c | 4→4 lines | ~68 |
| 02:07 | raft-apply-decoupling task 7.1: added apply-after-commit + idempotent-reapply assertions to raft_sim.c conformance harness | src/lib/raft/raft.h, src/lib/raft/raft_sim.c, src/lib/testsuite/test_raft.c | GREEN 287/287, RED confirmed via temp fault-disable | ~ok |
| 19:08 | Session end: 295 writes across 24 files (cluster.h, job.c, test_apply_seam.c, meson.build, controller_test.c) | 85 reads | ~823180 tok |
## Session: 2026-07-05 19:36
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-07-05 20:50
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 20:57 | Created reports/jun29-jul04-2026.md | — | ~914 |
| 20:57 | Session end: 1 writes across 1 files (jun29-jul04-2026.md) | 0 reads | ~979 tok |
| 22:13 | Session end: 1 writes across 1 files (jun29-jul04-2026.md) | 0 reads | ~979 tok |
| 22:20 | Edited src/lib/raft/raft_sim.c | 2→2 lines | ~37 |
| 22:20 | Edited src/lib/raft/raft_sim.c | 2→2 lines | ~36 |
| 22:21 | Edited src/lib/raft/raft_sim.c | inline fix | ~24 |
| 22:21 | Edited src/lib/raft/raft_sim.c | inline fix | ~23 |
| 22:25 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | inline fix | ~23 |
| 22:25 | Edited .kiro/specs/raft-apply-decoupling/tasks.md | modified XCE_TESTING() | ~836 |
| 22:27 | Session end: 7 writes across 3 files (jun29-jul04-2026.md, raft_sim.c, tasks.md) | 6 reads | ~81884 tok |
| 22:41 | Edited src/controller/testsuite/test.c | 2→4 lines | ~43 |
| 22:41 | Edited src/controller/testsuite/test.c | 3→5 lines | ~91 |
| 22:41 | Edited src/controller/testsuite/test.c | added 2 condition(s) | ~770 |
| 22:41 | Edited src/controller/testsuite/test.c | added 2 condition(s) | ~528 |
| 22:42 | Edited src/controller/testsuite/waldump.c | added 1 condition(s) | ~387 |
| 22:42 | Edited src/controller/testsuite/waldump.c | added 3 condition(s) | ~351 |
| 22:42 | Edited .claude/skills/xce-test-ha/soak-verdict.sh | modified verdict_emit_factstream() | ~339 |
| 22:44 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified 2() | ~420 |
| 22:44 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+8 lines) | ~322 |
| 22:44 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 3→4 lines | ~51 |
| 22:45 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | 4→8 lines | ~155 |
| 22:46 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified verify_factstream_no_phantom() | ~2504 |
| 22:46 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified PASS() | ~530 |
| 22:48 | Edited .claude/skills/xce-test-ha/SKILL.md | modified line() | ~813 |
| 22:48 | Edited .claude/skills/xce-test-ha/SKILL.md | 6→9 lines | ~213 |
| 22:53 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+21 lines) | ~724 |
| 22:53 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | "${XCE_HA_FACTSTREAM_KILL_" → "${XCE_HA_FACTSTREAM_KILL_" | ~16 |
| 22:53 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+7 lines) | ~297 |
| 22:54 | Edited .claude/skills/xce-test-ha/SKILL.md | 3→5 lines | ~131 |
| 23:16 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+15 lines) | ~349 |
| 23:22 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified synchronously() | ~572 |
| 23:23 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | inline fix | ~12 |
| 23:24 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+6 lines) | ~196 |
| 23:24 | Edited .claude/skills/xce-test-ha/SKILL.md | 2→3 lines | ~86 |
| 23:33 | Edited src/controller/testsuite/test.c | expanded (+10 lines) | ~228 |
| 23:34 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified values() | ~563 |
| 23:34 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | state() → raw() | ~149 |
| 23:45 | Task 7.2 complete: --factstream HA soak scenario (jstart/jfail commands, waldump jobstate decode, phantom-fact check via state_raw); found+logged bug-1416 (persist_recover DIVERGENT false-positive race vs catch-up burst) | src/controller/testsuite/test.c, waldump.c, .claude/skills/xce-test-ha/*.{sh,md} | scenario+tooling delivered, mechanism validated standalone, full soak blocked by bug-1416 (out of scope to fix) | ~15000 |
| 23:45 | Session end: 34 writes across 8 files (jun29-jul04-2026.md, raft_sim.c, tasks.md, test.c, waldump.c) | 19 reads | ~243517 tok |
| 23:58 | Session end: 34 writes across 8 files (jun29-jul04-2026.md, raft_sim.c, tasks.md, test.c, waldump.c) | 21 reads | ~249297 tok |
| 00:02 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | added error handling | ~927 |
| 00:16 | Edited src/include/persist.h | added error handling | ~634 |
| 00:16 | Edited src/include/persist.h | added error handling | ~273 |
| 00:16 | Edited src/controller/persist.c | expanded (+8 lines) | ~150 |
| 00:16 | Edited src/controller/persist.c | modified persist_last_applied_lsn() | ~124 |
| 00:17 | Edited src/controller/persist.c | added error handling | ~480 |
| 00:19 | Edited src/controller/persist.c | modified persist_recover() | ~338 |
| 00:19 | Edited src/controller/persist.c | added error handling | ~376 |
| 00:21 | Edited src/controller/persist.c | added 1 condition(s) | ~292 |
| 00:21 | Edited src/controller/ha.c | added error handling | ~471 |
| 00:21 | Edited src/controller/ha.c | added error handling | ~291 |
| 00:39 | Edited src/include/persist.h | expanded (+25 lines) | ~454 |
| 00:39 | Edited src/controller/persist.c | added 1 condition(s) | ~153 |
| 00:39 | Edited src/controller/myapp.c | added error handling | ~359 |
| 00:50 | Task 7.2 round-2 fixes complete: (1) fixed phantom-check `started` derivation in run-ha-soak.sh to check WAL_JOB_START presence directly instead of inferring from terminal state bits (real false-positive, proven via manual jfail-without-jstart repro against a live cluster); (2) root-caused+fixed bug-1416 with new persist_t.recovery_complete gate (persist.h/persist.c) closing persist_apply_pump_kick + ha_apply_single_entry/ha_apply_term_entry_via_library's in-memory apply during boot-time persist_recover(), PLUS persist_mark_recovery_pending() wired into myapp_run() (closes a live-verification-discovered gap: election-retry candidates apply live traffic before persist_recover() is ever first called). Verified: live 3-node --factstream soak PASS (2 rounds, kill+restart both, zero APPLY_DIVERGENT across 5 persist_recover calls); controller_unit_test 23-group sweep zero new failures (2 pre-existing task-6.1/6.2 FAILs unchanged); real-divergence fence tests still PASS | src/include/persist.h, src/controller/persist.c, src/controller/ha.c, src/controller/myapp.c, .claude/skills/xce-test-ha/run-ha-soak.sh | GREEN | ~9000 |
| 00:51 | Session end: 48 writes across 12 files (jun29-jul04-2026.md, raft_sim.c, tasks.md, test.c, waldump.c) | 28 reads | ~389364 tok |
| 05:07 | Session end: 48 writes across 12 files (jun29-jul04-2026.md, raft_sim.c, tasks.md, test.c, waldump.c) | 30 reads | ~407960 tok |

72629
.wolf/token-ledger.json
View file

File diff suppressed because it is too large Load diff

48
reports/jun29-jul04-2026.md Normal file
View file

@ -0,0 +1,48 @@
# Work Summary: June 30 – July 4, 2026
## Monday June 30 — Raft Election + Bug Fixes
- **Phase 5 RED** shipped with heartbeat-freshness guard
- **3 concurrency bugs fixed**: bug-1282 (ABBA deadlock netio.c), bug-1286 (NULL ep→owner), bug-1298 (endpoint UAF refcount)
- **Tasks 5.5–5.8 complete**: election timeout resonance fix (per-campaign jitter [T,2T], bounded terms), ASAN soak 6/6 rounds green (182 elections, zero runaway)
- Task 5.9 backfill identified blocker for 7.2 acceptance soak
## Tuesday July 1 — Spec Completion + Gap Audit
- **Raft_tla**: 12/13 tasks done (task 2 checkpoint refined via RED-reject-remediate, tasks 5-6 restore pipeline, task 7 ASAN regress shipped)
- **Ha-entity-snapshots**: 13/13 complete (checkpoint write, rotation guard, restore-before-advance)
- **Gap audit**: 3 blockers found blocking raft-ha-core-safety → raft-drive-async 7.2 acceptance gate (2 unscaffolded specs, 1 design-unapproved)
## Wednesday July 2 — Soak Validation + More Bug Fixes
- **Raft-ha-core-safety phase 7.2** soak green; raft-drive-async task 3.3 unblocked
- **3 more bugs fixed**: bug-1386 (silent-send), bug-1389 (65KB chunk corruption), bug-1381 (endpoint UAF)
- **Diagnostic tools** added: snapverify/snapdiff to xce_test
- **Ha-commit-durability phases 6.1–6.4** ext4-validated (5/5 rolling+cold-restart, zero committed-job loss)
- Enabled `g_drive_async_configpush_offload`
## Thursday July 3 — Raft-Apply-Decoupling Tasks 6.1–6.3
- **Task 6.1** (ordered apply pump): committed; discovered **bug-1409** (apply_pump_drain unbounded busy-loop, missing return-code check)
- **Task 6.2** (fire continuation post-apply): rejected by adversarial review for **bug-1415** (multi-vcpu race: apply-pump orphans pending-apply registrations → hang); healed at `persist_register_pending_apply()` with self-heal check; committed
- **Task 6.3** (apply/notify timing integration): approved, committed
## Friday July 4 — Tasks 6.4 + 7.1 + Bug Cleanup
- **Task 6.4** (quorum-loss catch-up + 4-vcpu burst tests): committed (2de410c), 30/30 consecutive runs pass
- **Task 7.1** (conformance harness extension): apply-after-commit + idempotent-reapply assertions in raft_sim.c; RED phase verified bidirectionally; 287/287 tests pass; adversarial review launched
- **Bug triage**: bug-1400 (submit.start_info leak), bug-1401 (xce_free_ctrlstatus macro freed inner fields not struct), bug-1396 (harness instrumentation)
- **buglog pruned**: 1406→721 entries (659 junk removed)
- **ha-consistency-invariants** spec closed
## Key Metrics
| Metric | Count |
|---|---|
| Commits | 7+ (across raft-apply-decoupling chain) |
| Bugs fixed | 9 (1282, 1286, 1298, 1381, 1386, 1389, 1400, 1401, 1415) |
| Bugs found+logged | 2 (1409, 1415) |
| Specs closed | 1 (ha-consistency-invariants) |
| Tests passing | 287/287 (xcelib_test) |
| ASAN soaks | 6-round election + 30-round apply-pump, all green |
## Insights
- **Adversarial review caught bug-1415**: a self-disclosed race condition the implementer flagged during task 6.2 — the review process worked as designed (RED→reject→remediate→GREEN)
- **buglog hygiene matters**: 659 junk entries (~47% of total) were removed — stale/duplicate observations degrade memory search quality and waste context tokens
- **The task chain dependency is tight**: 6.1→6.2→6.3→6.4 each depend on the prior's API surface (e.g., 6.2's `persist_take_pending_apply()` is 6.2's hand-off point to 6.3)

38
src/controller/ha.c
View file

@ -2229,6 +2229,26 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return prc;
}
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer the in-memory
* entity apply while this node's OWN boot-time persist_recover() replay+
* verification pass is still running. The entry is already durably
* persisted above (persist_wal_write_replicated, fsync'd) — only the
* ACK and applied-lsn advancement depend on that, per the persist-
* before-apply-before-ACK contract this function's own header comment
* documents, and bug-1043's precedent of decoupling durability/ACK from
* entity materialization. Applying here too, concurrently with
* replay_wal()'s single-threaded, unlocked bookkeeping (last_applied_lsn)
* and the live job table it reads, can produce a false APPLY_DIVERGENT
* verdict on a perfectly healthy boot (confirmed live). No entry is
* lost: this node's own in-progress replay_wal() pass will pick it up
* directly off disk if it lands within that pass's scan, and
* persist_recover()'s own deferred catch-up kick (persist_apply_pump_
* kick, issued right before it returns) covers it unconditionally
* either way, since apply_pump_read_entry() does a fresh on-disk lookup
* every call rather than trusting any earlier directory listing. */
if (!persist_recovery_complete(p))
return 0;
/* Logical entity restore only. persist_wal_write_replicated (above) already
* wrote the entry and owns the on-disk CRC chain + current_lsn — so we pass
* the entry fields directly rather than fabricating a header. (bug-778:
@ -2319,9 +2339,21 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
* find a committed job (it IS on disk; walcompare passes), looking like a
* committed-job loss. Re-apply here, mirroring the legacy path. Idempotent
* on a re-stream (restore_job upserts by jobid). WAL_NOOP / WAL_CONFIG_
* MARKER carry no entity payload — apply_wal_entry no-ops on them. */
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
* MARKER carry no entity payload — apply_wal_entry no-ops on them.
*
* bug-1416 (raft-apply-decoupling task 7.2 round 2): gated on
* persist_recovery_complete() for the identical reason documented on
* ha_apply_single_entry's own matching guard above — defer the
* in-memory apply while this node's own boot-time persist_recover()
* pass is in progress, never the ACK/applied_lsn advance below (the
* entry is already durably persisted by on_persist_entry's
* persist_wal_write_replicated, invoked via ha_consume_append_entries
* above). This node's own in-progress replay, or the deferred
* catch-up kick persist_recover() issues before returning, applies
* it instead. */
if (persist_recovery_complete(p))
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
ha_advance_applied_lsn(ha, e->lsn);
ha_send_wal_ack_reply(ha, ep, c, e->lsn);

17
src/controller/myapp.c
View file

@ -200,6 +200,23 @@ int myapp_run(myconf_t *myconf)
return rc;
}
/* bug-1416 round 2: close the recovery_complete gate HERE, at the
* earliest safe moment, not just around persist_recover()'s own call
* later (cluster_init_phase2, well after election). This node's own
* WAL/committed-state materialization is not truly "recovered" until
* that call runs -- and this process can receive and durably APPLY
* live-replicated entries (ha_apply_single_entry / ha_apply_term_
* entry_via_library) before then, e.g. while stuck retrying its own
* election as a candidate that keeps losing/timing out before
* discovering a live primary and stepping down (confirmed live: a
* node stuck ELECTING for tens of seconds while already accepting
* AppendEntries from a genuine leader). Set BEFORE ha_start_election_
* async/cluster_run() below can process any network traffic, so no
* live-replication apply is ever reachable before this gate is
* closed. persist_recover() clears it (and issues a catch-up kick)
* once its own replay+verification completes. */
persist_mark_recovery_pending(&c.persist);
/* ── Step 7: pre-load config_tag ─────────────────────────────────────── */
/* Pre-load config from disk if a local config file exists, so that this

102
src/controller/persist.c
View file

@ -618,6 +618,14 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
p->apply_verdict = APPLY_CLEAN;
atomic_store(&p->last_applied_lsn, 0);
/* bug-1416: defaults to true (nothing to gate) so every fixture that
* calls persist_init() but never persist_recover() -- most of this
* spec's own test suite -- sees identical, ungated apply-pump/live-
* replication behavior to before this flag existed. Only persist_
* recover() ever sets this false, for its own single-threaded
* replay+verify duration (see the field's doc comment in persist.h). */
atomic_store(&p->recovery_complete, true);
/* Durable-advance wake hook starts unregistered (ha-commit-durability 2.2):
* the commit engine registers it in commit_engine_init. Until then every
* flusher durable advance is a no-op wake. */
@ -3228,6 +3236,31 @@ void persist_apply_pump_kick(persist_t* p, void* pool_arg)
if (!p || !p->cluster)
return;
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer while this
* node's own boot-time persist_recover() replay+verification pass is
* still running -- last_applied_lsn is being written UNLOCKED by
* replay_wal() during that window (single-threaded-recovery
* assumption), and persist_verify_apply_correctness() is about to
* compare a live re-read of it (and of the live job table) against a
* WAL-scan-derived expectation. Letting this walk run concurrently
* would race BOTH: it could advance last_applied_lsn out from under
* replay_wal's own plain (non-CAS) store, and it mutates the same job
* table persist_verify_apply_correctness re-reads as "actual" --
* producing a false APPLY_DIVERGENT verdict on a perfectly healthy
* boot (confirmed live). No work is lost: persist_recover() sets this
* flag back to true and issues exactly one catch-up kick right before
* it returns, which will walk from last_applied_lsn+1 up through
* whatever committed_lsn has grown to by then -- a full, ordered
* drain, not a partial one. Intentionally checked with NO lock: a
* stale `true` read racing the flag's transition to `false` is
* harmless (the walk itself still only advances one entry at a time
* under apply_pump_lock, so at worst this call proceeds a few
* instructions into a window persist_recover() is about to close, and
* the deferred re-kick after recover() still runs); a stale `false`
* merely defers one no-op-cost kick to the next real trigger. */
if (!atomic_load_explicit(&p->recovery_complete, memory_order_acquire))
return;
int expected = 0;
if (!atomic_compare_exchange_strong_explicit(
&p->apply_pump_scheduled, &expected, 1, memory_order_acq_rel,
@ -3635,12 +3668,47 @@ uint64_t persist_last_applied_lsn(persist_t* p)
: 0;
}
/* bug-1416 (raft-apply-decoupling task 7.2 round 2) -- see persist.h's
* doc comment on the recovery_complete field for the full contract. */
bool persist_recovery_complete(persist_t* p)
{
return !p || atomic_load_explicit(&p->recovery_complete,
memory_order_acquire);
}
/* bug-1416 round 2 -- see persist.h's doc comment on this function for the
* full rationale (closes the gap between persist_init() and the FIRST
* persist_recover() call, which persist_recover()'s own false/true bracket
* cannot see). */
void persist_mark_recovery_pending(persist_t* p)
{
if (!p)
return;
atomic_store_explicit(&p->recovery_complete, false, memory_order_release);
}
/* ------------------------------------------------------------------ */
/* Public: persist_recover */
/* ------------------------------------------------------------------ */
int persist_recover(persist_t* p)
{
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): close the boot-time
* race window for the FULL duration of this call, not just the position
* check inside persist_verify_apply_correctness -- see the
* recovery_complete field's doc comment (persist.h) for the complete
* rationale. Held false until the natural (success) return at the
* bottom of this function, which restores it and issues one deliberate
* catch-up kick. The two hard-I/O-error early returns below
* (load_snapshot / replay_wal failure) deliberately do NOT restore it:
* those are pre-existing, rare disk-I/O-level failures the caller
* already treats as fatal to this boot attempt (cluster_init_phase2
* tears down persist_t on a nonzero return) -- leaving the gate closed
* is the conservative, fail-static choice (never let the live apply
* pump or live-replication path mutate state on top of a persist_t
* whose own recovery never finished), not a new hazard. */
atomic_store_explicit(&p->recovery_complete, false, memory_order_release);
/* --- Step 1: find latest snapshot (highest LSN filename) --- */
char snap_file[4096] = "";
uint64_t snap_lsn = 0;
@ -3823,6 +3891,40 @@ int persist_recover(persist_t* p)
/* --- Step 4: retention pass (TTL check) — no-op in P0 (no entities) --- */
/* bug-1416: replay + verification is fully complete -- safe to let the
* live apply pump and the standby live-replication apply path resume
* mutating state (see the recovery_complete field's doc comment).
* Restored unconditionally here regardless of the verdict computed
* above: a genuine APPLY_DIVERGENT already fenced this member via
* persist_recover_loud_fence_if_unrecoverable (non-promotable, stays
* alive, keeps retrying) -- that sticky fence and its clearing logic
* (task 5.3) are untouched by this flag, which governs ONLY whether
* in-memory apply may proceed, not whether this member may lead. Issue
* one deliberate catch-up kick for anything that committed locally (or
* arrived via live replication and was durably persisted but had its
* entity-apply deferred, see ha_apply_single_entry / ha_apply_term_
* entry_via_library) while this pass ran -- persist_apply_pump_run's
* own walk drains fully from last_applied_lsn+1 through the current
* committed_lsn, so this is a complete, ordered catch-up, not a partial
* one; a harmless no-op if nothing queued. */
atomic_store_explicit(&p->recovery_complete, true, memory_order_release);
/* n_vcpus==0 is this codebase's own established "not yet initialized"
* sentinel for vtaskmgr_t (mirrors cluster_init_phase2's own
* `if (c->vtaskmgr.n_vcpus == 0) vtaskmgr_init(...)` guard) -- a real
* cluster always has n_vcpus > 0 by the time persist_recover() runs
* (vtaskmgr_init happens earlier in myapp_run()/phase2), while a bare
* recovery-focused unit-test cluster_t (none of which call
* vtaskmgr_init, confirmed by inspection) leaves it zeroed. Passing a
* pointer to an uninitialized vtaskmgr_t into vtask_enq_classified
* would be unsafe, so fall back to persist_apply_pump_kick's own
* existing, already-exercised NULL-pool inline path in that case --
* identical to how every apply-pump test fixture in this spec already
* exercises the pump. */
void* pool = (p->cluster && p->cluster->vtaskmgr.n_vcpus > 0)
? &p->cluster->vtaskmgr
: NULL;
persist_apply_pump_kick(p, pool);
return 0;
}

146
src/controller/testsuite/test.c
View file

@ -107,6 +107,8 @@ extern int cmd_shutdown(int, char*[]);
extern int cmd_set(int, char*[]);
extern int cmd_reslist(int, char*[]);
extern int cmd_join_worker(int, char*[]);
extern int cmd_jobstart(int, char*[]);
extern int cmd_jobfail(int, char*[]);
extern int cmd_join_sched(int, char*[]);
extern int cmd_workerlist(int, char*[]);
extern int cmd_joblist(int, char*[]);
@ -206,7 +208,9 @@ static command_t worker_commands[] = {
{ "help", cmd_help, "Display this text" },
{ "quit", cmd_quit, "Quit xce test" },
{ "join", cmd_join_worker, "Join the cluster" },
{ "jstart", cmd_jobstart, "tell controller jobstart" },
{ "jfinish", cmd_jobfinish, "tell controller jobfinish" },
{ "jfail", cmd_jobfail, "tell controller jobfail" },
{ "resupdate", cmd_resupdate, "update res vals" },
{ "echo", cmd_echo, "Print a message" },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
@ -1534,6 +1538,16 @@ static void print_jobinfo(xce_job_t* job)
fmt_jobid(idstring, sizeof(idstring) - 1, job->jobid);
printf("JobID: %s\n", idstring);
printf("state: %s\n", job_state_str(job->state));
/* raft-apply-decoupling task 7.2: job_state_str's if/else-if chain checks
* JOB_STAT_RUN before JOB_STAT_EXIT, so a job that is simultaneously RUN
* and EXIT|PERR (e.g. a worker-reported FAIL received without a prior
* FINISH — apply_job_fail_transition ORs in EXIT|PERR without clearing
* RUN) prints as "RUN", silently hiding the EXIT/PERR bits from the
* pretty string. Print the raw bitmask too so a scriptable caller (the
* xce-test-ha factstream regression scenario's phantom-fact check) can
* test for a specific bit's presence unambiguously instead of relying on
* the display string's priority ordering. */
printf("state_raw: 0x%x\n", job->state);
printf("Queue: %s\n", job->submit->queue);
printf("User: %s\n", job->submit->user);
printf("Slots: %d,%d\n", job->submit->mintasks, job->submit->maxtasks);
@ -2443,6 +2457,69 @@ int cmd_join_worker(int argc, char* argv[])
return 0;
}
/* jstart -j jobid [-p pid] — raft-apply-decoupling task 7.2: synthetic
* worker-fact-stream primitive. Mirrors cmd_jobfinish's existing pattern
* exactly (same getopt/parse_jobid/SET_ERROR shape) but drives the
* xcecli_notify_jobstart wire call that cmd_jobfinish's sibling
* xcecli_notify_jobfinish already exercises — libxce (client.c) already
* implements the full worker-fact API (jobstart/jobfinish/jobfail/...); this
* command was the missing REPL-level primitive for jobstart specifically.
* Drives the SAME production receipt path (on_job_start -> record_job_start_fact
* -> persist_submit_job_fact -> durable append -> commit -> apply pump ->
* apply_committed_entry()) that a real worker's report would, without
* requiring a real scheduler dispatch (on_job_start only requires the job to
* already exist in jobmgr, i.e. have been submitted). */
int cmd_jobstart(int argc, char* argv[])
{
int rc;
int opt;
uint64_t jobid = 0;
unsigned int pid = 0;
unsigned int pids[1];
job_pidinfo_t info;
optind = 1;
memset(&info, 0x0, sizeof(info));
while ((opt = getopt(argc, argv, "j:p:")) != -1) {
switch (opt) {
case 'j':
parse_jobid(&jobid, optarg);
break;
case 'p':
pid = (unsigned int)atoi(optarg);
break;
default:
printf(" jstart -j jobid [-p pid]\n");
return -EINVAL;
}
}
if (jobid == 0) {
printf(" you need specify a positive jobid\n");
return -EINVAL;
}
pids[0] = (pid != 0) ? pid : (unsigned int)getpid();
info.jobid = jobid;
info.npids = 1;
info.pids = pids;
info.npgids = 0;
info.pgids = NULL;
info.exdata = NULL;
rc = xcecli_notify_jobstart(active_ctx->ctx, &info);
if (rc < 0) {
SET_ERROR(-rc);
perror("xcecli_jobstart");
return rc;
}
printf("ok\n");
return 0;
}
int cmd_jobfinish(int argc, char* argv[])
{
int rc;
@ -2451,6 +2528,17 @@ int cmd_jobfinish(int argc, char* argv[])
int code = 0;
xce_jobfinish_t finish;
/* raft-apply-decoupling task 7.2: reset optind so repeated jfinish
* invocations within one xce_test process (the fact-stream soak scenario
* calls this many times in a single worker session) each re-parse their
* own argv from the start. Without this, getopt's optind carries over
* from the PRIOR command's parse position, silently skipping this
* command's own "-j jobid" on the second and later calls (jobid stays 0
* and the command errors out) — matches the reset convention most other
* cmd_ handlers in this file already follow (e.g. cmd_join_worker just
* above, cmd_jobsub, cmd_resupdate). */
optind = 1;
while ((opt = getopt(argc, argv, "j:c:")) != -1) {
switch (opt) {
case 'j':
@ -2485,6 +2573,64 @@ int cmd_jobfinish(int argc, char* argv[])
return 0;
}
/* jfail -j jobid [-c code] [-s stage] — raft-apply-decoupling task 7.2:
* synthetic worker-fact-stream primitive, sibling of jstart/jfinish above.
* Drives xcecli_notify_jobfail -> on_job_fail -> record_job_fail_fact ->
* persist_submit_job_fact_ex (the _pad-discriminated path, task 6.1 round 2)
* -> apply_job_fail_transition() at apply time. Reuses the existing
* job_failinfo_t shape already exercised internally by this file's own
* notify_job_fail() helper (used by run_job() on a local exec failure). */
int cmd_jobfail(int argc, char* argv[])
{
int rc;
int opt;
uint64_t jobid = 0;
int code = 0;
int stage = JOB_EXEC_STAGE_POST_EXEC;
job_failinfo_t failinfo;
optind = 1;
memset(&failinfo, 0x0, sizeof(failinfo));
while ((opt = getopt(argc, argv, "j:c:s:")) != -1) {
switch (opt) {
case 'j':
parse_jobid(&jobid, optarg);
break;
case 'c':
code = atoi(optarg);
break;
case 's':
stage = atoi(optarg);
break;
default:
printf(" jfail -j jobid [-c code] [-s stage]\n");
return -EINVAL;
}
}
if (jobid == 0) {
printf(" you need specify a positive jobid\n");
return -EINVAL;
}
failinfo.jobid = jobid;
failinfo.stage = stage;
failinfo.code = code;
failinfo.detail = "synthetic fact-stream failure (xce_test jfail)";
rc = xcecli_notify_jobfail(active_ctx->ctx, &failinfo);
if (rc < 0) {
SET_ERROR(-rc);
perror("xcecli_jobfail");
return rc;
}
printf("ok\n");
return 0;
}
static int set_resval(xce_resval_t* vals, int idx, char* name, char* val)
{
static struct {

56
src/controller/testsuite/waldump.c
View file

@ -84,6 +84,43 @@ static int decode_submit_info(const uint8_t* payload, uint32_t len,
return 0;
}
/* True for the WAL types whose payload is the flat, non-capnp
* wal_jobstate_t {jobid, state, _pad} (wal_types.h) — every worker-reported
* job-state fact (START/FINISH/FAIL/STOP/RESUME/UPDATE/RUSAGE/CKPT/PREASON).
* Added for raft-apply-decoupling task 7.2: the kill-mid-fact-stream live
* regression needs to verify, per jobid, which fact WAL entries actually
* committed on each node — before this, waldump only decoded jobid for
* WAL_JOB_SUBMIT, leaving every fact entry printed as "jobid=?". */
static int wal_type_is_jobstate(uint32_t type)
{
switch (type) {
case WAL_JOB_START:
case WAL_JOB_FINISH:
case WAL_JOB_FAIL:
case WAL_JOB_STOP:
case WAL_JOB_RESUME:
case WAL_JOB_UPDATE:
case WAL_JOB_RUSAGE:
case WAL_JOB_CKPT:
case WAL_JOB_PREASON:
return 1;
default:
return 0;
}
}
/* Decode a wal_jobstate_t payload. Native-endian flat struct (single-node WAL
* — see wal_types.h), so this is a bounds-checked copy, not a parser. Returns
* 0 on success, -1 if the payload is too short to hold the struct. */
static int decode_jobstate_info(const uint8_t* payload, uint32_t len,
wal_jobstate_t* out)
{
if (!payload || len < sizeof(*out))
return -1;
memcpy(out, payload, sizeof(*out));
return 0;
}
/* True if `dir` directly contains at least one seg-*.wal file. */
static int dir_has_segments(const char* dir)
{
@ -342,6 +379,25 @@ static void scan_segment(const char* seg_path, FILE* out,
}
*prev_submit_jobid = jobid;
*prev_submit_pcrc = pcrc;
} else if (wal_type_is_jobstate(hdr.type)) {
/* Worker-reported job-state fact (task 7.2): decode the flat
* {jobid, state, _pad} payload so a per-jobid committed-fact check
* (e.g. "did WAL_JOB_FINISH for jobid X actually commit on this
* node") can grep waldump output directly, the same way the
* existing WAL_JOB_SUBMIT branch above already supports
* expect-job/joblist cross-checks. */
wal_jobstate_t js;
if (decode_jobstate_info(payload, hdr.payload_len, &js) == 0) {
char jidStr[128];
if (fmt_jobid(jidStr, sizeof(jidStr), js.jobid) < 0)
snprintf(jidStr, sizeof(jidStr), "%" PRIu64, js.jobid);
snprintf(info, sizeof(info), "jobid=%s state=0x%x pad=0x%x",
jidStr, js.state, js._pad);
} else {
snprintf(info, sizeof(info), "jobid=?");
}
/* a non-submit entry breaks the adjacency for dup detection */
*prev_submit_jobid = 0;
} else {
/* a non-submit entry breaks the adjacency for dup detection */
*prev_submit_jobid = 0;

71
src/include/persist.h
View file

@ -363,6 +363,38 @@ struct persist_s {
_Atomic uint64_t last_applied_lsn;
apply_verdict_t apply_verdict;
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): closes a boot-time
* TOCTOU between persist_recover()'s replay+verification pass above and
* two concurrently-reachable live mutation paths that write the SAME
* last_applied_lsn field / mutate the SAME live job table this pass
* reads: the live apply pump (persist_apply_pump_run, driven by any of
* the 6 commit.c committed_lsn-advance sites via persist_apply_pump_kick)
* and the standby live-replication apply path (ha_apply_single_entry /
* ha_apply_term_entry_via_library, ha.c). Both are reachable the moment
* the iomgr event loop starts processing connections in cluster_run()
* (myapp.c) -- which happens well BEFORE cluster_init_phase2() reaches
* its persist_recover() call (an election round-trip and the whole of
* phase2's own setup earlier in the same function separate the two).
* Without this flag a fast-electing peer's replicated traffic can mutate
* last_applied_lsn/the job table WHILE this node's own persist_recover()
* is still computing its replay-derived expectation, producing a FALSE
* APPLY_DIVERGENT verdict on a perfectly healthy boot (confirmed live,
* reproduced on a fresh 3-node cluster's ordinary two-phase bootstrap --
* committedLsn was identical across all nodes when it fired).
*
* Defaults to true (persist_init) so every existing unit-test fixture
* that calls persist_init() directly and never calls persist_recover()
* (the overwhelming majority of this spec's own test suite) sees
* IDENTICAL, ungated behavior to before this field existed. persist_
* recover() is the ONLY function that ever sets this false -- for the
* exact, single-threaded duration of its own replay+verify pass -- and
* always restores it to true (plus issues one deliberate catch-up kick)
* before returning, regardless of the resulting verdict. This does NOT
* touch the sticky recovery_unrecoverable fence or its clearing logic
* (task 5.3) -- a genuinely DIVERGENT verdict still fences exactly as
* before; this flag only prevents the comparison ITSELF from racing. */
_Atomic bool recovery_complete;
/* Paths (derived from workdir + name via persist_set_workdir) */
char wal_path[4096];
char snap_path[4096];
@ -706,6 +738,45 @@ persist_recover_loud_fence_if_unrecoverable(persist_t* p,
bool persist_recovery_unrecoverable(persist_t* p);
void persist_clear_recovery_unrecoverable(persist_t* p);
/* persist_recovery_complete (bug-1416, raft-apply-decoupling task 7.2 round
* 2) -- true unless a persist_recover() call is CURRENTLY in progress on
* this persist_t (see the recovery_complete field doc comment). Consulted
* by the standby live-replication apply path (ha_apply_single_entry /
* ha_apply_term_entry_via_library, ha.c) to defer in-memory entity mutation
* while a boot-time replay+verification pass is running, so it never races
* that pass's expectation-vs-actual comparison. Durability (WAL persist)
* and the replication ACK are NEVER gated by this -- only the in-memory
* apply; a deferred entry is picked up either by the in-progress replay
* pass itself (if it observes the entry on disk) or by the single catch-up
* kick persist_recover() issues right before returning. Safe on NULL p
* (returns true -- nothing to defer against). */
bool persist_recovery_complete(persist_t* p);
/* persist_mark_recovery_pending (bug-1416 round 2 -- live-cluster reproduction
* confirmed a gap in the original fix). persist_recover() itself already
* brackets recovery_complete false/true around its own call, but that window
* is too NARROW for the real production boot sequence: a node can be
* receiving and durably APPLYING live-replicated entries (ha_apply_single_
* entry / ha_apply_term_entry_via_library -- e.g. while stuck retrying its
* own election as a candidate that keeps losing/timing out before it
* discovers a live primary and steps down) for an arbitrarily long stretch
* BEFORE cluster_init_phase2() ever gets to call persist_recover() for the
* first time. During that whole stretch recovery_complete still holds
* persist_init()'s default (true), so the live-replication apply path runs
* UNGATED and can mutate jobmgr/last_applied_lsn well before persist_
* recover()'s own baseline capture, producing the exact same false
* APPLY_DIVERGENT this flag exists to prevent, just from an earlier entry
* point than persist_recover() alone can see.
*
* Call this ONCE, in the real boot path only (myapp_run, immediately after
* persist_init()/persist_start() succeed and BEFORE election / cluster_run()
* can process any network traffic), to close the gate from the earliest
* safe moment instead of only from persist_recover()'s own entry. Safe on
* NULL p (no-op). Never called by any test fixture -- every unit test that
* constructs a persist_t directly (bypassing myapp_run entirely) is
* unaffected and keeps persist_init()'s default (true, ungated). */
void persist_mark_recovery_pending(persist_t* p);
/* persist_verify_apply_correctness (raft-apply-decoupling task 5.1, Req
* 8.1/8.2). Pure comparison — mirrors persist_recover_decision's separation
* of decision from action: no logging, no fence side effects, no mutation of