run-ha-soak.sh: scope every pgrep/pkill to this run's --workdir, not just
binary name + --myname. An unscoped pkill -9 -x xce_controller (old
cleanup()) killed every xce_controller on the machine, including a
different concurrently-running soak's live primary — root cause of the
first bug-1436 misdiagnosis (a killed-out-from-under-it primary produced
a genuine, correctly-reported ECONNRESET that looked like a libxce bug).
.wolf/*: anatomy/cerebrum/buglog updated for bug-1435/1436, session and
token-ledger trail.
buglog.json had been reset to a small auto-detected-noise-only file at some
point this session, losing the real 738-entry history (still present in git
HEAD). Restored the original list verbatim and appended bug-1421 (reusing
the ID a prior session had already reserved for this exact election
regression before the file got wiped) and bug-1422, both logged with full
root-cause/fix detail in this branch's other commits.
memory.md carries the OpenWolf session action log (auto-appended by hooks
plus this session's manual entries) for both fixes.
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.
Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
buglog.json had two broken auto-detection hook signatures polluting it:
- 207 entries "Missing error handling in X" / fix "Added try/catch block"
(nonsensical on .md/.capnp files, and C has no try/catch — never a real
applied fix)
- 452 entries with a literally-empty template ("Significant refactor of "
with no subject ever filled in) / fix "Rewrote N->M lines"
Removed both (659 total). Also merged 24 legitimate duplicate bug IDs
(re-logged/updated entries sharing an ID) into single entries — kept the
most recent content as primary, unioned tags/related_bugs, summed
occurrences — and stripped noise tags ('auto-detected', 'refactor') that
had leaked onto real entries from a since-merged junk duplicate. Net:
1406 -> 721 entries, verified zero remaining duplicate IDs and zero
remaining junk signatures.
.gitignore: added patterns for the scratch soak-run logs and codegraph.html
this session generated at repo root (already deleted from the tree), and
fixed the ucai-formatter-cache.local.json pattern to match its nested
occurrences (.claude/skills/*/.claude/, src/*/.claude/) instead of only the
repo-root one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).
- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
step (WAL append + majority-commit barrier) returning the commit verdict
directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
reject on <0 before firing the now notification-only AFTER hook
(canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
commit_barrier_reply 22/0, commit_durability 38/0.
Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.
Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Verdict: raft-drive-async introduces NO new regression.
- Unit: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites green;
dispatch/snapshot/configpush flags default OFF, inert).
- Live kill-based failover soak on a real fs (~/tmp), release + ASAN builds:
zero committed-JOB loss every completed round (jobs 3->12 monotonic).
- The 6-round CHAOTIC soak does not run clean to round 6 (fails ~round 4), but a
baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4
IDENTICALLY (election stall / reconverge). The round-4 committed-prefix
divergence at LSN 9 is the documented pre-existing bug-851/795 family
(chaotic-failover silent divergence), present before Phase 3 and out of scope
for this spec. Logged as bug-1225 (baseline-confirmed).
Also: make the xce-test-ha soak driver's controller/xce_test paths
env-overridable (XCE_HA_BIN / XCE_HA_XT, backward-compatible defaults) — used to
run the baseline binaries through the same driver for the regression comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
With the dispatch flags flipped ON in-test, prove the three receiver-side
liveness properties (production defaults stay OFF):
- P1: while a snapshot (and config-push) transfer is parked in flight on the
slow lane, another endpoint's peer-control message is still processed
promptly on the vtask pool (different thread), with the transfer still
parked — the offloaded transfer does not block the drive path.
- P2: a peer-control ack (classified VTASK_LIVENESS) runs promptly even while
the pool is saturated to the LivenessAdmission reservation boundary with
REQ_BLOCK tasks (one more REQ_BLOCK fast-rejects with -E_XCE_NOT_READY,
proving real saturation). Narrowed to the liveness-dispatch-under-saturation
assertion; commit-advance itself is covered by the commit-engine tests.
- P3: while one endpoint's peer-control body is parked (holding a worker + that
ep busy), another endpoint's body runs concurrently on a different worker —
per-endpoint FIFO does not serialize across endpoints.
All deterministic (cond-var gates, bounded polls). Fixed a flaky teardown-race
hang (bug-1224): the tests released a parked vtask body then immediately called
the cancel-based vtaskmgr_fini while the just-released body was still unwinding
(P2's REQ_BLOCK bodies racing blk_lock teardown the worst). Fix adds end-of-body
completion counters polled to the full expected count before fini, so every
pool body has fully returned (worker idle) before the cancel. slowlane_fini is
graceful (join) so the lane side was already safe.
Verified: 5/5 deterministic isolated runs; full controller suite EXIT=0;
ASan-clean. Completes task 3.4 (3.4a UAF fix + 3.4b tests); the production
flag-flip remains deferred to after the 4.1 partition soak + 4.2 regression
sweep.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).
Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):
- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
FIFO makes it run strictly after any in-flight/queued same-ep peer-control
vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
run_task's post-body endpoint_unblock_tasks (which would touch the freed
ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.
- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
until dispatch_inflight drains to 0. So every dispatch that read the flag
false finishes its enqueue before mark returns (FIFO-before the teardown
vtask); any later dispatch reads the flag set and drops the packet. ep->lock
is deliberately NOT held across the enqueue (it inverts with the taskq lock in
vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
held window is just a calloc + non-blocking enqueue on the single io_worker.
Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.
Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
push_config_transport_only spawns one thread per standby then joins them with a
CONF_PUSH_TIMEOUT_MS bounded wait — blocking the caller. On the requester-less
drive-thread paths (promotion + heartbeat convergence) that wait stalls the
io_worker. Route those through ha_config_push_offloadable, gated by
g_drive_async_configpush_offload (default OFF → inline, byte-for-byte legacy;
ON → slow lane, non-blocking).
Flag ON deep-copies the config bytes + sha into a lane ctx (caller keeps owning
and freeing its own buffer); a single cluster-wide single-flight guard
(ha_state_t.configpush_inflight scalar CAS + lane key=1) bounds it to one
config-push fan-out in flight; the in-flight flag is cleared on every terminal
path (two alloc failures, -EEXIST, hard submit error, lane completion). Send
body factored into ha_config_push_do_send.
Three requester-less drive-thread call sites converted: raft_cb_become_leader
(promotion), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY
FSM promotion branch. The admin-upgrade barrier (push_config_to_standbys, has a
waiting requester) and join.c are untouched.
Lands DORMANT like tasks 3.1/3.2: the lane reads c->ha.standby_endpoints and
writes those borrowed standby endpoints, racing io_worker RDHUP teardown if run
off-thread (bug-1197 family). Flag OFF keeps push+teardown serialized on the
io_worker. Task 3.4 flips this ON together with the snapshot and peerctrl flags
after establishing endpoint-lifetime discipline.
Tests (controller suite EXIT=0): test_configpush_offload_runs_on_lane,
test_configpush_inline_when_flag_off, test_configpush_offload_dedups.
Adversarial cpp-review: no CRITICAL/HIGH/MEDIUM findings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Part B (LIVE, default-ON): trim_wal_segments now bounds WAL-segment deletion
by repl_trim_lsn(&p->repl, active_seg_lsn) = min(active segment LSN, every
active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn),
after repl_expire_disconnect_slots ages out stale slot guards. floor <=
active_seg_lsn structurally, so the guard can only preserve more segments,
never delete one a live standby still needs — closes the bug-795/828
trim-vs-cursor race for connected/recently-disconnected standbys (bug-1147
stopgap; empty/cursor=0 join still needs entity-bearing snapshots).
Part A (DORMANT, g_drive_async_snapshot_offload default OFF): ha_snapshot_send_
dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy,
on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body
factored into ha_snapshot_do_send. Per-peer single-flight via
ha_state_t.snapshot_inflight[] CAS + slowlane key; in-flight flag cleared on
all four terminal paths, mirroring the reconnect offload (task 2.1).
Lands dormant like task 3.1: snapshot_lane_fn borrows a standby endpoint it
does not own, so flag-ON would race io_worker RDHUP teardown (cross-thread UAF,
bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker.
Task 3.4 is the single gate that flips both this and the peerctrl flag ON after
establishing endpoint-lifetime discipline; bug-1198 stays open by design until
then. The offload flag was initially authored default-ON (would ship the UAF)
and corrected to OFF in ha.c and the ha.h accessor doc (bug-1210).
Tests (controller 196/0, 0 new leaks): test_snapshot_offload_runs_on_lane,
test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer,
test_trim_guard_preserves_standby_segments,
test_trim_without_standbys_proceeds_normally. Adversarial cpp-review: GO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Ran the live 3-node E2E (the last open task). Result:
PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led
(ctrl-1,ctrl-2,ctrl-3) (exit 0)
Live-covered: kill-primary -> promoted standby owns all confirmed jobs
(zero loss across 8 failovers, jobs 3->30, expect-job ok each round);
restart ex-primary -> converged rejoin, 3-of-3 each round, no phantom jobs;
deterministic leadership coverage (ctrl-3 forced via RequestVote-freshness
fence). The soak also served as the live regression for bug-1077 (~40
failover client round-trips, no SEGV) after rebuilding the controller (the
deployed binaries predated that fix).
Not exercised live (remain unit-covered, tasks 2/4/5): withhold-ACK barrier
single-outcome (test_demotion_convergence) and isolate/heal quorum-WARN
(test_convergence_verification). All 8 tasks now [x]; spec closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
fsync_worker slept the full wal_sync_ms in one uninterruptible nanosleep()
before re-checking fsync_stop. persist_stop sets the flag and immediately
pthread_join()s, so the join blocked until the in-flight sleep returned —
up to a full wal_sync_ms per teardown. Tests set wal_sync_ms=5000 ("slow
fsync — no races"), so each persist_stop cost ~5s; across the ~7 suites
that use it the cumulative teardown time blew past the 300s controller-test
meson cap, killing the suite with SIGTERM.
Replace the nanosleep with pthread_cond_timedwait on a new fsync_mutex/
fsync_cond (added to persist_t, init in persist_init, destroyed in
persist_fini). persist_stop now sets fsync_stop and signals fsync_cond
under fsync_mutex before joining, so the worker wakes immediately. fdatasync
still runs under p->lock, released before re-acquiring fsync_mutex — no lock
nesting. The "commit: WAIT deadline expired" log lines were unrelated
intentional negative tests (50-300ms), not the cause.
Full controller suite now completes in ~10s (was >300s TIMEOUT); 0 failures.
Logged as bug-604.
- Mark the PreVote denial log-reason nit RESOLVED in ha-prevote tasks notes (47f7338).
- File ha-elect-ctx-lifecycle spec scaffold (spec.json + brief.md) capturing the
pre-existing, bounded, UAF-sensitive election-carrier leak and the refcount/
teardown-ordered-free approach across both election and PreVote paths. Deferred
by decision — not folded into ha-prevote to avoid a risky lifetime refactor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full controller suite green under ASAN (508 PASS / 0 FAIL / 0 sanitizer
aborts, independently re-censused). All 16 tasks complete; barrier
before reply proven across all restructured opcodes; classification
guard active; barrier-before-state-removal precondition for
ha-single-leader-role satisfied. spec phase=implemented; roadmap
updated. Sub-spec 2 of 4 of the ha-consistency-invariants umbrella.
43-site classification (8 category-A barrier-gated, 6 category-B
receipt-ACK, ~29 category-C) recorded in tasks.md. Reviewer-confirmed
exhaustive from the producer side and the opcode dispatch: every
WAIT-class WAL producer reaches its requester reply only through a
commit barrier; no handler bypasses wal_classify. Zero production
changes. Noted: WORKER_UPDATE has a reply-less second producer
(mark_worker_back liveness sweep) — not a violation, future RETAIN
refinement candidate.
Guards at the top of mark_job_start/finish/update under the caller's
job lock, before BEFORE_EVENT: duplicate (FINISH on terminal) → INFO
no-op skipping the AFTER_EVENT hook (no WAL append, lsn frozen);
terminal→non-terminal backward (START/UPDATE) → WARN reject preserving
state. Shared job_state_is_backward predicate provably admits every
production forward merge (PDONE/PERR/suspend bit-OR) while rejecting
overwrites. mark_job_fail intentionally unguarded (its merges are
always terminal-forward; job_ctrl pre-gates terminal). 5/5 guard tests
green; full job-lifecycle suite green; census 0 FAIL.
3 RED carriers (duplicate FINISH on DONE appends a 2nd WAL entry +
overwrites state; START on terminal clobbers it; replay non-equivalent)
+ 2 preservation pins (forward postdone PDONE-merge, fail-after-finish
PERR-merge) that 6.2 must not break. Reviewer-confirmed: the
UPDATE-clears-terminal arm is production-unreachable (pinned, not
driven). 22 tests / 3 RED; census 508 distinct triples, only those 3
fail.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
notify/update/setpreason/clrpreason admitted in CTRL_PRIMARY without
write quorum (Req 3.2 / umbrella 2.8 / bug-396): all four terminate in
receipt-style RETAIN fact paths — reviewer-traced no-block proof (no
admitted opcode reaches a barrier/WAIT path; request opcodes stay
rejected). 21/21 group; census 494/494 zero FAIL; RED→GREEN
independently reproduced via stash-rebuild.
4 RED carriers (notify/update/setpreason/clrpreason must be admitted
without quorum — currently NOT_READY, the bug-396 hole) + 2 pins (six
request opcodes stay rejected; quorum-present admits all ten). Census
494 = 488 + 6; exactly 4 FAILs.
4 green pins: handle_jobdispatch replies receipt-only with zero WAL
appends through a real 2-alloc dispatch cycle (notify outcomes intact,
reply-precedes-processing); set/clr_preason and do_update_op rusage
stay receipt-style with RETAIN-pending fact appends ACK-committed.
Umbrella Reply-Ordering Contract cited per pin. DISCOVERY bug-451
(reviewer-verified): RUSAGE hook va_list type confusion — 4-byte OOB
read + garbage state in WAL_JOB_RUSAGE payloads; fix deferred to 7.1.
Census 488; zero failures.
7 cases: two covered short-circuits, timeout with committed_lsn
unmoved, covering-ACK release, sync_timeout_ms fallback (bounded both
directions), n_peers=1 standalone, NOT_READY passthrough via cap
saturation. Watermark resolved to current_lsn - 1 (design's literal
phrasing unsatisfiable per persist.h:71 invariant) with off-by-one
killing power both ways. 15 tests / 7 RED; suite baseline zero.
Verified table implemented (STOP/RESUME/UPDATE/FAIL RETAIN per
supersession; WORKER detail switch; CHECKPOINT LOCAL); unknown type =
ERROR + XCE_TESTING abort + production RETAIN fallback. All three
hooks route through wal_hook_append_classified (zero bare async
appends); RETAIN registers with the engine lock-free; facts propagate
local append rc only. 8/8 group tests green; full suite 465/465.
