- wal-layout-check.py: every WAL segment must hold one contiguous LSN run
starting at the LSN in its name (bug-1920 signature); the sequential,
chaotic and truncated-fence modes fail on a misfiled segment.
- --truncated-fence: breaks ctrl-1's WAL tail CRC so it boots
RECOVER_TRUNCATED, makes it win an election (promotion REFUSED), then
requires the fence to be healed and released after catch-up and the
node to lead with zero job loss (bug-1919 lifecycle, proven from logs).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The HA array soak checks small-array elements 1..N, large-array first,
middle and last as 1, N/2 and N, status/cancel/completion on 2, N/2+1
and N-1, and the chaos-retry round over seq 1 N; the multihost
verification drives element [1]. Comments now say that after jobsub -a N
the success line and $jobid name the array identity. Historical
live-run quotes that show [0] are kept and annotated as predating the
change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
buglog: bug-1433, bug-1450, bug-1451, bug-1469, bug-1474, bug-1728,
bug-1741 fixed; bug-1870 note (load_snapshot off-by-one follow-up,
cb1415d7); new bug-1871 (investigating): snapshot transfer to a rejoining
standby can stall mid-stream after an endpoint unblock.
run-ha-soak.sh --force-snapshot: poll up to 30s for 'HA: snapshot' instead
of one grep right after 3-of-3 convergence, which can be reported before
the last chunks are read (2 of 4 runs failed spuriously; 2 reruns with the
poll passed, installs 4-6 ms after the send).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py: --ids syncs named bugs regardless of dates (OR'd with
--since). Bulk triage bumped last_seen/status_updated across most of the
buglog, so --since 2026-09-01 matched ~1150 bugs while recently fixed older
bugs (bug-1798..1804, 1714, 1754, 1850, 1854) could never be reached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since bug-1790's fix (9caf1424) counts a job's full task total against
global limits, the repo's limit1 (all users except rshen share 3 slots)
pends xce_test's root-submitted 33-task job forever, so the multihost
e2e fixture failed even though the allocator is correct.
deploy-ha.sh gains an optional XCE_DEPLOY_CONFIG_DIR (default unchanged:
src/configs). run-multihost-verify.sh copies src/configs into its deploy
dir, replaces limits.yaml with an empty list, and deploys with that
override.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py fixes found on the first live run (bug-1855 follow-up):
- 204/empty-body replies (label DELETE) no longer crash in json.loads
- every request has a 60s timeout, so a stalled call fails instead of hanging
- spec.json sync state is inserted as a trailing "gitea" block instead of
re-serializing the file (15 hand-formatted spec.json files would reflow)
- implementation issue body labels generated vs approved correctly; the
validation body uses the same done-phase set as the close rules
Project memory now routes all buglog writes through the buglog skill:
CLAUDE.md key rule, a cerebrum Do-Not-Repeat entry (with the older
duplicate-id / json.dump / tags-vs-fix lessons tagged as superseded), and
xce-audit-invariants step 5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The buglog had no status field, so "is this bug open?" meant reading free
text, and gitea-sync closed any issue whose `fix` text was non-empty --
including "NOT FIXED"/"PENDING"/"PARTIALLY FIXED" ones (bug-1855).
- Backfill `status` (+ status_updated) on all 1,193 entries. The 85 entries
whose text said not-fixed/pending/out-of-scope were checked against HEAD
and git history; the rest are `fixed`. Result: 36 open, 10 partial,
2 investigating, 5 cannot-reproduce, 4 wontfix, 1 duplicate.
Insert-only diff: existing lines are untouched.
- .claude/rules/buglog-status.md defines the status values (kept out of
openwolf.md, which `openwolf update` overwrites).
- scripts/buglog.py + buglog skill: add (next free id), status, note,
recur, list, show, triage. Edits only the touched entry's lines; a full
json.dumps rewrite churns ~1,200 lines of mixed \u-escaping.
- gitea-sync (script + skill): state and a single bug:<status> label come
from `status`; reopen supported; issues shared by several entries follow
the most-open one; only status labels are swapped; per-entry write-back
with gitea_synced_status so unchanged entries cost no API calls; list-typed
`fix` no longer crashes; spec issues close on done phases (spec.json has
no implementation approval); --since no longer floods Gitea after the
backfill.
- Disable OpenWolf buglog auto-detection: its hook numbers new entries
len(bugs)+1, which collides with existing ids (bug-1856, upstream).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chaotic round's walcheck compared all three members' on-disk WALs with
boundary=auto. The just-killed term-4 primary still held its NACKed,
never-committed lsn=58, which differed from the new term-5 leader's
own-term noop at lsn=58, and was counted as class=committed.
walcheck_count now takes an optional "midloop" mode, used only by the
per-round chaotic check: the boundary is the serving primary's own
committedLsn (live_committed_boundary() needs 3-of-3 convergence, which a
chaotic round never reaches), and only members guaranteed to hold the
committed prefix are compared -- the primary plus live standbys whose
appliedLsn has reached the boundary. Every other caller, including the
settled final sweep, still compares all three. Log lines name members=.
Also logs bug-1854 (scheduler re-adds every configured resource on each
rejoin config push; ERROR noise, changed definitions ignored) for
follow-up.
10-round chaotic soak: PASS (divergence 0, job loss 0, SIMVERIFY 9 files,
0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the live worker + scheduler pair is up, the soak now starts the
scheduler with --record-dir, records (xceadmin scheduler record start)
from each scheduler (re)join until just before the next primary kill,
stops it cleanly so the .replay is flushed, and at the end replays every
clean recording through `xce_scheduler --sim --sim-verify`. Any decision
MISMATCH, a failed sim run, or zero clean recordings FAILs the soak:
[xce-test-ha] SIMVERIFY files=N mismatches=M errors=E unclean=U
Recordings that could not be stopped cleanly are counted as unclean and
never verified (their unflushed tail would give false mismatches), except
a stop whose reply was lost mid-failover: when the same scheduler pid
answers "not recording" and logged no recording abort, the first stop
landed and the file is kept. Mismatches are counted from both the sim's
log file and stderr (the logger creates its file lazily).
XCE_HA_SCHED_REPLAY_VERIFY=0 disables the gate.
First runs caught bug-1851 and bug-1852.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Existing worker-rejoin-resync coverage (sequential/chaotic) only proves a
worker-dispatched job survives a real failover -- it never re-checks the
worker's own resource accounting afterward, which is exactly where
bug-1841 lived. This mode over-submits jobs to empirically discover the
worker's real concurrency ceiling (self-verifying via a PEND overflow
probe, not assumed from resval), fails over the primary while jobs are
RUN, waits for genuine rejoin + DONE, then hard-asserts the probe job
(still PEND from before the kill) now reaches RUN -- proving the
scheduler's resInUse actually unfroze.
Live-validated against a real 3-node cluster + xcecd_virtual worker:
settled at 3 RUN / 41 PEND (limits.yaml's limit1, not raw worker
capacity, was the actual ceiling -- exactly the kind of environment
detail this design avoids needing to assume), all 3 recovered to DONE via
genuine rejoin, probe reached RUN 219ms after recovery. Two earlier
design iterations are documented in the round's own comments: sequential
submit-then-wait let early jobs finish before late ones were even
submitted, and assuming resval's raw capacity as the concurrency ceiling
doesn't hold when queue/user limits bind first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A live 3-node HA cluster went into a repeated PreVote-failover storm
(term climbing, RDHUP loops, "scheduler not connected") even though
system load was low. Root cause: a hand-typed cleanup command had a
transposed-word typo (xce-ha-deploy vs. the real xce-deploy-ha path)
and silently killed nothing, leaving a full controller cluster alive.
A fresh deploy then bound a second, independent cluster to the exact
same ports, and the two clusters' vote traffic genuinely cross-talked
-- producing symptoms that looked like a real HA bug but weren't.
Two structural gaps let that one typo cascade this badly:
1. deploy-ha.sh had no pre-flight check that target ports were free --
it would happily start a second Raft group on ports an existing one
already held, instead of failing cleanly.
2. start.sh/start-all.sh already had a correct "already running" guard
(reads $TOP/run/*.pid, checks kill -0), but cleanup.sh deleted those
pidfiles unconditionally as part of state cleanup, with no check
that the process each one pointed to was actually dead first --
silently erasing the only signal that guard depends on.
Fixes, all live-verified end to end against real controller/worker/
scheduler/apiserver binaries:
- Added a port pre-flight check before any deploy state is created:
refuses with the exact blocking PID/process if a target port is
already listening.
- cleanup.sh now stops every live PID in $TOP/run/*.pid before
removing any pidfile or state.
- Added stop-all.sh (generated alongside start.sh/cleanup.sh): the
canonical, typo-proof way to stop everything a specific deploy
started, since it reads that deploy's own tracked PIDs instead of
matching process names/paths by hand.
- Found and fixed a related race while testing the above: start-all.sh
didn't wait for worker/scheduler/apiserver to write their pidfiles
before returning "done", so cleanup.sh/stop-all.sh run immediately
after could orphan those two specifically -- the same zombie-process
class this whole fix targets, via a narrower window instead of an
operator typo. Added a short pidfile-appearance poll to close it.
SKILL.md updated to prefer a known deploy's own stop-all.sh over a
hand-typed kill command, document cleanup.sh's new stop-then-wipe
behavior, and add two Troubleshooting rows matching this incident
directly.
bug-1816.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
nproc - load_avg treats one ninja job as one process, but the compiler
driver forks cc1/cc1plus then as, and asan/debug keep those children
resident far longer than release's leaner ones -- under-throttling by
roughly that factor. Apply a per-profile PROCESS_FACTOR (0.5 for
asan/debug, 0.7 for release) to the CPU term before taking the min with
the memory-budget term.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Follow-up to the xce-harness-component spec, which relocated xce_test
from src/controller/testsuite/ into a standalone src/harness/
component (commits cf55d691..cf380750). These four files were outside
that spec's own file boundary (owned by other specs / general tooling)
so were deliberately left alone at the time, but their hardcoded
default path and error-message advice were now stale and would have
misled anyone who hit them:
- .claude/skills/xce-build/SKILL.md: added `harness` to both component
lists (frontmatter description + args section), and replaced the
"controller always builds xce_test too" claim with an accurate
description of the harness component, its own build command, and
the accepted Meson test-runner quirk (bug-1806) that can still
transitively rebuild xce_test as a side effect of another
component's default-profile test run.
- src/controller/testsuite/test_rcfile_integration.sh (owned by the
xce-test-rc-file spec): XCE_TEST_BIN default and failure message.
- src/controller/testsuite/clean_failover_preservation.sh (owned by
ha-term-safety task 5.2): same.
- src/tools/test-commit-durability.sh: same, plus its header comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
run-resource-verify.sh's XT default now points at
\$BUILD_DIR/harness/xce_test and fails fast (actionable "run
/xce-build harness" message) if missing -- it drives
inject_resources()'s worker join/resupdate and
internal_controller_check()'s client-context resval, so a hard
failure is correct, matching the precedent from tasks 3.2/3.4.
SKILL.md's build-prerequisite content folded into the existing Usage
section (this skill has no standalone Prerequisites heading, unlike
its siblings) plus a Dependencies section update.
Verified live: sourced the script (main() guarded), called
precondition_checks() present then absent then restored
(checksum-verified intact).
This completes all four downstream-workflow tasks (3.2-3.5) --
xce-test-ha, xce-deploy-ha, xce-test-multihost, and xce-test-resources
all now handle the harness/controller build decoupling correctly.
Task 3.5 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
run-multihost-verify.sh's XT default now points at
\$BUILD_DIR/harness/xce_test and fails fast (actionable "run
/xce-build harness" message) if missing. xce_test is the sole
mechanism (xtest()) by which the script's job-scenario functions
talk to the cluster, so a hard failure is correct, matching task
3.2's already-approved precedent.
SKILL.md's Prerequisites, env-var docs, and Dependencies sections
updated to match.
Verified live: sourced the script (main() guarded from running),
called precondition_checks() with the binary present and then moved
aside, confirmed the exact failure message, restored the binary
(checksum-verified intact).
Task 3.4 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Extend SKILL.md's existing worker/scheduler/xceadmin soft-skip
documentation to cover harness/xce_test: a Prerequisites bullet and a
Troubleshooting row, matching the existing style exactly.
Confirmed documentation-only: deploy-ha.sh (653 lines, read in full)
has zero reference to xce_test/harness -- it only builds/wraps
worker/scheduler/apiserver/xceadmin, never touches xce_test, which is
a manually-invoked tool rather than something deploy-ha.sh generates
a wrapper for. No script change needed or made.
Task 3.3 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
run-ha-soak.sh's XT default now points at src/xce.build/harness/xce_test
(where /xce-build harness places it) and fails fast (exit 2, actionable
"run /xce-build harness" message) if missing, rather than assuming a
controller build produced it. xce_test is the soak's own driver client
(~30 call sites: job submit/verify, walcompare, waldump, snapdiff,
worker join/cancel), so a hard failure is correct, not a soft-skip.
SKILL.md's build instructions, goal-loop rebuild step, and
troubleshooting table updated to mention building harness, with
--incremental guidance correctly conditioned on build-dir-clobbering
semantics (bug-1565) rather than blanket-recommended.
Verified live: moved the binary aside, ran the script, confirmed the
exact failure message, restored the binary (checksum-verified intact).
Task 3.2 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Non-functional: a repo-wide audit of every xce_test consumer against the
xce-test-cli-ux spec's completed changes (confirmation prompts, joblist/
reslist format fixes, stream routing, join's maxmem default, qctrl/getconf
fixes) found zero functional breaks -- every script drives xce_test via
piped non-interactive stdin (Req 9.4 auto-skips confirmation prompts) and
either checks exit codes/structured markers or already redirects 2>&1.
Add a one-line comment at each of the three bare `join` (no -m) call
sites noting the default changed from a stale 64 bytes to the correct
64MB (bug-1557), for future-reader clarity -- harmless in all three since
none of these synthetic hosts are ever scheduled against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
xce-build.sh has self-wrapped into nix develop since 028d8f6a; this
skill's two example command blocks still showed the old manual
"nix develop ./src --command bash ..." wrapper, now stale (harmless
since a nested nix develop is a no-op, but misleading documentation).
Simplified both to invoke the script directly, matching README.md's
same cleanup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Add --jobs N to xce-build.sh: for meson compile it's passed as ninja's -l N
load-average ceiling (re-checked before every job launch for the whole
build, so it adapts if a concurrent session in this shared sandbox loads
the system mid-build), and as a job-count ceiling (-j/--num-processes,
-p) for meson test/go test, which have no load-average equivalent. No
default -- omitting it preserves prior auto-parallelism behavior exactly.
The xce-build skill now computes N live at invocation time from nproc,
/proc/meminfo's MemAvailable, and /proc/loadavg, rather than a fixed
constant, and passes it through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DrxU1dSRYNMV9CDTXHFtcR
Extends xce-test-multihost with a second scenario: a 2-element array
job (jobsub -a 2) where each element independently requests the same
nproc+1 task count as the plain-job scenario. Drives element [0]
through the identical RUN-then-DONE, both-hosts-started proof, and
confirms its jobinfo Slots: field matches the plain job's exactly
(Requirement 5.5). Traced the array-expansion path
(handler.c's sched_on_job_array()/clone_job_submit_data()) to confirm
no array-specific shortcut bypasses apply_job_task_count() -- each
element flows through the same core_build_job() ingest as a
standalone job.
Fixes a real bug (bug-1775) found while building this: an array
element's bracketed jobid (e.g. "<base>[0]") was misparsed by
grep -E's bracket-expression syntax, causing a false negative in the
per-host verification. Added ere_escape(), a no-op for plain
(non-bracketed) jobids.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extends xce-test-multihost's harness to submit one real job via plain
jobsub, sized (nproc+1 tasks, queried live) so it must span both
mh-worker-1 and mh-worker-2, poll jobinfo for RUN then DONE, and
confirm each allocated host's own log independently shows it starting
its share of the job's tasks.
The two allocated hosts take genuinely different code paths: the
controller sends startjob only to the head-node host
(dispatch_a_job_(), job.c); that host's job_start_remote_task()
relays a runtasks request back through the controller
(do_runtasks()/run_tasks_on_host(), task.c) to the other host, which
independently calls submit_fork_task(). The harness's per-host
verification accepts either line as legitimate evidence for that
host, since which host lands as head node is an allocator-internal
detail this test correctly doesn't hardcode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
New xce-test-multihost skill deploys one controller/scheduler node
plus two named xcecd_virtual workers, following xce-test-resources's
single-node deploy pattern and xce-test-ha's pre-flight zombie-process/
port-check convention (chosen over a fresh-mktemp-per-run model
specifically because a meaningful pre-flight check needs a stable
directory -- see tasks.md's Implementation Notes). Confirms all four
roles join via their own log lines and tears down cleanly with a
HARNESS_READY sentinel. No job scenario wired in yet -- that's tasks
4.2/4.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
APISERVER_BIN's default changes from the fixed /tmp/apiserver path
to $BUILD_DIR/frontend/apiserver/xce_apiserver, matching the sibling
BINARY/SCHED_BIN/XT convention exactly (no dual-path fallback, per
research.md's explicit Design Decision). Failure message now names
/xce-build apiserver instead of the Makefile target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replaces the "always dispatch via the Agent tool" instruction with a
direct Bash-tool run_in_background pattern, following the harness's
own task-notification mechanism instead of trusting a subagent's
self-reported "I'll report back."
Confirmed broken 2026-09-03: a general-purpose subagent given the
exact "wait for it, do not background it" instruction redirected the
build to a log file, announced it would report back once finished,
and returned immediately with no result -- its own Agent-tool call
showed completed with nothing to relay. The harness did eventually
deliver a second, fuller notification for that same subagent several
minutes later, but a controller cannot rely on that arriving, per the
prior "fork claimed will report back, never reachable in a new
session" incident already logged in .wolf/cerebrum.md.
The new pattern (proven reliable twice this session: a full `all
debug` build and a `controller` ASAN rebuild) runs the resolved
command directly with Bash's own run_in_background, waits for the
harness's real completion notification, then reads only the log's
tail plus a targeted error grep -- keeping context usage low without
depending on a subagent's compliance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Starts a real worker (auto-detects xcecd_virtual vs. real xcecd, mirroring
xce-deploy-ha's own detection) + xce_scheduler alongside the controller
cluster, dispatches a genuine job, kills the primary while it's RUN, and
verifies the worker rejoins the new primary and the job recovers to DONE
via real reconciliation -- closing the "no real daemon-to-daemon smoke
test" gap flagged by worker-rejoin-resync's own feature validation.
Live-verified end-to-end multiple times (sequential and chaotic, 2 rounds
each) against a real 3-node cluster.
Also logs two bugs discovered along the way (not fixed, out of scope for
this test-infra change): bug-1759 (scheduler sim-mode dispatch/PEND
inconsistency) and bug-1760 (scheduler has no periodic post-failover
reconnect check, only a reactive one gated on its own next successful
publish -- a chicken-and-egg deadlock, live-confirmed via 60s+ of stuck
PEND with zero scheduler log activity). bug-1760 is worked around in this
harness only by restarting the scheduler process after each new-primary
election; the underlying gap is unfixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN
Generates/updates a rolling per-component change log from git history,
mirroring /weekly_report's date-window and continuity-check patterns but
organized by component (controller/worker/scheduler/etc.) instead of a
narrative weekly write-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the save/restore-specific gate added moments ago with a
broader, self-maintaining mechanism: rather than enumerating each
xce-audit-invariants dimension individually in the design-review
files (which would drift out of sync as new dimensions get added --
already up to 9 for controller, 4 for scheduler), both gates now
instruct walking the relevant component's whole dimension list against
the draft design. Catches design-time-checkable invariants (async I/O
thread-role placement, fail-loud/never-self-exit process lifecycle,
Raft/TLA+ safety checklist, commit-before-materialize ordering,
redirect-completeness for new opcodes, driver/sim state-equivalence)
before implementation rather than only via a later audit or live soak.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Now that xce_test auto-loads a personal rc file at startup (this spec),
harden the three existing non-interactive xce_test callers so a future
developer's ~/.config/xce_test/rc can never silently affect a CI/soak run:
run-ha-soak.sh, run-resource-verify.sh, and clean_failover_preservation.sh
all now pass --norc to their xtest() invocation. No behavior change today
(no rc file exists in this environment yet, per xce-test-rc-file task 3.3's
own verification) -- this is forward-looking reproducibility hardening.
Independently re-verified end-to-end: run-ha-soak.sh (--sequential, 2
rounds, zero job loss) and run-resource-verify.sh (1 resource, 3 layers,
zero mismatches) both PASS unchanged with --norc added.
clean_failover_preservation.sh could not be run in this sandbox -- its
DATADIR precondition (<repo>/ha/data/cluster.bin + certs) was deleted by an
unrelated earlier commit (ee8473bd, admin-config-tool) and never restored;
confirmed via git history this is pre-existing and unrelated to this edit
(the precondition check runs before xtest() and fails identically on the
unmodified script). Logged as bug-1721, not fixed here -- out of this
spec's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Open-ended dimension that walks .wolf/cerebrum.md's Decision Log section
fresh each audit (not a fixed list, since the log grows over sessions) --
seeded with two candidates: no-timeout-as-hang-fix, and commit-first-
then-reply ordering distinct from dimension 4's materialization-timing
check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
persistence.md's APPLY_CLEAN verdict only checks entity count and apply-
cursor position, not per-field completeness -- bug-1678 (job allocation
silently dropped on restart replay despite APPLY_CLEAN reporting healthy)
fell through exactly this gap. Documents the sharper invariant online-
apply and replay-apply must produce field-complete equivalent state, not
just matching counts/cursor, and adds it as a standing audit dimension
(#7) to xce-audit-invariants' controller dispatch list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
New scheduler-sim-eager-reconciliation requirements spec; weekly_report skill;
weekly status report for Aug 16-22. Also records cross-spec dependency notes
on scheduler-allocation and scheduler-preemption reflecting how
scheduler-dispatch-commit's redefined RUN-state timing affects their designs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
- .claude/settings.json: worktree.baseRef = "head"
- .kiro/specs/scheduler-core/.solomd: session tracking state
- .wolf/{anatomy,cerebrum,memory,token-ledger}.json, hooks/_session.json:
accumulated OpenWolf bookkeeping (includes this session's fmt_jobid work)
- .gitignore: ignore Python __pycache__/*.pyc (scripts/gitea-sync.py) and
.agent-shell/ (session transcripts can capture raw env dumps/secrets --
one already had a live-looking ANTHROPIC_API_KEY in a captured `env`
output; excluded from this commit entirely, never checked in)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds auto-detection of real vs virtual worker based on systemd
availability (/run/systemd/system), plus an XCE_REAL_WORKER override.
Real xcecd requires root (systemd D-Bus transient units, hardcoded
/var/run paths) and is used opportunistically now that passwordless sudo
is available in the sandbox; defaults to xcecd_virtual when systemd/D-Bus
isn't available.
verdict-logic-selftest.sh sources the real aggregate_verdict()/
build_verdict_record() (via run-resource-verify.sh's BASH_SOURCE guard, no
live deployment triggered) and exercises them with synthetic per-layer
records: all-match/PASS positive control, a single-resource single-layer
MISMATCH within a multi-resource batch (asserting both correct attribution
of the failure AND that the other resources independently still report
PASS), UNREACHABLE distinct from MISMATCH, REJECTED, and a zero-records
degenerate case. 53 assertions total.
Proven non-tautological: breaking aggregate_verdict()'s comparison logic
(two different ways, independently reproduced by review) makes the selftest
genuinely fail; restoring makes it pass again.
This completes the resource-value-verification spec - all 15 tasks (1.1
through 8.2) implemented, reviewed, and committed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aggregate_verdict()/build_verdict_record() combine the three per-layer
check results into a single per-resource comparison, self-contained and
invokable with synthetic records (task 8.2). Uses ASCII Unit Separator
(\x1f) as the record delimiter after discovering bash's IFS tab-splitting
silently collapses adjacent empty fields even with IFS set to only a tab
(bug-1581), which corrupted UNREACHABLE-record field alignment.
Emits one machine-parseable VERDICT line (resources/layers_checked/
mismatches/unreachable/rejected/result) plus the terminal PASS/FAIL oracle
line and matching exit code. Cleanup is the same trap-based mechanism task
5.1 registered for the abort path - logs always preserved regardless of
--keep-running.
Round-2 fix: main()'s live-data wiring (build_verdict_records_from_tsv())
derived a layer's status from array-key existence rather than the actual
recorded outcome - since tasks 6.3/6.4 always set the key (even on a
genuine timeout, just with an empty value), every real UNREACHABLE was
silently misclassified as MISMATCH. Fixed to check SCHEDULER_CHECK_RESULTS/
APISERVER_CHECK_RESULTS string content instead (bug-1582). Live-confirmed
by killing the scheduler mid-run: VERDICT now correctly reports
unreachable=9 mismatches=0, not the reverse.
SKILL.md rewritten to describe the now-complete, functional tool.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_apiserver_log()/query_apiserver_resvals() mirror task 6.3's
scheduler-check pattern: bounded-wait polling (batched across all pending
resources per iteration) of apiserver.log for task 4.2's "resource-location
stored" trace line, parsing the trailing zap JSON block. Reports MATCH /
MISMATCH (both values) / UNREACHABLE (timeout, distinct from mismatch),
tracked in APISERVER_OBSERVED/APISERVER_CHECK_RESULTS +
apiserver-check-results.tsv.
This completes all three per-layer checks. Live-confirmed: the full default
resource set (including cpuf, post-bug-1579-fix) MATCHes simultaneously at
the controller, scheduler, and apiserver layers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously worked around (task 6.2) by removing the one host-attrs resource
(cpuf) from the default set - a custom resource=value arg using any
host-attrs name would still false-mismatch at the controller layer, since
inject_resources() disconnected the worker before the separate controller-
check session ran, and the controller resets host-attrs resources on
disconnect.
Real fix: xce_test's `!<shell-command>` escape is handled in the single
top-level REPL loop before context-specific dispatch, so it works in any
context - including worker mode - without calling leave_ctx()/disconnecting.
inject_resources() now shells out via `!` to run the controller check
in-line, between resupdate and quit, while the worker's connection is still
open. query_controller_resvals()'s comparison logic is unchanged - this is
a pure sequencing fix. cpuf restored to DEFAULT_RESOURCE_ARGS as live proof.
Live-verified: cpuf now MATCHes at the controller layer (was always a false
MISMATCH before); adversarial shell-injection attempt via a malicious
resource name confirmed safe (printf %q quoting); SIGTERM delivered exactly
during the nested check confirmed zero orphaned/zombie processes across 4
independent runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_scheduler_log()/query_scheduler_resvals() tail the live scheduler
daemon's log (task 4.1's trace line) for each accepted injected resource,
with a bounded ~15s wait batched across all pending resources per poll
iteration (not N times the timeout). Reports MATCH / MISMATCH (both
values) / UNREACHABLE (timeout, distinct from mismatch), tracked in
SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS + scheduler-check-results.tsv.
Live-proven immune to bug-1579 (unlike the controller check): injecting a
host-attrs resource (cpuf) simultaneously produces a controller MISMATCH
and a scheduler MATCH, since this check reads task 4.1's trace log (written
once, at apply-time) rather than re-querying live state after the worker
disconnects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
query_controller_resvals()/controller_values_match() query the controller
via resval for every accepted (not rejected) injected resource/worker pair
and report MATCH/MISMATCH with both values on a mismatch, tracked in
CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS + controller-check-results.tsv
for tasks 6.3/6.4/7.1.
Also fixes bug-1579 (round-2, in default set only): cpuf, a host-attrs
resource the controller resets to baseline on worker disconnect, always
false-mismatched since inject_resources() disconnects before this check
runs. Swapped for r1m (periodic/dynamic, disconnect-safe) in
DEFAULT_RESOURCE_ARGS. Custom host-attrs resource=value args still hit this
- documented as a known follow-up, not fixed here (would need a C-level
xce_test change or a coprocess restructuring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
inject_resources() joins a synthetic worker through xce_test's worker
context and runs resupdate for the requested resource=value pairs
(default: Requirement 1's numeric gap set + mem/cpuf regression names,
deliberately excluding type/model due to bug-1574's live segfault risk).
Per-name accept/reject results captured into INJECTED_VALUES/
INJECTED_RESULTS plus injection-results.tsv for tasks 6.2-6.4.
Also suppresses the real xcecd_virtual worker daemon that deploy-ha.sh's
pre-existing worker branch would otherwise silently start whenever the
build dir contains one (XCE_WORKER_BIN=/bin/true), which would have
violated Requirement 2.1 - bug-1577.
Blocked pending a separate prerequisite fix (bug-1578, commit b406dd7):
live testing surfaced a pre-existing controller crash on worker disconnect,
unrelated to this feature, now fixed with regression coverage. Live-
reconfirmed post-fix: controller survives the full join+resupdate+quit flow
indefinitely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug
logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh
scratch dir, confirms all three are genuinely up, and cleans up via a trap
registered before any process starts - covering both normal completion and
a mid-deployment abort. Logs always preserved regardless of --keep-running.
Independently stress-tested with 8+ SIGTERM interrupts at varying
deployment points and a genuine crashing-apiserver scenario - zero leftover
processes in every case. Single-node by default (propagation testing, not
HA failover), overridable via $XCE_RESVERIFY_PEERS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.
Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
