xce_scheduler -i -s <replay>: WARN "failed to resolve job picker
plugin (rc=-2)" on load, and every job already present in the
replay file's INITIAL snapshot stayed PEND forever -- never even
reached the picker, let alone got dispatched.
Root cause: sched_sim_driver_start() (sim.c) loads the initial
snapshot via xce_sched_core_load_snapshot() BEFORE any picker
plugin is registered (sim.h's own documented "Known scope
boundary" -- only the picker's NAME is threaded through from the
ReplayHeader). sched_resolve_job_picker() therefore always returns
-ENOENT there, and xce_sched_core_load_snapshot()'s own
picker-rebuild walk (on_queue_update/on_job_push per queue/job) is
silently skipped. The real builtin_fcfs picker only gets registered
moments later in sched_sim_driver_run() -- but nothing then replayed
that rebuild walk, so any job/queue from the INITIAL snapshot (as
opposed to one arriving later via a ReplayEvent, which correctly
reaches the picker through xce_core_push_events()'s own picker!=NULL
check) was permanently invisible to next_job().
Fixed by a new file, sim_picker.c: sim_bootstrap_picker() (replacing
sim_loop.c's sim_register_default_picker(), moved out because
sim_loop.c was already at its own documented 800-line "zero
line-budget headroom" cap) registers the picker AND then replays the
missed rebuild walk against it. Wired into scheduler_srcs,
sched_sim_isolation_lib, and all 9 white-box test TUs that #include
sim_loop.c directly.
New regression test (test_sim_event_loop.c): a job baked into a
fixture's INITIAL snapshot (not a ReplayEvent) now gets a live PEND
decision. RED confirmed by temporarily no-op'ing the rebuild walk
(test failed as expected), GREEN with the real fix. Full scheduler
suite green (41/41, excluding the pre-existing unrelated bug-1560).
Note: sim mode still never DISPATCHes (only PENDs) -- that is a
separate, intentionally out-of-scope gap (real resource-matching
allocation belongs to the not-yet-designed scheduler-allocation
spec), not addressed here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three related enhancements to make /xce-deploy-ha's output a
self-contained, directly-usable cluster for manual testing:
- xceadmin now builds an installable wheel (meson-build.sh) with
correct dependencies (click/pycapnp/PyYAML/jsonschema, verified
against actual imports -- pyproject.toml was missing several).
- deploy-ha.sh resolves the deploy environment's own bash
($BASH_BIN, baked into every generated script's shebang -- this
environment's deploy-ha.sh had drifted to #!/bin/sh while still
using bash-only syntax) and generates scripts/start-worker.sh /
scripts/start-scheduler.sh sharing the controllers' --peers, plus
a standalone xceadmin install (wheel extracted directly, resolved
$PYTHONPATH baked into a wrapper script -- nix devShell packages
are $PYTHONPATH entries, not merged into python3's own
site-packages, so a --system-site-packages venv can't see them).
- bug-1566: xceadmin's --controller option had no relationship to a
deployment's actual peers (hardcoded localhost:9991 default) --
every command required an explicit --controller or silently tried
the wrong port. Fixed via a $XCEADMIN_CONTROLLER envvar fallback
(click precedence: explicit flag > envvar > bare default),
exported by deploy-ha.sh's generated scripts/xceadmin wrapper.
Fixed/validated via TDD + live reproduction against a real deployed
cluster in a fully stripped environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setup_build() unconditionally rm -rf'd the whole profile build
directory on every single-component invocation -- running
/xce-build xceadmin after /xce-build controller silently deleted
xce_controller/xce_test/etc. from the same build dir.
Added an opt-in --incremental flag (meson setup --reconfigure in
place instead of wiping) rather than changing the default: this
project's capnp-generation custom_target has known missing-
dependency-edge races that a full wipe currently papers over as a
loud, retry-fixable failure. Making incremental the default risked
turning that same gap into silent stale-generated-code instead, so
the safer full-wipe behavior is preserved unless explicitly opted
into.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
persist_flush()'s synchronous fdatasync path advanced durable_lsn on
success but never called on_durable_advance() -- only the async
fsync_worker did. This gap was previously masked by bug-1563's own
busy-spin (a fast accidental repoll covered for the missing
notification); fixing that spin unmasked it, causing
test_durable_majority_commit_end_to_end to fail deterministically
(3/3 runs).
Fixed by firing on_durable_advance() from persist_flush() too,
whenever it genuinely advances durable_lsn. Fixed via TDD (RED test
before the fix, GREEN after) and re-validated with a fresh 40-round
sequential + chaotic HA soak (both pass cleanly, zero fdatasync
errors, zero divergence/loss).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.
bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.
Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ha-config-bootstrap, scheduler-controller-msg-gap, and resval-id-decoupling
were the only 3 specs (of ~40) carrying a nonstandard status:"active"
field, each paired with a note about residual gap-audit findings that
never got refreshed once later work actually resolved them:
- ha-config-bootstrap: all 3 known deviations resolved/superseded
(redirect fields -> libxce-failover; WAL majority-commit scope -> the
deliberate wal_classify() WAIT/RETAIN design; commit-timeout durability
-> bug-661 fix via ha-commit-durability, live-reverified 2026-07-18).
One narrower, genuinely new gap found in passing (OP_upgradeConfig
replies lack redirect wiring, admin-config-tool task 6.3) is flagged
in the note for a future spec, not folded back into this one.
- scheduler-controller-msg-gap: both open items (sched_pend_update_killer
leak, test coverage) were fixed via bug-765 across 4 commits on
2026-06-19, verified directly in current scheduler.c and the test
suite -- the note just never caught up.
- resval-id-decoupling: not just an investigation -- the recommended
name-keyed collect_resval() fix shipped one day after the spec was
created (commit d47b3375), verified as the sole implementation left
in current code. Description no longer named the deleted function;
status:"active" flipped to "closed" and a note added to match its
sibling specs' convention.
All three verified via direct code/commit/buglog inspection (not just
trusting stale spec text) before closing. status:"active" -> "closed"
in all three; phase stays "implemented" (already correct).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 6.3's own "before deciding" gate (re-run the flaky primary-push
E2E 20-25x standalone with real stderr capture) was already satisfied
by bug-1485's 2026-08-01 follow-up: 40/40 clean runs, zero captured
crash evidence, after replacing the DEVNULL stderr redirect with real
per-node log capture. Decision reached there (mergeable as-is, bug-1485
downgraded to low-priority/environment-flake) was never reflected back
into tasks.md/spec.json. Marks 6.3 and its parent task-group 6 [x], and
reconciles spec.json's stale "tasks-generated" phase (21/21 tasks were
already complete) to "implementation-complete" with a validate_impl
block recording the known limitations (bug-1485 stays open/low-priority,
single-node ackedCount==0 exit-1 behavior, no CI live-E2E run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xcecli_jobsubmit() could return rc>=0 (success) while leaving its *jobid
output parameter at 0, reproducible under concurrent client load
immediately after a fresh single-node election (~1-3/9 clients per round
in the original repro).
Root cause: XCECLI_FAILOVER_CALL's idempotent-replay rule
(client_failover.c's xce_failover_classify) normalizes a retried resend
that hits the server's EEXIST/inflight-collision rejection into overall
success -- but that rejection reply carries no payload
(handle_jobsubmit's EEXIST branches reply with RequestReply_data_nodata,
job.c), so xce_jobsubmit() never populates *jobid on the winning
attempt, nor did it on the earlier failing attempt (raw status was
already negative).
Adds jobsubmit_finalize_jobid() (client_internal.h/client.c), called
from xcecli_jobsubmit() right after XCECLI_FAILOVER_CALL returns. Sets
*jobid = submit->jobid (the client-generated id already on the wire --
the controller commits/echoes this exact value on every genuine success
reply, bug-786/job.c:1439) whenever the overall call succeeded,
otherwise leaves it untouched. Placed at the jobsubmit-specific wrapper
layer, not the generic classify/decide layer, which stays fully generic.
3 new socket-free unit tests in test_libxce_retry.c (following this
codebase's direct-state-simulation style for I/O-adjacent logic).
Live-validated: full ASAN rebuild clean, libxce test suite 1/1 OK with
all 3 new tests passing, and a 5-round/45-client concurrent-submit
repro against a live single-node cluster showed 0 "no clean ack"
failures (vs. 1-3/9 per round pre-fix).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds `xceadmin config dump CLUSTER_BIN_PATH`, the reverse of `config
build` -- decodes a serialized ClusterConfig back to human-readable text
so a built config can be inspected without a throwaway pycapnp script.
Motivated directly by how bug-1556 (hosts.yaml's max_slot: -1 auto-detect
sentinel silently becoming mxj=0) was actually found: by hand-decoding a
live cluster.bin. The dump renders the same sentinel as
"max_slot=-1 (auto-detect from CPU count)", not the raw uint32.
Also adds parse_size_or_number() to xcecommon.mapping so a queue/host's
sched_stop threshold on a byte-valued resource (maxmem/mem/tmp/maxswap/
maxtmp) can be written as a human-friendly size string ("500MB", "2GB")
instead of a raw byte count; a plain number still passes through
unchanged for non-byte resources like load averages. hosts.schema.json
and queues.schema.json's sched_stop schema gains a matching SizeOrNumber
type. dump_cluster_config() humanizes the same byte-valued thresholds
back on the way out (_format_bytes_human).
New ClusterBinFileError covers every local cluster.bin read/decode
failure mode without letting a raw OSError/capnp.KjException escape the
CLI. RESOURCE_VALTYPE/RESOURCE_ORDTYPE/RESOURCE_FLAG_* made public in
mapping.py so dump_cluster_config can reuse the same int<->name tables
build_cluster_config already uses, keeping both directions in sync by
construction.
256 xceadmin tests pass (incl. new test_config_dump.py and the
parse_size_or_number/SizeOrNumber cases in test_mapping.py/test_cli.py),
verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install
+ pytest run before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Recovered .wolf/buglog.json (828 entries lost when this worktree's copy
silently reset to bug-001 sometime after commit ce96fd6d, 3 days
uncommitted with zero overlap detection) by merging the committed
history back in and renumbering the 43 real entries logged since.
To stop this recurring:
- OPENWOLF.md: require searching for an existing same-file/root_cause
entry before appending a new bug id (fixes the bug-1418 x4 duplication
pattern found in the recovered history); never restart the id counter.
- stop.js: validate buglog.json every turn (JSON validity, {version,bugs}
wrapper, duplicate ids, count/max-id not regressing vs. the last
commit) and auto-commit it standalone when it grew cleanly, so git
history is always a recoverable backup instead of a 3-day-stale one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Final regression sweep (task 5.4) confirms zero new regressions across
the full stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli)
and proves the single-candidate path is identical to the multi-candidate
path across all three components (Req 1.5), closing out all 21 tasks
(1.1-5.4) of component-failover-adoption.
Two production bugs found during live E2E validation (5.1/5.2) remain
open and out of this spec's scope: bug-039 (apiserver idle-period rejoin
structurally unobservable on a real single-election failover) and
bug-040 (job pushes not reaching apiserver's local store, likely
downstream of bug-039).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live E2E of apiserver's backend-health circuit breaker (task 4.5):
killed all 3 controller nodes (not just the primary, unlike 5.1/5.2 --
this scenario needs a call to genuinely exhaust its retry budget, not
successfully rotate to a survivor). A write call burns through the
20-attempt budget in ~3ms (ECONNREFUSED classifies as zero-backoff
XCE_FO_RETRY_NOW, not the ~70-100s backoff track reserved for a
reachable-but-electing controller), returning the real -E_XCE_NOT_READY
(-10013) and triggering MarkDegraded(10s). Immediately after, a second
write call fast-fails in ~1ms via the literal ErrBackendUnavailable
string -- proving the real C call was skipped entirely, not just fast
-- while a concurrent GET /ping (confirmed by source read to never
touch the backend) keeps succeeding normally throughout.
Cross-corroborated across two independently-generated log subsystems
(apiserver's Go zap log and libxce's own C-level retry log) matching
to the millisecond. Bonus: confirmed the 10s TTL's half-open auto-clear
correctly re-attempts the real call afterward.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Heartbeat-only idle-period E2E (apiserver alone, zero write traffic
during a 268s+ post-kill observation window) designed as a targeted,
cleaner confirmation experiment for task 5.1's bug-039 finding.
Result: bug-039 reproduces a second time, more cleanly than task 5.1
-- zero rejoin log lines throughout the observation window, and even
a post-window diagnostic write call (which itself succeeds against
the confirmed-new primary) triggers zero rejoin activity, ruling out
"task 5.1 was just unlucky timing." Controller-side logs corroborate
independently: zero "apiserver" references anywhere in the new
primary's log, meaning no join attempt of any kind ever reached it.
This task's own literal pass bar was not met -- approved on the same
basis as task 5.1: the validation itself succeeded rigorously and
honestly surfaced a real, now doubly-confirmed gap in the already-
shipped task 4.3/4.4 rejoin mechanism, which is the task's actual
purpose. A dedicated follow-up to fix heartbeat.go's transition
detection is warranted, separate from this spec's remaining scope.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
