run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug
logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh
scratch dir, confirms all three are genuinely up, and cleans up via a trap
registered before any process starts - covering both normal completion and
a mid-deployment abort. Logs always preserved regardless of --keep-running.
Independently stress-tested with 8+ SIGTERM interrupts at varying
deployment points and a genuine crashing-apiserver scenario - zero leftover
processes in every case. Single-node by default (propagation testing, not
HA failover), overridable via $XCE_RESVERIFY_PEERS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One logger.Debug("resource-location stored", ...) call in
OnResourceLocationUpdated, immediately after store.ResourceLocations.Put,
logging resource/host/value as zap fields. Debug-level only, matches this
package's existing logging convention (reuses convert.go's field names).
Live-confirmed against a real deployment with logging.level: debug.
Also surfaced (but did not fix, out of boundary - pure Go change, no
possible causal link) a pre-existing xce_test segfault on string-typed
resupdate (type/model) - uninitialized stack memory, non-deterministic,
invisible to ASan. Logged as bug-1574; flagged in tasks.md as a risk for
task 7.1's default resource set and task 8.1's live e2e run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One TRACE (tlog_debug3) line in xce_core_push_events()'s RESVALS_DELTA case,
logging host/resource/value in a fixed format ("resval applied host=...
resource=... value=..."), immediately after the value is written to the
worker's live state. Debug-level only, no control-flow change.
Explicit svalue/fvalue branching (round-2 fix after review) - never reads a
Value_u_t union member without confirming .which matches first, since
Value_u_bvalue (boolean resources) is reachable at this path in production
and would otherwise be misread as a garbage float.
Live-confirmed against a real deployment at -vvv. Also surfaced (but did not
fix, out of boundary) a pre-existing double-free crash on a resource's
second periodic report - logged as bug-1572.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.
Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cmd_resupdate() now issues a follow-up xcecli_resval() query per submitted
(resource, worker) pair after the update RPC, since update_resval_list()'s
reply can't carry per-item status. Reports accepted/rejected per name, with
a bounded retry (5x100ms) against propagation races.
Live-confirmed: `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports
mem accepted and not-a-real-resource rejected in the same invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_resval() now delegates type resolution to resval_typeresolve.c; unknown
resource names default to numeric, overridable via a new -t <numeric|string>
flag. cmd_resupdate()'s getopt loop defers per-pair finalization so a
trailing -t still applies to its pair, while preserving the exact original
duplicate-value rejection behavior (round-2 fix after review caught a
last-write-wins regression in that refactor).
Live-confirmed: resupdate -n r15m -v 1.5 (previously silently dropped) now
succeeds and is visible via resval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documentation-only stub: usage, VERDICT line format, and PASS/FAIL/cleanup
contract per design.md, explicitly marked not-yet-functional. The actual
run-resource-verify.sh orchestration script is built incrementally by
tasks 3.1-7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
No job could ever be scheduled in normal mode -- every round produced
decision=(nil), forever. Three independent bugs were stacked on the
same symptom, each masking the next (only reachable once the prior
one was fixed):
1. scheduler.c's sched_init_queues() -- the scheduler's one-time
synchronous initial-config-push handler -- only fired the legacy
job_elect_fn-era plugin_queue_update() fan-out. builtin_fcfs (the
default picker) has no .hooks, only picker_iface, so
on_queue_update() was never called for any queue from the initial
cluster config. Its own candidate registry stayed empty forever,
so next_job() found zero candidates on every session.
2. Once #1 was fixed and a job first reached the allocator,
core_events.c's core_build_job() never constructed job->schedobj
-- a documented but incorrect assumption from the scheduler-core
rewrite ("no picker hook reads job->schedobj... belongs to a
separate allocation spec" -- no such spec exists in the live
code). sched_allocator() dereferences it directly with no NULL
check; reproduced live as a SIGSEGV in limit_job_election().
3. Once #2 was fixed and a job reached genuine dispatch,
resource.c's put_resvals() crashed during post-dispatch resource
charging: `idxs[i] <= 0` treated resource index 0 ("slots",
almost always the first registered resource) as an invalid
sentinel, leaving its value slot NULL for an unconditional write
two lines later.
All three root-caused via gdb `bt full` on live SIGSEGVs, not
guesswork, and live-verified end-to-end: a 3-element job array now
runs PEND -> DISPATCH -> RUN -> DONE with the scheduler staying alive
throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
collect_load_indices() (resource_sync.c) tagged every periodic
mem/r1m/r15m/r15s/ut/pg/ls/it/swp/io/tmp report with gethostname()'s
raw OS/container hostname instead of the worker's own join identity
(xcecd_ctx.virtual_name, the -n/--name value). Every other worker-keyed
lookup downstream (scheduler's sched_get_worker(), and the same
HOOK_DETAIL_RESOURCE_UPDATE payload forwarded on to the Go apiserver)
matches by that join identity, so the mismatch silently dropped every
periodic resource report for the real worker.
Also adds a startup INFO log (main.c) naming the resolved job picker
plugin, used to rule out a legacy-picker hypothesis while chasing a
related (still-open) dispatch issue.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xce_scheduler -i -s <replay>: "info worker <name>" showed
total=0.00 available=0.00 for every resource, even though the live
cluster (xce_test resval/workerlist) reported real non-zero values
for the same worker.
Two independent root causes, both live-traced end-to-end against a
real deployed cluster (value-level TRACE at every hop: controller
push, scheduler apply, snapshot save, snapshot load/decode):
1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH
called sched_add_worker() -- a plain key_add(), add-only --
unconditionally on every push. A worker connecting triggers
multiple WORKERS_PUSH events in quick succession (an early push
before the controller finishes writing real capacity values,
carrying stale/zero data, followed moments later by a push
carrying the real values). The FIRST push always won the race;
every push after it for the same worker name silently failed
-EEXIST and its resource values were discarded -- not just at
join, for the rest of the process's lifetime (unlike jobs, there
was no "update an existing worker" event path at all). Fixed via
a new core_upsert_worker() (mirrors the existing
core_upsert_queue() add-or-update precedent): looks up the
worker by name first, updates status+resvals in place if found,
else builds+adds a new one as before.
2. src/scheduler/src/repl_info.c: repl_info_worker_one()/
repl_info_resource_one() checked `xce_val_t.which ==
XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1)
is a resource-DEFINITION value-type constant
(xce_resource_t.valtype's own enum), a completely different enum
from xce_val_t.which's real domain, Value_u_which
(controller.capnp.h, Value_u_fvalue=5). Every producer in this
codebase sets xce_val_t.which to Value_u_fvalue, never
XCE_VAL_NUMERIC, so the comparison was always false -- every
genuinely-numeric resval displayed as 0.00 unconditionally,
independent of fix (1). Corrected all 4 occurrences to
Value_u_fvalue.
Both fixed via TDD (RED test before each fix, GREEN after):
test_core_events.c's test_workers_push_twice_updates_existing_
worker_in_place (two pushes, same name, different values -- both
must apply) and test_repl_breakpoints_inspection.c's
test_info_worker_reports_resource_values (a real numeric resval
must display, not fall back to 0.00).
Live-validated by reproducing the user's exact original scenario
end-to-end (deploy a real cluster, record, `xce_scheduler -i -s
<replay>`, `info worker <name>`) -- now shows slots=22.00,
maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching
the user's own live xce_test resval/workerlist numbers exactly.
Full scheduler suite green (41/41, excluding pre-existing unrelated
bug-1560); controller suite green (1102/1104, the same pre-existing
bug-1561 pair, confirmed unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xce_scheduler -i -s <replay>: WARN "failed to resolve job picker
plugin (rc=-2)" on load, and every job already present in the
replay file's INITIAL snapshot stayed PEND forever -- never even
reached the picker, let alone got dispatched.
Root cause: sched_sim_driver_start() (sim.c) loads the initial
snapshot via xce_sched_core_load_snapshot() BEFORE any picker
plugin is registered (sim.h's own documented "Known scope
boundary" -- only the picker's NAME is threaded through from the
ReplayHeader). sched_resolve_job_picker() therefore always returns
-ENOENT there, and xce_sched_core_load_snapshot()'s own
picker-rebuild walk (on_queue_update/on_job_push per queue/job) is
silently skipped. The real builtin_fcfs picker only gets registered
moments later in sched_sim_driver_run() -- but nothing then replayed
that rebuild walk, so any job/queue from the INITIAL snapshot (as
opposed to one arriving later via a ReplayEvent, which correctly
reaches the picker through xce_core_push_events()'s own picker!=NULL
check) was permanently invisible to next_job().
Fixed by a new file, sim_picker.c: sim_bootstrap_picker() (replacing
sim_loop.c's sim_register_default_picker(), moved out because
sim_loop.c was already at its own documented 800-line "zero
line-budget headroom" cap) registers the picker AND then replays the
missed rebuild walk against it. Wired into scheduler_srcs,
sched_sim_isolation_lib, and all 9 white-box test TUs that #include
sim_loop.c directly.
New regression test (test_sim_event_loop.c): a job baked into a
fixture's INITIAL snapshot (not a ReplayEvent) now gets a live PEND
decision. RED confirmed by temporarily no-op'ing the rebuild walk
(test failed as expected), GREEN with the real fix. Full scheduler
suite green (41/41, excluding the pre-existing unrelated bug-1560).
Note: sim mode still never DISPATCHes (only PENDs) -- that is a
separate, intentionally out-of-scope gap (real resource-matching
allocation belongs to the not-yet-designed scheduler-allocation
spec), not addressed here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three related enhancements to make /xce-deploy-ha's output a
self-contained, directly-usable cluster for manual testing:
- xceadmin now builds an installable wheel (meson-build.sh) with
correct dependencies (click/pycapnp/PyYAML/jsonschema, verified
against actual imports -- pyproject.toml was missing several).
- deploy-ha.sh resolves the deploy environment's own bash
($BASH_BIN, baked into every generated script's shebang -- this
environment's deploy-ha.sh had drifted to #!/bin/sh while still
using bash-only syntax) and generates scripts/start-worker.sh /
scripts/start-scheduler.sh sharing the controllers' --peers, plus
a standalone xceadmin install (wheel extracted directly, resolved
$PYTHONPATH baked into a wrapper script -- nix devShell packages
are $PYTHONPATH entries, not merged into python3's own
site-packages, so a --system-site-packages venv can't see them).
- bug-1566: xceadmin's --controller option had no relationship to a
deployment's actual peers (hardcoded localhost:9991 default) --
every command required an explicit --controller or silently tried
the wrong port. Fixed via a $XCEADMIN_CONTROLLER envvar fallback
(click precedence: explicit flag > envvar > bare default),
exported by deploy-ha.sh's generated scripts/xceadmin wrapper.
Fixed/validated via TDD + live reproduction against a real deployed
cluster in a fully stripped environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setup_build() unconditionally rm -rf'd the whole profile build
directory on every single-component invocation -- running
/xce-build xceadmin after /xce-build controller silently deleted
xce_controller/xce_test/etc. from the same build dir.
Added an opt-in --incremental flag (meson setup --reconfigure in
place instead of wiping) rather than changing the default: this
project's capnp-generation custom_target has known missing-
dependency-edge races that a full wipe currently papers over as a
loud, retry-fixable failure. Making incremental the default risked
turning that same gap into silent stale-generated-code instead, so
the safer full-wipe behavior is preserved unless explicitly opted
into.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
persist_flush()'s synchronous fdatasync path advanced durable_lsn on
success but never called on_durable_advance() -- only the async
fsync_worker did. This gap was previously masked by bug-1563's own
busy-spin (a fast accidental repoll covered for the missing
notification); fixing that spin unmasked it, causing
test_durable_majority_commit_end_to_end to fail deterministically
(3/3 runs).
Fixed by firing on_durable_advance() from persist_flush() too,
whenever it genuinely advances durable_lsn. Fixed via TDD (RED test
before the fix, GREEN after) and re-validated with a fresh 40-round
sequential + chaotic HA soak (both pass cleanly, zero fdatasync
errors, zero divergence/loss).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.
bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.
Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ha-config-bootstrap, scheduler-controller-msg-gap, and resval-id-decoupling
were the only 3 specs (of ~40) carrying a nonstandard status:"active"
field, each paired with a note about residual gap-audit findings that
never got refreshed once later work actually resolved them:
- ha-config-bootstrap: all 3 known deviations resolved/superseded
(redirect fields -> libxce-failover; WAL majority-commit scope -> the
deliberate wal_classify() WAIT/RETAIN design; commit-timeout durability
-> bug-661 fix via ha-commit-durability, live-reverified 2026-07-18).
One narrower, genuinely new gap found in passing (OP_upgradeConfig
replies lack redirect wiring, admin-config-tool task 6.3) is flagged
in the note for a future spec, not folded back into this one.
- scheduler-controller-msg-gap: both open items (sched_pend_update_killer
leak, test coverage) were fixed via bug-765 across 4 commits on
2026-06-19, verified directly in current scheduler.c and the test
suite -- the note just never caught up.
- resval-id-decoupling: not just an investigation -- the recommended
name-keyed collect_resval() fix shipped one day after the spec was
created (commit d47b3375), verified as the sole implementation left
in current code. Description no longer named the deleted function;
status:"active" flipped to "closed" and a note added to match its
sibling specs' convention.
All three verified via direct code/commit/buglog inspection (not just
trusting stale spec text) before closing. status:"active" -> "closed"
in all three; phase stays "implemented" (already correct).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
