Terminal jobs (DONE/EXIT/PERR) were only filtered from checkpoints and
WAL replay (job_should_retain, finished_retain_s=3600) — the live jobmgr
never dropped them, so joblist and the scheduler's core->jobs view grew
without bound across a controller's uptime (N4).
Three pieces:
1. kvmgr delete path (src/lib/kvmgr.c): xkvmgr_type had .del_key=NULL
and .vclear=NULL, so key_del()/vclear() on a kvmgr only returned
-EINVAL. New xkvmgr_del_key (key_find to locate the slot idx ->
key_del — kvtab's val_killer is NULL so the idx scalar is untouched ->
vclear(data, idx)) and xkvmgr_vclear forwarder, wired into the
typeinfo. A missing key propagates -ENOENT (same contract as
xkvmgr_find).
2. Controller periodic sweep (job.c): job_live_gc_sweep — primary-only
(standbys rebuild their live table from WAL replay; deleting there
diverges), collects !job_should_retain jobs under the jobmgr READ
lock with reference() (check_workers_timeout collect pattern), then
per-job WRITE-locked key_del. TWO dereferences per purge: the
collection reference and the table's own makeJob reference — the
jobmgr's datakiller (job_obj_killer) is a deliberate no-op, so
key_del transfers the table's ownership duty to the caller; one
dereference would leak at refcount 1 forever (the very accumulation
this sweep drains). Registered by cluster_start_job_live_gc with
ha_start_primary_heartbeat's idempotent static-arm pattern +
delaymgr.running guard (60s, runcount=-1).
3. Registration on both promotion tails (ha.c): raft road after
ha_start_primary_heartbeat and the non-raft ha_promote_to_primary.
NOT cluster_start — it is orphaned on the HA myapp_run path.
Safety: apply_committed_entry's jobstate group and WAL_JOB_UPDATE
already treat a key_find miss as a benign skip (bug-1970, which
explicitly anticipates post-purge facts); persist_verify_apply_correctness
counts only within the boot replay window. The TRANSIENT TTL now takes
live effect too (previously checkpoint-exclusion only).
UTs: kvmgr del INT+STRING (miss propagation, sibling intact, freed-slot
reuse, killer invocation), controller test_job_live_gc.c (expired DONE
purged + refcount 0, fresh/RUN retained, idempotent re-run, EXIT/PERR,
TRANSIENT TTL + transient_since==0 guard, standby no-op, NULL-safe,
arming guard). xcelib 761/761, controller 1354/1354.
E2E (t1): promote arms the sweep; the pre-restart DONE array jobs were
purged across five sweeps (33 jobs) as their 3600s windows elapsed;
submit→RUN→END flows and all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes so a pushed config edit takes effect on the RUNNING cluster
instead of waiting for a scheduler rejoin (admin-plane-gaps 1.1):
1. queue_refresh_def (controller/queue.c): reconcile_config_queues no
longer leaves existing runtime queues untouched -- it refreshes their
CONFIG-owned fields in place (resreq/pre_exec/post_exec interned
swaps; users/hosts/thresholds/windows via free_* + MOVE_PTR, same
ownership as queue_init; scalars copied) and notifies the live
scheduler/apiserver via the same NO-WAL hook a fresh queue gets.
2. config_refresh (controller/cluster.c): cluster_build_config is now
idempotent -- boot creates the confvar entries, activation re-runs
refresh their interned values in place (makeConfig refuses EEXIST,
kvmgr has no del_key), so heartbeat_interval/
inactive_heartbeat_count exports stop being frozen at boot values.
3. activation re-push (controller/config_repl.c): config_activate_
committed now calls cluster_build_config then xcesched_push_configs
(un-static'd) to the connected scheduler, on both the hash-verified
and -ENOENT fallback activation paths. Lock order follows the
established apply_pump_lock -> sched.lock seam; push failure only
WARNs (scheduler keeps its previous config until next join/push).
E2E-found display gap: apiserver's Go handleNotifyData had no
NotifyData_data_schedUpdateQueue case, so the controller's per-queue
notify was silently dropped and `xce queues list` kept the join-time
snapshot. Added the case (OnQueueUpdated) so hot-applied priority and
queue state changes are visible immediately.
Unit tests: reconcile refresh-in-place + notify counts (test_config_
queue_reconcile.c); activation repushes once per activation, not while
deferred (test_config_repl.c capturing fake); OnQueueUpdated in-place
refresh (handler_test.go). Full controller suite green.
E2E on t1/t2/t3: two config pushes (gen 8->9->10) with no restarts --
controller activation + per-queue refresh + scheduler re-push logged,
scheduler applied at the round boundary, `xce queues list` showed the
new priority immediately, and a submit->RUN->DONE job round-tripped on
the hot-edited queue with all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P1-1: sched_resolve_job_picker() now validates all nine MUST hooks --
the four state-change hooks (on_queue_update/on_job_push/
on_job_state_change/on_job_update) were called unconditionally by
core_events.c, scheduler.c and snapshot_load.c but never validated at
resolution time, so a picker omitting one segfaulted on its first event
instead of failing startup validation with -EINVAL. Doc comment on
core.h's declaration updated to match.
P1-2: handle the picker hooks' return values instead of dropping them.
The known real producer of a rejection is builtin_fcfs's "queue not
(yet) in the candidate registry" -EINVAL: the job stayed tracked in
core->jobs but was never a scheduling candidate, with no log at all --
invisible forever (no dispatch, no PEND reason, nothing). Five sites in
core_events.c now WARN on rc < 0: the JOBS_PUSH fresh-job site plus a
root-cause WARN in core_upsert_queue (a rejected queue makes every job
in it fail too), both JOBS_PUSH resync-merge on_job_state_change sites,
JOB_STATE_CHANGE and JOB_UPDATE. All non-fatal: the push itself still
succeeds and the job stays tracked.
P1-3: sim_bootstrap_picker() is now name-driven with an explicit WARN
and graceful degrade for any picker name that has no sim-viable builtin
picker -- including the legacy "xce_sched_fcfs", whose job_elect_fn
reads plugin->sched (an xce_scheduler_t* the sim driver never has, and
which plugin_register_core() deliberately leaves NULL with no legacy
wrapper), so bootstrapping it would crash the rebuild walk on
picker_iface == NULL. Previously an unknown name silently returned
while sim still warned downstream. A short experiment with bootstrapping
the legacy plugin via an fcfs_new_plugin() factory was reverted: making
that plugin core-only-viable is a refactor of a plugin slated for
removal (audit P2 #18), not a bootstrap fix.
Tests:
- sched_picker_wiring_test: new test_picker_missing_state_change_hooks_
fails_with_einval wires the five ORIGINAL MUST hooks and leaves ONLY
the four state-change hooks NULL, so the asserted -EINVAL can only
come from the new checks.
- sched_core_events_test: new test_jobs_push_picker_rejection_still_
tracks_job_and_succeeds proves the push succeeds, the hook fires
once, and sched_get_job() still resolves the job after a rejection.
P2 items from the same audit are recorded (not fixed) in gaps.md
section 七 (items 18-21): legacy fcfs zombie, dead scaffolding fields,
preemption without a producer, job-priority chain.
134/134 tests pass.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Limit-subsystem audit fixes (#1, #2, #4b, #5, #7; #3/#6/#8 deferred):
#1 PEND reason semantics inversion (the real fix):
- plugins/limit.c: remove the bogus rc<=0 PREASON_LIMIT_GLOBAL/HOST attach
blocks from match_job_global_limit_()/match_job_host_limit_(). match_limit()
only knows scope membership (1 = in scope, 0 = not), never usage-vs-cap --
the attach fired "quota exhausted" for every job OUTSIDE the limit's scope
and never for a real exhaustion.
- resource.c sched_try_alloc_on_host_() + alloc_runner.c
sched_alloc_prepare_shared_(): attach the real verdict in the generic
satisfy check's unmet loops -- an unmet resource with XCE_RESF_LIMIT now
reports PREASON_LIMIT_HOST (per-host loop, host named) / PREASON_LIMIT_GLOBAL
(shared loop, host-less) instead of the generic NO_RESOURCE; the vres name
("limit=<name>[/u=..][/q=..][/p=..]") rides in the reason's resource field,
reverse-parseable to the bare limit name.
#2: rtr_reason_name() now covers codes 4-9 (aligned with pendReasonName() in
the apiserver and reasonScenario() in the MCP server); preason.h's
main-reason comment states the real producer contract.
#4b: make_sched_obj() normalizes obj->project ""/NULL -> "default", matching
sched_get_project()'s default -- a limit scoped to projects.members:[default]
now matches obj->project the same as job->project (the wire decodes an absent
project as "", not NULL).
#5: percent_capacity_resource_map: "swap" -> "swp" (the live load index name
workers register; "swap" matched no live resource, so a percent limit on it
fell back to the live-value base bug-1834 exists to avoid) + add tmp ->
maxtmp. Fixture and lib tests follow the live names.
#7 (coverage): test_alloc_runner.c asserts an unmet shared limit vres gets
PREASON_LIMIT_GLOBAL; test_alloc_runner_search.c asserts an unmet per-host
limit vres gets PREASON_LIMIT_HOST with the host named;
test_repl_trace_e2e.c / test_trace_render.c and the repl_trace_baseline
fixtures (XCE_REPL_BASELINE_REGEN=1) follow the corrected reason rendering.
Full tree: 134/134 tests green (scheduler 100, libxce, harness, controller,
worker, autoscaler, apiserver).
Co-Authored-By: Claude Code <noreply@anthropic.com>
A failed test run's evidence (meson-logs/testlog.txt + the mkdtemp sweep's
FAILED-case /tmp/xce_* fixture dirs) was destroyed by the next build:
setup_build rm -rf's the build dir, and any subsequent meson test overwrites
testlog.* in place. t2 2026-10-22: a controller case failed at ~05:46, the
make clean + rerun at ~05:52 left 134/134 OK and zero trace of what failed.
Two chokes in xce-build.sh:
- finalize_failure_evidence (EXIT trap): on rc!=0 with RUN_TESTS=1, copy
meson-logs to testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/ and drop
the .meson_test_failed_rc marker; also releases the build lock (the old
lock-release trap folded in).
- archive_previous_failure (run start, pre-wipe): if the marker exists,
move the previous run's meson-logs to prev-<stamp>-rc<N>/ -- outside the
build dir, unreachable by the wipe; sweep aged (>5 min) /tmp/xce_*
FAILED-case fixture dirs into prev-<stamp>-fixtures/ so they sit with the
testlog that explains them; cap the archive at newest 10 per build dir.
Also -type d on BOTH /tmp/xce_* sweeps (this one and bug-924's
prune_stale_tmp_fixtures): mkdtemp fixtures are always directories, but
the bare name also matches regular files -- the untyped find live-moved
daemon debug logs (xce_scheduler.log, xce_strace*.log) into the archive,
and the bug-924 variant rm -rf's its matches.
Test: src/tools/test-failure-evidence-archive.sh, 6 scenarios / 11
assertions exercising the real function bodies (sourced with surgical
stubbing) -- all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two independent fixes:
1. Worker pre-exec failure (TODO fix). queue_pre_exec()/job_pre_exec()
previously called exit(1) on failure, killing the one-shot worker
before any status report reached the master -- the job then hung with
no user-visible reason and the worker process vanished. Now the
failure path reports the job FAILED via ipc_worker_send_status with
stage JOB_EXEC_STAGE_{Q_PRE,PRE}_EXEC and the command's real exit
code, skips the job body (no post-exec -- the body never ran,
mirroring LSF pre-exec semantics), and takes the same tail as the
VNC/X11 start failures. main.c now carries job_status->stage through
to xcecd_job_finished() so the stage (not the MAIN default) reaches
the controller's fail fact, deciding both the exit-status merge
(apply_job_fail_transition) and whether the exit code is recorded
(apply_job_fail_exit_code). Also fixes the error print: uv_proc_pre_exec
returns a WEXITSTATUS, not a uv errno, so uv_strerror mangled a plain
exit status into "Unknown system error N".
E2E verified on t1: job-level pre-exec `exit 7` -> job EXIT,
worker logs "Job pre-exec command failed with exit code 7", marker
file shows only the pre-exec ran (no post-exec, no job body), and
the exit code reaches the apiserver's OnJobFinished (exit_code=7)
and the committed WAL_JOB_FAIL fact.
2. Snapshot queue no longer in config wedged election forever.
Checkpoint snapshots record every queue in the live qmgr; if a queue
is deleted from queues.yaml between the checkpoint and a restart,
restore_queue_state returns -ENOENT for it ("queue not yet built
from config -- skip" per its own contract), but
load_snapshot_entities treated any non-zero as fatal ->
persist_recover failed -> cluster_init_phase2 retried forever
(bug-1477 never self-terminates) and no primary was ever elected.
The snapshot queue loop now treats -ENOENT/-EINVAL as a benign skip
with a WARN, matching the same documented policy as
apply_committed_entry's and apply_wal_entry's WAL_QUEUE_CTRL cases;
every other non-zero return stays a genuine failure. Doc comments
updated accordingly.
Live-reproduced and verified on t1 (2026-10-22): deleting a test
queue + restarting the controller previously wedged election
permanently; with the fix the controller boots past the stale frame
with a WARN and reaches CTRL_PRIMARY.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Controller suite flake (user-reported, -j parallel runs): the commit
engine's async timeout sweep armed continuation deadlines with the
hardcoded DEFAULT_SYNC_TIMEOUT_MS (500ms) instead of the configured
sync_timeout_ms, while the blocking persist_commit_barrier honors the
config. Fixtures configuring longer or shorter sync timeouts saw the
sweep resolve -ETIMEDOUT at 500ms before their explicit
truncation/second-call steps, flipping "Expected -17/-2 Was 0/-110"
assertions (test_commit_resolve.c:1458/:1616). commit_submit_async and
commit_sweep_worker now both use p->config.sync_timeout_ms with
DEFAULT_SYNC_TIMEOUT_MS only as the zeroed-config fallback (mirroring
persist_commit_barrier's timeout_ms == 0 convention), and the sweep
WARN reports the effective value.
Apiserver flake (found during full-build verification):
deriveArrayJobStats took the array identity from the first element
seen over store.Jobs.List(), but List() walks a Go map whose order is
randomized per call -- when element rows disagree (the pinned test
case), the stats identity flipped run to run (~12% of -count=1000
runs). The identity carrier is now the lowest Unique_JobID (the array
head, element 1), tracked via arrayAgg.carrierUID; other rows
contribute counters only. Counters were already order-independent and
are unchanged. The test now pins the head-carries-identity contract
explicitly.
Also fixes a latent parse-error trap in the test: a string literal
ending a line (C-style implicit concatenation) triggers Go's automatic
semicolon insertion -- the next line's argument list parses as
"missing ',' before newline". String concatenation across lines must
keep the '+' at the end of the line.
Verified: controller suite 1342/0 (three runs incl. -j 7 parallel
stress); sweep WARNs now match fixture configs (300ms fixture warns at
300, longer-config fixtures emit none); go test -run
TestDeriveArrayJobStats -count=1000 all green (was 118-130 failures);
full ./tools/xce-build.sh all green end to end.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The single JOBS value was doing double duty: the PROCESS_FACTOR
forked-children discount baked into it made sense for meson test -j and
go test -p (flat process caps with no live check), but meson compile
consumes the same number as ninja's -l LOAD-AVERAGE ceiling -- and -l
already re-checks the live run-queue before every launch, so applying the
halved number there too just capped compile parallelism twice. On t1 the
recalibrated 512MB budget produced mem_jobs=9 but cpu_jobs=(8-0.4)*0.5=3,
leaving the memory recalibration without effect on the compile step.
Now: COMPILE_JOBS (raw nproc-minus-load, memory-capped) feeds -l; JOBS
(the same value discounted by PROCESS_FACTOR) feeds the flat test caps.
An explicit --jobs N still sets both, N and N, as before.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The original 1536MB/job figure was calibrated around two memory hogs that
the build/de-nix work has since removed: the 293MB-peak serial capnp
compile (now compiled once via libxce_dep) and the 17 scheduler test
executables each statically linking all of scheduler_srcs (now one shared
sched_srcs_lib archive; link peaks ~125-150MB with -g1). Measured per-job
compile RSS ceiling post-dedup is ~125-150MB; 512MB keeps ~3x headroom
while letting an 8GB box run ~8 concurrent compiles instead of 2-3
(t1: mem_jobs goes from ~2-3 to ~9, so the -l ceiling becomes CPU-bound
rather than memory-bound).
SKILL.md's hand-computed formula is updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Production (hand-fixed, semantics preserved with width-limited %.*s):
- persist.c (33 sites): workdir/name joins share the destination budget;
WAL segment/snapshot paths via a WAL_SEG_WALPATH_MAX macro derived from
seg_path's own declared size; trim loops use d_name's NAME_MAX (255) cap.
- config_repl.c (16 sites) and ha.c's config-hash twin: same half-split
and fixed-literal-tail budgets, byte-identical path construction kept
between ha.c and conf_save_config.
- main.c: certdir/key/cert paths leave room for their fixed suffixes.
Test suites (-Wno-format-truncation, rationale in-tree): the 68 remaining
sites are all snprintf(dir, 4096, "%s/<literal>") fixture scratch paths
under mkdtemp /tmp/xce_* dirs that never approach PATH_MAX -- gcc cannot
prove the tail fits, but every one is a false positive. Applied as a
compile_args declare_dependency on each suite's shared deps list.
Vendored c-capnproto: -Wno-switch (~40 exhaustive-dispatch false
positives), -Wno-unused-function (empty define_encode_function stub),
-Wno-format-overflow (255-byte uniontag + 3-byte prefix vs 256-byte buf,
unreachable schema lengths). Both the wrap checkout and the
packagefiles overlay patched identically.
Also: test_worker_launch_type_dispatch removes its fixed-path fake-xrun
/tmp fixture at exit (bug-924 /tmp hygiene; the mkdtemp shim can't see it).
Verified: fresh full recompile of every source = 0 warnings (gcc 13.3,
-Wall asan profile); meson test 134/134 green; no /tmp/xce_* residue.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sched_unit_test aggregate and 16 zero-#include standalone tests each
compiled all ~50 scheduler sources as their own TUs (~800 redundant
compiles of identical sources per clean build). Add a shared
sched_srcs_lib static archive (scheduler_srcs minus main.c) in the parent
meson.build; those 17 targets now link it and compile only their own test
TUs.
- Only extern-API consumers convert: these tests call nothing but symbols
declared in scheduler.h/core.h. Textual includers (white-box statics
access) keep their own compilation -- their TUs define the included
files' symbols and could collide with archive members; the 4
exclusion-list foreach blocks (snapshot_load_allocator_regression,
repl_trace_wiring, trace_error_paths, trace_bounds) are untouched.
- -Wl,--wrap= interposition is link-time symbol redirection and works
identically against archive members; the three --wrap tests
(publish_retry/backpressure/redelivery) are the sensitive probes and
pass.
- The aggregate keeps -Werror=incompatible-pointer-types and all baked
path defines; its src foreach is gone (the lib carries the sources).
Verified: fresh asan tree 1135 steps (was 1934), ninja green, meson test
134/134 including all converted targets; disk unchanged at ~1.3G after
the previous commit's capnp dedup.
Co-Authored-By: Claude Code <noreply@anthropic.com>
libxce_dep's sources:[gen_srcs] handed every consumer target the generated
.c output, so 143 targets each compiled controller.capnp.c independently:
a 9.5s / 293MB-RSS / 3.3MB-object TU under the asan profile, ~35% of a
clean build's serial work and ~470MB of duplicate objects on disk.
Index the dep's sources to gen_srcs[1] (the .h output) instead: the
custom_target edge still orders every consumer's compiles after codegen
(bug-1798's guarantee, unchanged), .h sources are never compiled, and the
.c is compiled exactly once inside libxce. libxce_src keeps the unindexed
custom_target for the same reason it always did.
Verified: fresh asan tree builds green (1934 steps vs 2076), meson test
134/134 including the three --wrap interposition tests; tree size 2.0G ->
1.3G.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Collapse the 28 zero-#include white-box scheduler tests (whose meson
executable blocks each duplicated the full scheduler_srcs list) into a
single sched_unit_test aggregate, following the controller
controller_test.c precedent: one executable, one process, one runner.
- sched_unit_runner.c: SUITES registry (28 entries) with per-suite
setUp/tearDown dispatch, -t <suite> filter, UnitySetTestFile per entry
- testsuite/meson.build: runner + 28 test TUs + every scheduler_srcs
entry except src/main.c as separate TUs; foreach over scheduler_srcs
so the aggregate can never drift from the production source list;
all baked-path defines (XCE_SCHED_*_PATH, XCE_SIM_*) move to the
aggregate's c_args; -Werror=incompatible-pointer-types carried
suite-wide (verified clean across all 28 tests + 50 src TUs)
- transformed tests drop main()/setUp()/tearDown() and local stubs that
now resolve against the aggregate's real src TUs
- test() registrations 126 -> 99 (126 - 28 + 1 aggregate); the three
--wrap= interposition tests (decision publish retry, backpressure,
decision redelivery) keep their standalone executables unchanged
- isolation libs (sched_core_isolation_lib, sched_sim_isolation_lib)
stay build-only and are wired as test()-level depends; the nm(1)
text-scan tests find them via XCE_CORE_LIB_PATH/XCE_SIM_LIB_PATH
Verified: meson test --suite scheduler 100/100 green x3 consecutive
runs (43s total, aggregate 3.3s); /tmp/xce_* sweep clean (the one
leftover is test_worker_launch_type_dispatch.c's by-design fixed-path
fake xrun launcher).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Deduped census insight: the earlier "730 warnings" figure counted per-TU
echoes of the same source line (white-box textual inclusion recompiles
scheduler.c/limit.c/etc. dozens of times, each copy re-reporting). The
distinct warning sites in our code were 24 unused-variable + 11
unused-function + 2 Wcomment. This commit removes all of them; a clean
full rebuild now reports zero warnings outside the format-truncation
family (125 sites, deferred to its own pass) and vendored capnpc-c.
Notable non-mechanical fixes (the rest is dead-code deletion):
- schedobj.c make_sched_obj() silently swallowed its sub-constructor's
rc (ENOMEM/-EINVAL) and returned 1 unconditionally -- a half-built
schedobj with a NULL union payload could escape to callers. It now
propagates the failure, and make_sched_job() frees and returns NULL
on it instead of continuing with a broken schedobj.
- scheduler.c make_allocs(): vget() rc was set-but-unused; the r==NULL
skip below is the intended handling, so the call is now (void)'ed with
a comment saying so.
- commit.c highest_appended_lsn(): kept but scoped #ifdef XCE_TESTING --
it backs the never-beyond-the-log assert; it had no production caller.
- limit.c get_vres_value(): deleted -- its only call site sits inside a
#if 0 block and its own body was #if 0'ed to a constant.
- group.c update_group_dirty(): deleted recursive dead subtree, no
callers outside itself.
- test_sim_reconcile_unaccounted.c: test_reconcile_respects_host_
capacity_under_contention() was never registered in RUN_TEST; removed
with an NB pointing at the registered bug-1836 coverage.
Verified: clean full rebuild = 0 unused/Wcomment/implicit warnings in
our code (89 unique sites remain, all -Wformat-truncation or vendored);
controller_unit_test 1342/1342, xcelib_test 758/758, 126 scheduler
suites and 18 worker suites all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Warning fixes (from a 730-warning full-build census; these were the
semantically risky ones):
- test_repl_majority.c:272: remove a broken "field-existence" loop whose
condition (i < ep != NULL) parsed wrong (-Wparentheses); the direct
pending_ack_count write it guarded already exists and still fails the
build if the field disappears.
- ha.c:8593: ha_rdhup_election_hook now takes const xce_candidate_t**
so RDHUP election can hand over candidates straight from the const
peers_cfg without stripping const. All 5 test-side hook impls only
(void)peers -- signature updated in ha.h + 3 test files.
- ha.c:726/872/10804 endpoint_connect: constify the host parameter of
endpoint_connect/socket_connect (netio.h/netio.c). The functions only
pass host to gethostbyname(); peer->hostname etc. can now flow in
without discarding const. All callers are read-only and unaffected.
Debug profile (src/meson.build): drop -gdwarf-4 -g3
-fvar-tracking(-assignments) in favor of -ggdb -g1. Debug objects were
~16x larger with full DWARF (scheduler.c: 1.3MB vs 0.08MB), pushing the
build tree to 3.9G; at -g1 it is 850M and a clean build drops from
6m43s to 4m36s. -g1 keeps file/line and backtraces; gdb variable/
macro inspection needs the asan profile. -fno-omit-frame-pointer kept.
Verified: controller_unit_test 1342/1342, xcelib_test 758/758, all 126
scheduler suites + 18 worker suites green; modified sources compile
with zero warnings.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Section 3 documented the retired nix path (nix.conf edits, nix build
.#xcecore, AppImage, nix develop self-wrap). Replace it with the actual
build as it now works on plain Ubuntu:
- 3.1 one-shot apt install (toolchain + 11 C libraries), pointing at
DEPS.md for the versioned dependency table and the wrap-subproject
source closure for everything apt does not carry
- 3.3 daily development via tools/xce-build.sh (profiles, --incremental
caveat), plus the plain meson commands it wraps; Go vendor refresh
without any special shell
- 3.4 tarball packaging via tools/xce-package.sh + install.sh (the
shipping distribution form, replacing the AppImage section)
- 3.5 scripts/ci-ubuntu.sh as the CI entry point
Co-Authored-By: Claude Code <noreply@anthropic.com>
ttyd is the interactive-job launcher for worker TTY/DISPLAY modes: the
worker resolves it next to its own binary (inter_job_utils.c
readlink(/proc/self/exe) + /ttyd, /opt/xce/bin/ttyd fallback), so a
tarball without bin/ttyd silently loses -I/--tty on the target host.
The packager now picks it up from $BUILD_DIR/worker/ttyd first, then
/opt/xce/bin/ttyd, and warns loudly if neither exists.
NB it is NOT an upstream release: this is rshen's modified ttyd (version
string 1.7.7-eccebc6), and the binary we bundle today is that same
modified build. Source lands in-tree later and moves to a meson target;
the packager already prefers a build-dir copy for that day.
DEPS.md: document the ttyd situation (including the MIT-compatibility
audit needed before commercial distribution) and record the user-flagged
license note on bundling libhwloc/libedit/libtirpc -- fine for the
development/test rollout, needs a per-item license audit (bundled
LICENSE file at minimum) before commercial release.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.
Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.
Also:
- test_apply_committed_entry: three new tests covering the skip for all
three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
command. capnpc-c deliberately never frees its compile-time structures
(one-shot tool); under the de-nix asan profile the wrap-built generator
carries -fsanitize=address and LSAN's end-of-process check failed the
generation step. The legacy nix build never propagated -fsanitize to
the subproject. Same exemption convention as every other one-shot/
never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
even though the build host resolves them from /usr/lib.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).
Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
subprojects under src/subprojects/ (source closure, ~500 files): the
upstream trees plus packagefiles/ overlays and diff_files patches
registering dependency/program overrides the nix .pc files used to
provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
Fresh-clone reproducibility verified: wiped trees + `meson subprojects
download` reproduce the committed state exactly (fixed librtti.wrap's
dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.
Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
the dangling build RPATH makes ldd drop the very lines the naive filter
needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
(kernel does not parse quotes in shebangs), ships systemd unit samples +
install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
both bin/ wrapper and venv script.
Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
loopback port so the CLI-error tests assert the no-controller failure
path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
sig_handler re-raises received signals via kill(0, sig) -- process GROUP
wide (strace -k evidence) -- and the child previously shared the
harness's pgrp, so an interrupted SSL handshake killed the whole suite
(exit 130).
Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).
Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes from the MPI parallel-job path review, each unit-tested in the
launch_type_dispatch suite (5 -> 8 tests):
B1 (controller_connector.c): handle_startjob's fan-out-failure path reported
xcecd_task_fail(jobid, body.runtaskmsg->clientidx, ...). In the startjob
request the body union arm is jobstartmsg, so runtaskmsg->clientidx aliases
the same storage as the start_info pointer's high half -- uninitialized
garbage passed to the controller. Pass -1 (the honest value; this start has
no client-side task index, job_ctx->clientidx is -1 downstream). The same
path also fell through to the local start after a failed fan-out; it now
returns the error so a job missing its remote legs is not reported healthy.
B2 (controller_connector.c): xcecd_job_remote_start leaked the taskhost
strings on every remote start -- xcecli_runtask serializes the task
synchronously but takes no ownership, so the caller must free. Add owned
copies for cmdline/host and free all seven heap copies after the call.
Verified clean under ASan with detect_leaks=1.
B3 (xrun.c): the mpirun plm_rsh_agent shim built full_command[4096] with
unbounded strcat from mpirun-passed argv -- a stack smash on any command
line over MAXLINELEN. Bounded append with quoting; overflow is rejected
with rc=1 (verified at the exact 4096-byte boundary).
Co-Authored-By: Claude Code <noreply@anthropic.com>
type=array_stats (xce jobs list -A) read store.ArrayJobs, which has no
production writer, so it always answered zero rows. Stats are now derived
at query time by scanning store.Jobs and aggregating per array base ID
(controllers/arraystats.go), mirroring hosts.go's computeHostSlotUsage
precedent: counters can never drift from the element rows.
- GetJobsInfo/GetJobByIDs array_stats branches run the derived rows
through the same ExecuteQuery filter/sort/paginate pipeline as store
queries; unique_job_id rides in the field list so processJobData can
annotate job_id_human (SelectFields drops unlisted fields).
- processJobData: new rowInt64 helper (uint64 bit-preserved + float64/
json.Number tolerant) replaces brittle bare type assertions for
unique_job_id/array_idx reads; dead code removed (StackLimit
wrong-source copy, unreachable index error).
- QueryData(TableArrayJob) now returns a guard error instead of always
answering zero rows from the never-written store.
- xcecli: ArrayJobStatsData got UnmarshalJSON via UnmarshalOrderMapRow —
the server's OrderMap wrapper ({Fields,Data}) decoded to all-zero
fields, which is why table mode showed 0/- rows against live arrays.
- analysis.md CL5 marked fixed with the real root cause.
Cluster-verified on t1/t2/t3: submit→PEND→RUN→DONE per element, kill
path, job_id filter, and per-status counters all live; all daemons
healthy. Unit suites: apiserver + xcecli green (CGO_ENABLED=0).
Co-Authored-By: Claude Code <noreply@anthropic.com>
API keys were a pure in-memory map (store/apikey.go) — every apiserver
restart silently revoked every service-account key (advisor, MCP
server) until re-issued, with only an inexplicable 401 to show for it.
Keys now persist to <top>/state/apikeys.json: hash → metadata only
(never the raw key, preserving the existing "a dump cannot leak usable
credentials" property), written atomically (temp file + fsync + rename,
0600) on each issuance, loaded at InitStores. The load also restores
the user→role mapping, without which a reloaded key would authenticate
but be denied every permission check. A missing file is a normal fresh
install; a corrupt one fails fast at startup instead of quietly
starting with zero keys. A failed save fails the issuance request —
the raw key is shown exactly once, so we must not hand out a key we
could not persist.
The memory-only path (no SetAPIKeyStatePath) is kept for unit tests.
Cluster-verified on t1-t3: issue key → restart apiserver → key still
authenticates (hosts OK, advisor:report POST published 1) — previously
impossible. Verify job submitted, ran, exited; all daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Case 2b documents `advisor compare` with the live-observed echo-vs-LLM
table (echo 0ms unverified "unknown" vs deepseek ~1.5s severe
cross-field root cause). Case 2 notes deepseek-v4-flash as the
preferred model after glm timed out twice back-to-back on
2026-10-20; pitfalls table updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
One snapshot through BOTH providers per fired scenario, side by side:
CompareRun rows (scenario, trigger, per-side model/latency/tokens/
findings/unverified). The LLM side errors are reported in-band
(llm_error) rather than degrading to echo, so a flaky router shows up
as data instead of a silent echo-only row. The compare path is pure
measurement: nothing logged, nothing pushed — the run stays the only
writer of the decision log.
Live data (pend_stall, cmp job on t1-t3 cluster):
echo 0ms 1 finding, unverified (reason "unknown" opaque)
ds-v4 ~1.6s cross-field catch: job PEND 876-902s while all
nodes idle + no dispatch -> severe, actionable
Echo answers in 0ms with a coarse degraded verdict; the LLM leg adds
cross-field correlation (pend vs host state vs dispatch) that the
static rules structurally cannot express. This is the quantified
incremental-value page for the advisor pitch.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor now installs to ~/bin/xce-advisor (survives /tmp cleanup and
reboots) and is managed by ~/bin/advisor-restart.sh: --check (status +
last log line), --stop, bare start (reads /tmp/advisor_apikey_v2, warns
if the apiserver is unreachable, self-checks liveness ~10s after launch,
refuses to double-start). Handbook pre-flight step 3, the pitfall table
row, and the design doc deployment line now reference the script instead
of a hand-rolled env nohup command.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The dedup disarm fix (e97f6b16) makes close → open → close fire on every
outage, so the handbook's caveat is replaced with a repeatable demo beat
(verified cycle: 11s fire / silent recovery / 22s re-fire).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Live-cluster gap found 2026-10-20 (t1/t2/t3): t3 went down, recovered,
went down again — and the second outage never reported. The dedup key
survived the healthy rounds, so the recurring situation looked
"unchanged" forever. Same hazard for pend_stall: a stall that empties
and later re-forms identically stays silent.
Fix: fire/clear pairing. When a detector's condition is fully absent
(all nodes healthy, no stalled jobs, no skew), it clears its scenario's
dedup key; the next occurrence is a fresh situation and fires. Unit
tests pin the outage→recovery→identical-outage cycle for worker_health
and the stall→resolve→re-form cycle for pend_stall.
exit_burst deliberately does NOT clear: RecentExits is a per-scan
window (consumed once), so an empty window is the steady state, not a
resolution — clearing there would re-fire bursts whose situation never
changed.
Cluster-verified cycle: t3 close → fired (glm, 11.1s); open → silent;
close again → fired again (22.1s, 2 findings). All detect tests green;
advisor restarted on the fix.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Four demo scenarios, all live-verified on the test cluster (2026-10-19/20):
1. pend_stall diagnosis (most reliable; includes the change-triggered
second-fire follow-up demo)
2. real-LLM cross-field inconsistency catch (glm queue-accounting finding,
reproduced twice)
3. worker_health node-down detection (fastest; documents the repeat-outage
dedup gap found while verifying: lastKey is never cleared on recovery,
so a second identical outage stays silent — one-line fix candidate)
4. MCP scheduler_advisor_findings query (zero-risk closer, auditability)
Plus pre-flight checks, advisor start command, known pitfalls table
(in-memory API keys die on apiserver restart, glm timeouts degrade to
echo, SSE listener timeout vs 5-min stall threshold), and cleanup
checklist.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor's findings previously lived only in its JSONL decision log on
its own host. P1 adds two observe-only outlets so diagnoses reach humans
and other agents live:
apiserver side:
- backend/broker.go: EventAdvisorFinding ("advisor.finding") event type +
AdvisorFinding struct + Event.AdvisorFindings field.
- controllers/events.go: ReportFindings — POST /xce/v1/advisor/findings
(generated_at/scenario/findings), publishes one broker event. Broadcast
only: the decision log remains the store of record.
- router/routers.go: route behind RequirePermission("advisor:report").
- middleware/whitelist.go: admit /xce/v1/advisor/findings in the POST URL
whitelist (the URL middleware 403s before the router's RBAC otherwise —
same shape as bug-1489).
- config/rbac.yaml: advisor_agent role (advisor:report + the six *:view
grants the snapshot needs) and advisor:report in the permission
whitelist. mcp_readonly stays strictly *:view. New
TestRBACYAML_AdvisorAgentRole_PinsReportSurface pins the role surface.
advisor side:
- internal/push: Pusher — best-effort POST of each round's findings, one
retry, nil/unconfigured/empty findings are no-ops; failures log and
never affect the observe loop.
- internal/agent: SetPusher wired after the decision-log append (the log
is authoritative; push is a fan-out).
- internal/config + cmd: push default-on to the same XCE_API_URL with the
same XCE_API_KEY (the key's role needs advisor:report); ADVISOR_PUSH=
off|false|0 disables.
mcp-server side:
- internal/advisor.go: lenient JSONL reader for the advisor's decision
log (ADVISOR_LOG env), tail-bounded, corrupt lines skipped.
- scheduler_advisor_findings tool: recent records with parsed findings;
friendly errors when the log is missing/unconfigured. Passive reader —
never talks to the advisor process.
Cluster verification (t1/t2/t3): redeployed apiserver closure, deployed
rbac.yaml, issued advisor_agent key. Forced-PEND job → pend_stall fired
on the real LLM (glm-5-3-flash, 19.4s, degraded=false) → 2 findings, all
evidence verbatim in the snapshot, unverified=false → SSE broadcast
captured on /xce/v1/events → MCP tool returned both records. Second PEND
job joining the stalled set fired a new decision within one round
(change-triggered semantics). Verification jobs killed; sanity job
submitted→ran→DONE; all daemons and 3 hosts healthy.
Design doc: architecture diagram gains the push branch; roadmap P1
marked done (CLI outlet not built).
Co-Authored-By: Claude Code <noreply@anthropic.com>
cJobToModel pre-filled main_reason_name with pendReasonName(main_reason),
so a detail-only record (main_reason 0, the allocator's ordinary shortfall
shape) arrived at rawJobToModel named "unknown(0)" — and the promotion that
derives the overall reason from the first detail row requires an EMPTY
name. The promotion's own tests passed because they build rawJobData
directly; the live C path never hit it. Found by the advisor P0
verification: jobs whose res_reasons named no_resource read
main_reason_name=unknown(0) over the API.
main_reason 0 now leaves the name empty for the promotion to fill; a
non-zero main reason keeps its mapped name unchanged.
Co-Authored-By: Claude Code <noreply@anthropic.com>
snapshot: list jobs with AllUsers + submit_time:desc + limit 200 — the
advisor must see every user's PEND jobs newest-first, or pend_stall sees
nothing on a shared cluster (the default list is the caller's own jobs,
oldest first).
detect: drop the per-scenario cooldown from the fire path. The original
cooldown-first order suppressed a CHANGED situation for up to 10 minutes —
on the live cluster a new job joining the stalled set went unreported
until the cooldown expired. Semantics now: an unchanged situation (same
dedup key) never re-evaluates; any changed situation fires immediately.
A thrashing situation (A→B→A) firing per change is honest reporting — a
flapping node or oscillating stalled set is real signal.
Verified on the t1/t2/t3 cluster: two forced-PEND jobs (unsatisfiable
select; 64-slot limit rejection) both diagnosed in the first scan, a
kill changing the stalled set re-triggered within one 30s round, and
identical situations stayed deduped.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two live-verification defects in the jobs-list path the advisor depends on:
1. The client never passed "sort", so the apiserver's default
unique_job_id:asc + limit returned the OLDEST jobs first — every
recent PEND job was invisible to any limited list. JobFilter.Sort now
forwards to the sort query param.
2. The apiserver's job list handler appends submit_user:eq:<caller> when
no submit_user filter is given (controllers/jobs.go), so even a
scope-all service key saw only its own jobs (0 rows on the live
cluster). JobFilter.AllUsers emits filter={submit_user:eq:all} to get
the cluster-wide view; an explicit Owner still wins.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Architecture rationale (LLM out of the hot path, perception/authority
split, uncalibrated confidence never gates), provider contract, detector
table, runbook-as-poor-man's-RAG, honest limitations of commodity LLMs,
and the P1/P2 path.
Appendix A records the TypeSafe Jev (System One) integration design for
future use only: single /v1/systemone endpoint, noul/choice/score
primitives, the pend_stall_v1 four-question pack and its Finding mapping,
confidence-gated act mode, jev-1.13 jaggedness risk. Deliberately not
wired in P0.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/agent: observe-mode loop — periodic scan + SSE job-event pull
(events only pull the next scan earlier, debounced 5s; job.finished
feeds exit_burst). Provider chain: primary (LLM) → echo fallback on
failure, recorded as degraded. VerifyEvidence firewall runs on every
LLM answer. Observe-only by construction: no write path reachable.
- internal/declog: JSONL decision log (scenario, trigger, provider, model,
latency, tokens, findings, raw output, state snapshot + digest,
degraded flag), size-rotated.
- internal/runbook: incident-pattern seed (embedded YAML) from the
cluster's real debugging history (bug-1494 stale config, IPC ABI skew,
bug-1513 rejoin crashes, bug-1521 reason race, first-fit bias,
dispatch-fail repend family), rendered into the provider prompt.
- cmd: advisor run (signal-handled observe loop) and advisor once
(single round, JSON report to stdout, no decision log).
- internal/config tests; full suite CGO_ENABLED=0 go test ./... clean.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/snapshot: compact cluster state via schedulerclient (nodes,
PEND top-N most-starved with parsed pending_reason, cluster sums, queue
garnish, recent exits). Field whitelists + stall-threshold filter keep
the state a few KiB. Module-local PEND-reason table mirrors
mcp-server/internal/pendreason.go (lift into schedulerclient is a P1
cleanup).
- internal/detect: pend_stall / worker_health / util_skew / exit_burst
triggers — threshold, then per-scenario cooldown, then dedup on a hash
of the situation (job set + reasons / node set) so an unchanged cluster
is never re-evaluated.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P0 of the scheduler advisor (observe-only LLM ops copilot). New standalone
Go module src/frontend/advisor (CGO_ENABLED=0, mcp-server layout, replaces
schedulerclient). No LLM executes anything: providers turn scenario
snapshots into structured Findings only.
- internal/config: env-driven config (XCE_API_URL/KEY, ADVISOR_LLM_*,
ADVISOR_INTERVAL, ADVISOR_LOG); LLM unset is valid (echo-only).
- internal/provider: Provider interface + Finding contract (kind/severity/
summary/evidence/confidence/suggestion). Confidence is display-only —
never calibrated for LLM backends, never gated on.
- echo: deterministic rules provider (fallback, test baseline, yardstick).
- openai: OpenAI-compatible chat-completions (GLM/Qwen/DeepSeek/local
vLLM), strict JSON response_format, enum-validated findings, bounded
429/5xx retries with backoff, 30s default timeout.
- VerifyEvidence: hallucination firewall — evidence elements must appear
verbatim in the state snapshot or the finding is demoted (unverified,
confidence 0.2) while staying in the log.
Jev (TypeSafe System One) deliberately not wired: recorded in the design
doc (later commit) as a future provider behind the same interface.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The per-job MIT-MAGIC-COOKIE-1 that unlocks an -X job's X11 forwarding
port rode every job reply that asked for it: any job:view caller could
read another user's cookie and connect straight to the worker's
forwarding port (6100-6199), bypassing the xattach relay's ownership
check. The SSE job stream carried it too, since its events embed the
full JobInfo row.
- apiserver: stripX11Cookie drops the cookie from GetJobsInfo,
GetJobByIDs and the SSE stream for every job the caller does not own
and holds no job:control grant over — the same rule XAttachJob
enforces. The row itself stays visible; only the secret goes.
- xcecli: preflightX11 checks DISPLAY, the Xauthority file and the
xauth binary before the job is submitted, so a misconfigured display
fails in one second instead of dispatching a job nobody can watch.
Verified end-to-end on t1: owner reads the cookie, a group mate with
job:view(self_ug)+job:control(self) gets the row with the cookie
stripped, an unrelated user is refused at the row level; controllers
tests cover the matrix plus the SSE path.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Render -X jobs' windows on the user's own X server with zero client
install, mirroring the ssh -X experience: the job command starts with
DISPLAY pointed at a per-job TCP listener on the worker, and the user's
xcecli relays that listener's byte stream to their local X server.
- xcecli: new -X/--x11 flag (interactive=2 + exdata "x11":1). After
submit it fetches the per-job MIT-MAGIC-COOKIE-1, registers it in the
user's Xauthority, then bridges one xattach websocket to the local X
server address parsed from $DISPLAY (unix socket or TCP). The cookie
entry is removed when the session ends. Pre-flight checks report
missing DISPLAY/xauth before submit.
- apiserver: GET /xce/v1/jobs/:id/xattach relays websocket binary frames
to the worker's forwarding port (raw TCP, no ttyd). Close codes match
the ttyd attach (4000 unreachable, 4001 ended). The job-finished event
ends the relay. Teardown closes both hops as soon as either pump
exits, so a client disconnect propagates instead of deadlocking in a
blocking read.
- worker: XJOB jobs with exdata x11=1 skip VNC, listen on 6100-6199,
generate a per-job xauth file (/tmp/job_<id>_xauth, worker-side since
the worker has no session), export DISPLAY/XAUTHORITY to the command,
and bridge the app connection (loopback) with the relay connection
(non-loopback; roles classified by source address, not arrival order).
A 60 s no-client watchdog reaps jobs nothing ever attached to; the
v1 single-connection boundary refuses a second X connection loudly.
t1-verified per the plan matrix: window rendering (xclock/xeyes on
Xvfb), normal end -> DONE + xauth removed, no-client 60 s reap (137),
kill -> immediate teardown with no residue, -I/-I --browse regression,
daemon health, missing-DISPLAY error, second-connection refusal.
(--xjob VNC on t1 fails for lack of a vncserver binary; pre-existing.)
Co-Authored-By: Claude Code <noreply@anthropic.com>
An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.
New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:
- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
on its own (-Q), then reap: "Command exited; ttyd did not stop on its
own"
The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.
Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.
Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.
Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.
Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.
Co-Authored-By: Claude Code <noreply@anthropic.com>
waitForInteractiveURL polled PEND every 5s, adding ~4.4s of dead time
to every interactive submit (the job typically reaches RUN with its
session URL ~0.6s after submit once the apiserver applies the
controller's jobPidInfo push), and the RUN-but-no-URL-yet round waited
another 2s. Front-load fast 500ms polls (10 rounds) so a normally
dispatched job attaches within ~1s, then back off to 2s so a job stuck
in a long queue costs the server a query every couple of seconds
instead of a permanent 2 qps; the RUN race polls 500ms too.
checkJobStatus also failed the wait outright on any state other than
PEND/RUN: treat PSUSP like PEND (still queued), and USUSP/SSUSP/
TRANSIENT as keep-waiting states — the allocation is held, a resume
(or failover settle) may follow, and tearing down the attach on a
transient state is wrong.
Cluster-verified on t1: submit-to-attach-attempt 0.56-1.08s across
three runs (previously ~6s even after the jobPidInfo fix), full
submit/run/finish lifecycle clean, daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
handleNotifyData had no case for NotifyData_data_jobPidInfo, so the
controller's start-report push (apiserver_job_start_fun, fired from
HOOK_DETAIL_JOB_START the moment WAL_JOB_START commits) was silently
dropped. interactive_job_url then stayed invisible until the next
periodic rusage-driven job UPDATE re-delivered the full row — up to
10s later (job_rusage_collect_interval), the dominant chunk of an
-I --tty job's submit-to-first-output latency (measured ~6s on t1,
server-side work itself is ~0.6s).
Add OnJobPidInfo to the notification interface and DefaultHandler:
it merges the worker-reported pids/pgids and the exdata {"url": ...}
onto the already-stored job record (get-mutate-put, mirroring
OnJobFinished) and publishes a job.updated event. The CGO decode
side is factored into cPidInfoToRaw/pidInfoJSONB, shared with the
existing xce_job_t conversion.
Cluster-verified on t1: URL is pollable ~0.6s after submit
(previously ~11s), full submit/run/finish lifecycle clean, all
daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three attach-mode (-I) fixes, verified end-to-end on the t1 cluster:
1. Session hang on job finish (job_attach.go): the relay's client->worker
pump blocked forever in ReadMessage after the worker hop closed, so the
old wg.Wait()-before-Close teardown never ran and the client hung, then
died with an abnormal 1006. A broker watcher now subscribes to
EventJobFinished and on job end sends the 4001 policy close and closes
both hops, unblocking the pumps; the teardown follows normally. The
client (attach.go) treats 4001/1000/1001 -- and 1006-after-output as a
final safety net -- as a clean session end.
2. "<<Starting on t1 >>" printed repeatedly (submit.go): the status poll
printed the banner on every transition; now gated on a reportedStart
flag. The URL is also no longer printed in attach mode -- it only makes
sense for browse (-I --tty --browse) and DISPLAY jobs.
3. ttyd left behind after an attach job exits (xcecd_worker.c): a new
tty_stop_attach_session(), called from process_exit_callback for TTY
jobs, kills the per-job ttyd right away. This is insurance -- the sh -c
exec means uv_proc monitors ttyd's own pid, so ttyd is normally gone
before the callback fires -- but it guarantees no orphan when the
timing lines up badly. Web-viewed sessions are untouched: they keep
ttyd and output alive for later browsing, per design.
Co-Authored-By: Claude Code <noreply@anthropic.com>
-I without --xjob now means -I --tty: the submit attaches the interactive
session right in the calling terminal through the apiserver relay. The
previous bare -I mapping (interactive=3, readonly DISPLAY web view) had no
interactive input and gave the user no hint; --browse still opens the web
URL for TTY jobs and --xjob keeps the VNC path. The API's interactive=3
semantics are unchanged -- only this CLI's default remaps.
Help text updated: -I/--tty/--xjob descriptions and the long-help examples.
Co-Authored-By: Claude Code <noreply@anthropic.com>
xjob_start_vnc's four failure points (XCE_SERVER_DIR unset, pre-exec spawn
failure, DISPLAY export failure, port-wait timeout) called exit(1), taking
down the whole per-job worker with no user-visible reason and leaving the
job RUNNING forever (the master never learned of the death until its
instance reap path). It now returns int; the worker_job_callback call site
reports the standard failed-start pattern -- JOB_FAILED + status_message
"Failed to start VNC server (check worker log)" + worker_exit_flag so the
master reaps the instance and starts a fresh one for the next job.
xjob_stop_vnc is cleanup-only: its failures now log and return instead of
exit(1) -- a failed stop must not kill the worker or clobber the job's
exit code.
generate_url sized its snprintf with CMD_MAX_LENGTH (4096) while the url
buffer is URL_MAX_LENGTH (256). Latent since import: the old build's
non-fortified snprintf silently truncated, but any _FORTIFY_SOURCE build
aborts with SIGABRT (__chk_fail) on the FIRST interactive job -- found via
core dump after t1 bare -I jobs died ~1s after start ("*** buffer overflow
detected ***" in generate_url). Size both snprintfs with URL_MAX_LENGTH.
Verified on t1: -I --xjob on a host without vncserver fails in ~1s with
the reason in worker/master logs, worker exits cleanly, next batch job
completes; bare -I submits, dispatches, attaches via the relay and runs
until killed; daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Step 2 of the interactive-job UX work: `-I --tty` now attaches the session
to the submitting terminal by default instead of opening a browser.
Producer/consumer shape (user clusters cannot reach worker ttyd ports):
xcecli (-I --tty, raw terminal)
<-> GET /xce/v1/jobs/:id/attach (JWT auth + job:control ownership)
<-> ws://worker:ttyd-port/ws (ttyd)
apiserver (controllers/job_attach.go):
- new websocket relay endpoint, route gated by RequirePermission("job:control");
owner may always attach, others need a job:control scope covering the owner
- JWT accepted from the Authorization header or ?token= query param
(browser WebSocket API cannot set headers)
- bidirectional frame pump with policy close codes: 4000 = cannot reach the
session on the worker, 4001 = session ended
- the worker hop dials with the "tty" websocket subprotocol: ttyd 1.7.7
completes the handshake without it but then stays silent (verified against
the deployed binary)
xcecli (cmd/jobs/attach.go):
- attach client: dials the relay, raw mode via golang.org/x/term, ttyd
wire format (TEXT hello {"AuthToken",columns,rows}; binary frames
'0'+input, '1'+resize JSON, output '0' to stdout), SIGWINCH resize
forwarding, terminal restored on every exit path
- submit.go: --browse flag opens the web URL instead of attaching (TTY jobs
attach by default; XJOB/VNC keeps the browser path); the URL line is
always printed as a record
- waitForInteractiveURL no longer fails when the job is RUN but the worker's
start report (carrying the URL) has not landed yet -- poll and retry
Verified on t1: raw-client and real-xcecli attach round trips through the
relay (echo both ways), kill -> EXIT with URL retained, batch jobs unaffected,
all daemons healthy. Suites: meson 'apiserver backend' 1/1, CGO_ENABLED=0
controllers/router and xcecli tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The worker packs the interactive session URL (ttyd/XJOB) into the start
report's exdata as {"url": ...}; the controller stores it on
job->job.pidinfo->exdata and the capnp Job wire carries pidinfo (pidinfo@9,
JobPIDInfo.exdata@3) — but apiserver's cJobToModel never mapped pidinfo, so
models.JobInfo.InteractiveJobURL (declared only in the query-field whitelist
JobAdditionalFields) had no producer and interactive_job_url was permanently
empty in API responses. xcecli's waitForInteractiveURL/openBrowser flow could
never fire.
- models.JobInfo: add InteractiveJobURL (json interactive_job_url, matching
the JobAdditionalFields whitelist name).
- convert_pure.go: rawPidInfo (pids/pgids/exdata) + interactiveURLFromExdata;
rawJobToModel marshals pids/pgids into PIDInfo (pid_info is a JSONBJFields
entry, parsed by processJobData) and promotes the exdata URL to
InteractiveJobURL.
- convert.go: cJobToModel extracts job.pidinfo via pidinfo_pid/pidinfo_pgid
cgo helpers (CGO cannot index the C arrays directly).
Cluster-verified end to end on t1 (fresh apiserver deployed): tty job
(xcecli -I --tty equivalent, exdata {"interactive":1}) now returns
interactive_job_url=http://<worker>:9000 and a populated pid_info; non-tty
jobs return ""/no pid_info. All daemons healthy, zero errors in window.
Co-Authored-By: Claude Code <noreply@anthropic.com>
