Strategy layer (spec .kiro/specs/external-license-resources design.md 3.5/3.6):
- limits.yaml samples: extlic_share (per-project, projA/projB, cap 3.0) and
extlic_qdist (per-queue, normal/night, cap 4.0) on the externally reported
extlic pool -- instances "limit=extlic_share/p=<proj>" and
"limit=extlic_qdist/q=<queue>" under the existing limit vres mechanism.
Multi-daemon redundancy is free (resval last-writer-wins); manual override
REST deliberately out of scope.
- White-box scheduler test (test_limit_project_scope.c): project-scope
match + vres instantiation, per-project instance fan-out with caps,
unknown underlying resource failing clean. 3/3 under ASAN.
Controller fix (the Phase 3 E2E blocker, TDD red->green):
- dispatch_a_job_() rebuilt j->job.allocs from the scheduler's dispatch
payload but migrated only the per-host records -- allocs_shared (the
limit vres charge among them) was dropped, and its payload pointer
leaked. That rebuilt record is what sched_job_update_fun() re-sends to
the scheduler as ground truth (the JOB_UPDATE adoption there rebuilds
job->accounts from it), so sharedInUse never accumulated the vres
charge: unsatisfy_ admitted past every global limit forever.
Fix: MOVE the shared records alongside the host records (test
test_dc_success_migrates_shared_allocs).
t1 E2E (4 licenses, extlic_share projA/projB cap 3.0): 3 projA RUN, 4th
projA PEND, projB unaffected (own per-project instance); killing one
RUN job re-admits the pending projA; external free pool and per-project
shares enforced independently; all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.
Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
update_resval_list(c, NULL, ...) — host=="" entries route through
update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
— EEXIST-tolerant so definitions added by an activation register on a
RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
fields (valtype/ordtype/flags/interval) in place so a corrected config
heals a stale registration (bug-1974); seed_sharedinst_from_config skips
EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
capacity while the daemon is down)
Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
snapshot_load's host=="" precedent — previously WARN-skipped as unknown
worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
(F >= uncomit + wanted; F is already net of our checkouts)
xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
nothing and reconnects only fire inside a push — after a controller
restart the daemon sat on a dead connection forever; the main loop now
forces a full re-push when xcecli_is_connection_broken() (bug-1760
predicate) latches
Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
xceadmin tests
Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.
Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
%lu of the packed uint64, so the D-Bus property lookups hit the real
per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
non-zero (previously all 0).
XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
to another host matches the job.
select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
resource ("...references an unregistered resource -- dropping the select
and skipping any rusage[] clause of this primary") instead of failing
silently; scheduler log shows the WARN on t1 and the job schedules
without the constraint.
CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
drops a bare key or an '=value' fragment instead of silently mis-parsing.
Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
primary drop behavior) matching the WARN.
bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).
Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.
Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).
Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
LSF License Scheduler-like external resource management: xceextresd
daemon (flexlm/script collectors) joins the controller with a new
HostRole member and pushes server-free counts through the existing
resval pipeline (wire unchanged apart from HostRole); EXTERNAL-shared
resources skip the inuse term in the scheduler's satisfy check
(Slurm-deficit-equivalent). Covers the elim-compat survey (worker
resource_sync load-index registry activation, separate D-size item),
the license-merge attribution layering (dispatch-ledger quotas +
tiered evidence + residual bucket), and the mergeability review
against Rongsong's unified resource model specs (P9 resolution
proposal: per-resource accounting mode, one writer each).
Proposal stage only: no EARS requirements/tasks yet; the
cross-team items (P9 mode split, project-config cluster-level pool,
peer-auth allowlist) are recorded in design.md section 3.6 for
discussion. Not to be merged to Rongsong's repo.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The LLM server on this host is restarted by hand (llmserver relaunches),
so the observe loop must surface availability instead of discovering an
outage only when a trigger happens to fire — on a quiet cluster that gap
used to be hours. Each scan round now pings the primary (1-token chat
call, single attempt, 5s cap — a probe reports state, it is not a
business call): a flip logs "llm DOWN: <cause>" once and "llm UP again
after <d> down" on recovery, and every clean-scan line carries llm=up/
down/off. Echo-only mode and providers without a Ping method are never
probed and report off/assumed-up — no state is claimed that we do not
know.
Co-Authored-By: Claude Code <noreply@anthropic.com>
`xce id` printed empty name/status: two breaks in the chain.
1. CLI decode: the clusters API returns store-backed rows in the OrderMap
wire format ({"Fields":[...],"Data":{...}}), but ClusterResponse.Rows
([]ClusterInfo) had no custom UnmarshalJSON, so every row landed
all-empty. Same gap array_stats hit; fixed the same way
(UnmarshalOrderMapRow).
2. Server data: the cluster row only comes from OnClusterUpdated (config
push), which never writes Status or ControllerName -- both stayed at
the store zero value (status 0 = DISCONNECT) forever even on a
healthy cluster. The controller's real identity/health snapshot
already exists on the wire (CtrlStatus reply, xce_ctrlstatus_t); the
apiserver just never asked for it.
Fix:
- backend.FetchCtrlStatus(): CGO wrapper of xcecli_ctrlstatus through
callBackend (failover + circuit breaker like every other read),
decoded into ClusterCtrlStatus and freed via free_CtrlStatus_ptr
(bug-1401 leak contract); no-CGO stub added alongside the others.
- GetClustersInfo merges the snapshot into each row before rendering:
cluster_name must match the responder's own (a stale-named row never
claims another cluster's identity), controller_name <- primaryName,
status <- CONNECTED. A backend that doesn't answer keeps the store's
last-known state instead of failing the query.
- CLI ClusterInfo gains the OrderMap-aware UnmarshalJSON.
t1 live: `xce id` now shows name <eda1> / status <CONNECTED> /
controller t1; submit->run->DONE job verified with all daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Terminal jobs (DONE/EXIT/PERR) were only filtered from checkpoints and
WAL replay (job_should_retain, finished_retain_s=3600) — the live jobmgr
never dropped them, so joblist and the scheduler's core->jobs view grew
without bound across a controller's uptime (N4).
Three pieces:
1. kvmgr delete path (src/lib/kvmgr.c): xkvmgr_type had .del_key=NULL
and .vclear=NULL, so key_del()/vclear() on a kvmgr only returned
-EINVAL. New xkvmgr_del_key (key_find to locate the slot idx ->
key_del — kvtab's val_killer is NULL so the idx scalar is untouched ->
vclear(data, idx)) and xkvmgr_vclear forwarder, wired into the
typeinfo. A missing key propagates -ENOENT (same contract as
xkvmgr_find).
2. Controller periodic sweep (job.c): job_live_gc_sweep — primary-only
(standbys rebuild their live table from WAL replay; deleting there
diverges), collects !job_should_retain jobs under the jobmgr READ
lock with reference() (check_workers_timeout collect pattern), then
per-job WRITE-locked key_del. TWO dereferences per purge: the
collection reference and the table's own makeJob reference — the
jobmgr's datakiller (job_obj_killer) is a deliberate no-op, so
key_del transfers the table's ownership duty to the caller; one
dereference would leak at refcount 1 forever (the very accumulation
this sweep drains). Registered by cluster_start_job_live_gc with
ha_start_primary_heartbeat's idempotent static-arm pattern +
delaymgr.running guard (60s, runcount=-1).
3. Registration on both promotion tails (ha.c): raft road after
ha_start_primary_heartbeat and the non-raft ha_promote_to_primary.
NOT cluster_start — it is orphaned on the HA myapp_run path.
Safety: apply_committed_entry's jobstate group and WAL_JOB_UPDATE
already treat a key_find miss as a benign skip (bug-1970, which
explicitly anticipates post-purge facts); persist_verify_apply_correctness
counts only within the boot replay window. The TRANSIENT TTL now takes
live effect too (previously checkpoint-exclusion only).
UTs: kvmgr del INT+STRING (miss propagation, sibling intact, freed-slot
reuse, killer invocation), controller test_job_live_gc.c (expired DONE
purged + refcount 0, fresh/RUN retained, idempotent re-run, EXIT/PERR,
TRANSIENT TTL + transient_since==0 guard, standby no-op, NULL-safe,
arming guard). xcelib 761/761, controller 1354/1354.
E2E (t1): promote arms the sweep; the pre-restart DONE array jobs were
purged across five sweeps (33 jobs) as their 3600s windows elapsed;
submit→RUN→END flows and all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes so a pushed config edit takes effect on the RUNNING cluster
instead of waiting for a scheduler rejoin (admin-plane-gaps 1.1):
1. queue_refresh_def (controller/queue.c): reconcile_config_queues no
longer leaves existing runtime queues untouched -- it refreshes their
CONFIG-owned fields in place (resreq/pre_exec/post_exec interned
swaps; users/hosts/thresholds/windows via free_* + MOVE_PTR, same
ownership as queue_init; scalars copied) and notifies the live
scheduler/apiserver via the same NO-WAL hook a fresh queue gets.
2. config_refresh (controller/cluster.c): cluster_build_config is now
idempotent -- boot creates the confvar entries, activation re-runs
refresh their interned values in place (makeConfig refuses EEXIST,
kvmgr has no del_key), so heartbeat_interval/
inactive_heartbeat_count exports stop being frozen at boot values.
3. activation re-push (controller/config_repl.c): config_activate_
committed now calls cluster_build_config then xcesched_push_configs
(un-static'd) to the connected scheduler, on both the hash-verified
and -ENOENT fallback activation paths. Lock order follows the
established apply_pump_lock -> sched.lock seam; push failure only
WARNs (scheduler keeps its previous config until next join/push).
E2E-found display gap: apiserver's Go handleNotifyData had no
NotifyData_data_schedUpdateQueue case, so the controller's per-queue
notify was silently dropped and `xce queues list` kept the join-time
snapshot. Added the case (OnQueueUpdated) so hot-applied priority and
queue state changes are visible immediately.
Unit tests: reconcile refresh-in-place + notify counts (test_config_
queue_reconcile.c); activation repushes once per activation, not while
deferred (test_config_repl.c capturing fake); OnQueueUpdated in-place
refresh (handler_test.go). Full controller suite green.
E2E on t1/t2/t3: two config pushes (gen 8->9->10) with no restarts --
controller activation + per-queue refresh + scheduler re-push logged,
scheduler applied at the round boundary, `xce queues list` showed the
new priority immediately, and a submit->RUN->DONE job round-tripped on
the hot-edited queue with all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P1-1: sched_resolve_job_picker() now validates all nine MUST hooks --
the four state-change hooks (on_queue_update/on_job_push/
on_job_state_change/on_job_update) were called unconditionally by
core_events.c, scheduler.c and snapshot_load.c but never validated at
resolution time, so a picker omitting one segfaulted on its first event
instead of failing startup validation with -EINVAL. Doc comment on
core.h's declaration updated to match.
P1-2: handle the picker hooks' return values instead of dropping them.
The known real producer of a rejection is builtin_fcfs's "queue not
(yet) in the candidate registry" -EINVAL: the job stayed tracked in
core->jobs but was never a scheduling candidate, with no log at all --
invisible forever (no dispatch, no PEND reason, nothing). Five sites in
core_events.c now WARN on rc < 0: the JOBS_PUSH fresh-job site plus a
root-cause WARN in core_upsert_queue (a rejected queue makes every job
in it fail too), both JOBS_PUSH resync-merge on_job_state_change sites,
JOB_STATE_CHANGE and JOB_UPDATE. All non-fatal: the push itself still
succeeds and the job stays tracked.
P1-3: sim_bootstrap_picker() is now name-driven with an explicit WARN
and graceful degrade for any picker name that has no sim-viable builtin
picker -- including the legacy "xce_sched_fcfs", whose job_elect_fn
reads plugin->sched (an xce_scheduler_t* the sim driver never has, and
which plugin_register_core() deliberately leaves NULL with no legacy
wrapper), so bootstrapping it would crash the rebuild walk on
picker_iface == NULL. Previously an unknown name silently returned
while sim still warned downstream. A short experiment with bootstrapping
the legacy plugin via an fcfs_new_plugin() factory was reverted: making
that plugin core-only-viable is a refactor of a plugin slated for
removal (audit P2 #18), not a bootstrap fix.
Tests:
- sched_picker_wiring_test: new test_picker_missing_state_change_hooks_
fails_with_einval wires the five ORIGINAL MUST hooks and leaves ONLY
the four state-change hooks NULL, so the asserted -EINVAL can only
come from the new checks.
- sched_core_events_test: new test_jobs_push_picker_rejection_still_
tracks_job_and_succeeds proves the push succeeds, the hook fires
once, and sched_get_job() still resolves the job after a rejection.
P2 items from the same audit are recorded (not fixed) in gaps.md
section 七 (items 18-21): legacy fcfs zombie, dead scaffolding fields,
preemption without a producer, job-priority chain.
134/134 tests pass.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Limit-subsystem audit fixes (#1, #2, #4b, #5, #7; #3/#6/#8 deferred):
#1 PEND reason semantics inversion (the real fix):
- plugins/limit.c: remove the bogus rc<=0 PREASON_LIMIT_GLOBAL/HOST attach
blocks from match_job_global_limit_()/match_job_host_limit_(). match_limit()
only knows scope membership (1 = in scope, 0 = not), never usage-vs-cap --
the attach fired "quota exhausted" for every job OUTSIDE the limit's scope
and never for a real exhaustion.
- resource.c sched_try_alloc_on_host_() + alloc_runner.c
sched_alloc_prepare_shared_(): attach the real verdict in the generic
satisfy check's unmet loops -- an unmet resource with XCE_RESF_LIMIT now
reports PREASON_LIMIT_HOST (per-host loop, host named) / PREASON_LIMIT_GLOBAL
(shared loop, host-less) instead of the generic NO_RESOURCE; the vres name
("limit=<name>[/u=..][/q=..][/p=..]") rides in the reason's resource field,
reverse-parseable to the bare limit name.
#2: rtr_reason_name() now covers codes 4-9 (aligned with pendReasonName() in
the apiserver and reasonScenario() in the MCP server); preason.h's
main-reason comment states the real producer contract.
#4b: make_sched_obj() normalizes obj->project ""/NULL -> "default", matching
sched_get_project()'s default -- a limit scoped to projects.members:[default]
now matches obj->project the same as job->project (the wire decodes an absent
project as "", not NULL).
#5: percent_capacity_resource_map: "swap" -> "swp" (the live load index name
workers register; "swap" matched no live resource, so a percent limit on it
fell back to the live-value base bug-1834 exists to avoid) + add tmp ->
maxtmp. Fixture and lib tests follow the live names.
#7 (coverage): test_alloc_runner.c asserts an unmet shared limit vres gets
PREASON_LIMIT_GLOBAL; test_alloc_runner_search.c asserts an unmet per-host
limit vres gets PREASON_LIMIT_HOST with the host named;
test_repl_trace_e2e.c / test_trace_render.c and the repl_trace_baseline
fixtures (XCE_REPL_BASELINE_REGEN=1) follow the corrected reason rendering.
Full tree: 134/134 tests green (scheduler 100, libxce, harness, controller,
worker, autoscaler, apiserver).
Co-Authored-By: Claude Code <noreply@anthropic.com>
A failed test run's evidence (meson-logs/testlog.txt + the mkdtemp sweep's
FAILED-case /tmp/xce_* fixture dirs) was destroyed by the next build:
setup_build rm -rf's the build dir, and any subsequent meson test overwrites
testlog.* in place. t2 2026-10-22: a controller case failed at ~05:46, the
make clean + rerun at ~05:52 left 134/134 OK and zero trace of what failed.
Two chokes in xce-build.sh:
- finalize_failure_evidence (EXIT trap): on rc!=0 with RUN_TESTS=1, copy
meson-logs to testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/ and drop
the .meson_test_failed_rc marker; also releases the build lock (the old
lock-release trap folded in).
- archive_previous_failure (run start, pre-wipe): if the marker exists,
move the previous run's meson-logs to prev-<stamp>-rc<N>/ -- outside the
build dir, unreachable by the wipe; sweep aged (>5 min) /tmp/xce_*
FAILED-case fixture dirs into prev-<stamp>-fixtures/ so they sit with the
testlog that explains them; cap the archive at newest 10 per build dir.
Also -type d on BOTH /tmp/xce_* sweeps (this one and bug-924's
prune_stale_tmp_fixtures): mkdtemp fixtures are always directories, but
the bare name also matches regular files -- the untyped find live-moved
daemon debug logs (xce_scheduler.log, xce_strace*.log) into the archive,
and the bug-924 variant rm -rf's its matches.
Test: src/tools/test-failure-evidence-archive.sh, 6 scenarios / 11
assertions exercising the real function bodies (sourced with surgical
stubbing) -- all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two independent fixes:
1. Worker pre-exec failure (TODO fix). queue_pre_exec()/job_pre_exec()
previously called exit(1) on failure, killing the one-shot worker
before any status report reached the master -- the job then hung with
no user-visible reason and the worker process vanished. Now the
failure path reports the job FAILED via ipc_worker_send_status with
stage JOB_EXEC_STAGE_{Q_PRE,PRE}_EXEC and the command's real exit
code, skips the job body (no post-exec -- the body never ran,
mirroring LSF pre-exec semantics), and takes the same tail as the
VNC/X11 start failures. main.c now carries job_status->stage through
to xcecd_job_finished() so the stage (not the MAIN default) reaches
the controller's fail fact, deciding both the exit-status merge
(apply_job_fail_transition) and whether the exit code is recorded
(apply_job_fail_exit_code). Also fixes the error print: uv_proc_pre_exec
returns a WEXITSTATUS, not a uv errno, so uv_strerror mangled a plain
exit status into "Unknown system error N".
E2E verified on t1: job-level pre-exec `exit 7` -> job EXIT,
worker logs "Job pre-exec command failed with exit code 7", marker
file shows only the pre-exec ran (no post-exec, no job body), and
the exit code reaches the apiserver's OnJobFinished (exit_code=7)
and the committed WAL_JOB_FAIL fact.
2. Snapshot queue no longer in config wedged election forever.
Checkpoint snapshots record every queue in the live qmgr; if a queue
is deleted from queues.yaml between the checkpoint and a restart,
restore_queue_state returns -ENOENT for it ("queue not yet built
from config -- skip" per its own contract), but
load_snapshot_entities treated any non-zero as fatal ->
persist_recover failed -> cluster_init_phase2 retried forever
(bug-1477 never self-terminates) and no primary was ever elected.
The snapshot queue loop now treats -ENOENT/-EINVAL as a benign skip
with a WARN, matching the same documented policy as
apply_committed_entry's and apply_wal_entry's WAL_QUEUE_CTRL cases;
every other non-zero return stays a genuine failure. Doc comments
updated accordingly.
Live-reproduced and verified on t1 (2026-10-22): deleting a test
queue + restarting the controller previously wedged election
permanently; with the fix the controller boots past the stale frame
with a WARN and reaches CTRL_PRIMARY.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Controller suite flake (user-reported, -j parallel runs): the commit
engine's async timeout sweep armed continuation deadlines with the
hardcoded DEFAULT_SYNC_TIMEOUT_MS (500ms) instead of the configured
sync_timeout_ms, while the blocking persist_commit_barrier honors the
config. Fixtures configuring longer or shorter sync timeouts saw the
sweep resolve -ETIMEDOUT at 500ms before their explicit
truncation/second-call steps, flipping "Expected -17/-2 Was 0/-110"
assertions (test_commit_resolve.c:1458/:1616). commit_submit_async and
commit_sweep_worker now both use p->config.sync_timeout_ms with
DEFAULT_SYNC_TIMEOUT_MS only as the zeroed-config fallback (mirroring
persist_commit_barrier's timeout_ms == 0 convention), and the sweep
WARN reports the effective value.
Apiserver flake (found during full-build verification):
deriveArrayJobStats took the array identity from the first element
seen over store.Jobs.List(), but List() walks a Go map whose order is
randomized per call -- when element rows disagree (the pinned test
case), the stats identity flipped run to run (~12% of -count=1000
runs). The identity carrier is now the lowest Unique_JobID (the array
head, element 1), tracked via arrayAgg.carrierUID; other rows
contribute counters only. Counters were already order-independent and
are unchanged. The test now pins the head-carries-identity contract
explicitly.
Also fixes a latent parse-error trap in the test: a string literal
ending a line (C-style implicit concatenation) triggers Go's automatic
semicolon insertion -- the next line's argument list parses as
"missing ',' before newline". String concatenation across lines must
keep the '+' at the end of the line.
Verified: controller suite 1342/0 (three runs incl. -j 7 parallel
stress); sweep WARNs now match fixture configs (300ms fixture warns at
300, longer-config fixtures emit none); go test -run
TestDeriveArrayJobStats -count=1000 all green (was 118-130 failures);
full ./tools/xce-build.sh all green end to end.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The single JOBS value was doing double duty: the PROCESS_FACTOR
forked-children discount baked into it made sense for meson test -j and
go test -p (flat process caps with no live check), but meson compile
consumes the same number as ninja's -l LOAD-AVERAGE ceiling -- and -l
already re-checks the live run-queue before every launch, so applying the
halved number there too just capped compile parallelism twice. On t1 the
recalibrated 512MB budget produced mem_jobs=9 but cpu_jobs=(8-0.4)*0.5=3,
leaving the memory recalibration without effect on the compile step.
Now: COMPILE_JOBS (raw nproc-minus-load, memory-capped) feeds -l; JOBS
(the same value discounted by PROCESS_FACTOR) feeds the flat test caps.
An explicit --jobs N still sets both, N and N, as before.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The original 1536MB/job figure was calibrated around two memory hogs that
the build/de-nix work has since removed: the 293MB-peak serial capnp
compile (now compiled once via libxce_dep) and the 17 scheduler test
executables each statically linking all of scheduler_srcs (now one shared
sched_srcs_lib archive; link peaks ~125-150MB with -g1). Measured per-job
compile RSS ceiling post-dedup is ~125-150MB; 512MB keeps ~3x headroom
while letting an 8GB box run ~8 concurrent compiles instead of 2-3
(t1: mem_jobs goes from ~2-3 to ~9, so the -l ceiling becomes CPU-bound
rather than memory-bound).
SKILL.md's hand-computed formula is updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Production (hand-fixed, semantics preserved with width-limited %.*s):
- persist.c (33 sites): workdir/name joins share the destination budget;
WAL segment/snapshot paths via a WAL_SEG_WALPATH_MAX macro derived from
seg_path's own declared size; trim loops use d_name's NAME_MAX (255) cap.
- config_repl.c (16 sites) and ha.c's config-hash twin: same half-split
and fixed-literal-tail budgets, byte-identical path construction kept
between ha.c and conf_save_config.
- main.c: certdir/key/cert paths leave room for their fixed suffixes.
Test suites (-Wno-format-truncation, rationale in-tree): the 68 remaining
sites are all snprintf(dir, 4096, "%s/<literal>") fixture scratch paths
under mkdtemp /tmp/xce_* dirs that never approach PATH_MAX -- gcc cannot
prove the tail fits, but every one is a false positive. Applied as a
compile_args declare_dependency on each suite's shared deps list.
Vendored c-capnproto: -Wno-switch (~40 exhaustive-dispatch false
positives), -Wno-unused-function (empty define_encode_function stub),
-Wno-format-overflow (255-byte uniontag + 3-byte prefix vs 256-byte buf,
unreachable schema lengths). Both the wrap checkout and the
packagefiles overlay patched identically.
Also: test_worker_launch_type_dispatch removes its fixed-path fake-xrun
/tmp fixture at exit (bug-924 /tmp hygiene; the mkdtemp shim can't see it).
Verified: fresh full recompile of every source = 0 warnings (gcc 13.3,
-Wall asan profile); meson test 134/134 green; no /tmp/xce_* residue.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sched_unit_test aggregate and 16 zero-#include standalone tests each
compiled all ~50 scheduler sources as their own TUs (~800 redundant
compiles of identical sources per clean build). Add a shared
sched_srcs_lib static archive (scheduler_srcs minus main.c) in the parent
meson.build; those 17 targets now link it and compile only their own test
TUs.
- Only extern-API consumers convert: these tests call nothing but symbols
declared in scheduler.h/core.h. Textual includers (white-box statics
access) keep their own compilation -- their TUs define the included
files' symbols and could collide with archive members; the 4
exclusion-list foreach blocks (snapshot_load_allocator_regression,
repl_trace_wiring, trace_error_paths, trace_bounds) are untouched.
- -Wl,--wrap= interposition is link-time symbol redirection and works
identically against archive members; the three --wrap tests
(publish_retry/backpressure/redelivery) are the sensitive probes and
pass.
- The aggregate keeps -Werror=incompatible-pointer-types and all baked
path defines; its src foreach is gone (the lib carries the sources).
Verified: fresh asan tree 1135 steps (was 1934), ninja green, meson test
134/134 including all converted targets; disk unchanged at ~1.3G after
the previous commit's capnp dedup.
Co-Authored-By: Claude Code <noreply@anthropic.com>
libxce_dep's sources:[gen_srcs] handed every consumer target the generated
.c output, so 143 targets each compiled controller.capnp.c independently:
a 9.5s / 293MB-RSS / 3.3MB-object TU under the asan profile, ~35% of a
clean build's serial work and ~470MB of duplicate objects on disk.
Index the dep's sources to gen_srcs[1] (the .h output) instead: the
custom_target edge still orders every consumer's compiles after codegen
(bug-1798's guarantee, unchanged), .h sources are never compiled, and the
.c is compiled exactly once inside libxce. libxce_src keeps the unindexed
custom_target for the same reason it always did.
Verified: fresh asan tree builds green (1934 steps vs 2076), meson test
134/134 including the three --wrap interposition tests; tree size 2.0G ->
1.3G.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Collapse the 28 zero-#include white-box scheduler tests (whose meson
executable blocks each duplicated the full scheduler_srcs list) into a
single sched_unit_test aggregate, following the controller
controller_test.c precedent: one executable, one process, one runner.
- sched_unit_runner.c: SUITES registry (28 entries) with per-suite
setUp/tearDown dispatch, -t <suite> filter, UnitySetTestFile per entry
- testsuite/meson.build: runner + 28 test TUs + every scheduler_srcs
entry except src/main.c as separate TUs; foreach over scheduler_srcs
so the aggregate can never drift from the production source list;
all baked-path defines (XCE_SCHED_*_PATH, XCE_SIM_*) move to the
aggregate's c_args; -Werror=incompatible-pointer-types carried
suite-wide (verified clean across all 28 tests + 50 src TUs)
- transformed tests drop main()/setUp()/tearDown() and local stubs that
now resolve against the aggregate's real src TUs
- test() registrations 126 -> 99 (126 - 28 + 1 aggregate); the three
--wrap= interposition tests (decision publish retry, backpressure,
decision redelivery) keep their standalone executables unchanged
- isolation libs (sched_core_isolation_lib, sched_sim_isolation_lib)
stay build-only and are wired as test()-level depends; the nm(1)
text-scan tests find them via XCE_CORE_LIB_PATH/XCE_SIM_LIB_PATH
Verified: meson test --suite scheduler 100/100 green x3 consecutive
runs (43s total, aggregate 3.3s); /tmp/xce_* sweep clean (the one
leftover is test_worker_launch_type_dispatch.c's by-design fixed-path
fake xrun launcher).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Deduped census insight: the earlier "730 warnings" figure counted per-TU
echoes of the same source line (white-box textual inclusion recompiles
scheduler.c/limit.c/etc. dozens of times, each copy re-reporting). The
distinct warning sites in our code were 24 unused-variable + 11
unused-function + 2 Wcomment. This commit removes all of them; a clean
full rebuild now reports zero warnings outside the format-truncation
family (125 sites, deferred to its own pass) and vendored capnpc-c.
Notable non-mechanical fixes (the rest is dead-code deletion):
- schedobj.c make_sched_obj() silently swallowed its sub-constructor's
rc (ENOMEM/-EINVAL) and returned 1 unconditionally -- a half-built
schedobj with a NULL union payload could escape to callers. It now
propagates the failure, and make_sched_job() frees and returns NULL
on it instead of continuing with a broken schedobj.
- scheduler.c make_allocs(): vget() rc was set-but-unused; the r==NULL
skip below is the intended handling, so the call is now (void)'ed with
a comment saying so.
- commit.c highest_appended_lsn(): kept but scoped #ifdef XCE_TESTING --
it backs the never-beyond-the-log assert; it had no production caller.
- limit.c get_vres_value(): deleted -- its only call site sits inside a
#if 0 block and its own body was #if 0'ed to a constant.
- group.c update_group_dirty(): deleted recursive dead subtree, no
callers outside itself.
- test_sim_reconcile_unaccounted.c: test_reconcile_respects_host_
capacity_under_contention() was never registered in RUN_TEST; removed
with an NB pointing at the registered bug-1836 coverage.
Verified: clean full rebuild = 0 unused/Wcomment/implicit warnings in
our code (89 unique sites remain, all -Wformat-truncation or vendored);
controller_unit_test 1342/1342, xcelib_test 758/758, 126 scheduler
suites and 18 worker suites all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Warning fixes (from a 730-warning full-build census; these were the
semantically risky ones):
- test_repl_majority.c:272: remove a broken "field-existence" loop whose
condition (i < ep != NULL) parsed wrong (-Wparentheses); the direct
pending_ack_count write it guarded already exists and still fails the
build if the field disappears.
- ha.c:8593: ha_rdhup_election_hook now takes const xce_candidate_t**
so RDHUP election can hand over candidates straight from the const
peers_cfg without stripping const. All 5 test-side hook impls only
(void)peers -- signature updated in ha.h + 3 test files.
- ha.c:726/872/10804 endpoint_connect: constify the host parameter of
endpoint_connect/socket_connect (netio.h/netio.c). The functions only
pass host to gethostbyname(); peer->hostname etc. can now flow in
without discarding const. All callers are read-only and unaffected.
Debug profile (src/meson.build): drop -gdwarf-4 -g3
-fvar-tracking(-assignments) in favor of -ggdb -g1. Debug objects were
~16x larger with full DWARF (scheduler.c: 1.3MB vs 0.08MB), pushing the
build tree to 3.9G; at -g1 it is 850M and a clean build drops from
6m43s to 4m36s. -g1 keeps file/line and backtraces; gdb variable/
macro inspection needs the asan profile. -fno-omit-frame-pointer kept.
Verified: controller_unit_test 1342/1342, xcelib_test 758/758, all 126
scheduler suites + 18 worker suites green; modified sources compile
with zero warnings.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Section 3 documented the retired nix path (nix.conf edits, nix build
.#xcecore, AppImage, nix develop self-wrap). Replace it with the actual
build as it now works on plain Ubuntu:
- 3.1 one-shot apt install (toolchain + 11 C libraries), pointing at
DEPS.md for the versioned dependency table and the wrap-subproject
source closure for everything apt does not carry
- 3.3 daily development via tools/xce-build.sh (profiles, --incremental
caveat), plus the plain meson commands it wraps; Go vendor refresh
without any special shell
- 3.4 tarball packaging via tools/xce-package.sh + install.sh (the
shipping distribution form, replacing the AppImage section)
- 3.5 scripts/ci-ubuntu.sh as the CI entry point
Co-Authored-By: Claude Code <noreply@anthropic.com>
ttyd is the interactive-job launcher for worker TTY/DISPLAY modes: the
worker resolves it next to its own binary (inter_job_utils.c
readlink(/proc/self/exe) + /ttyd, /opt/xce/bin/ttyd fallback), so a
tarball without bin/ttyd silently loses -I/--tty on the target host.
The packager now picks it up from $BUILD_DIR/worker/ttyd first, then
/opt/xce/bin/ttyd, and warns loudly if neither exists.
NB it is NOT an upstream release: this is rshen's modified ttyd (version
string 1.7.7-eccebc6), and the binary we bundle today is that same
modified build. Source lands in-tree later and moves to a meson target;
the packager already prefers a build-dir copy for that day.
DEPS.md: document the ttyd situation (including the MIT-compatibility
audit needed before commercial distribution) and record the user-flagged
license note on bundling libhwloc/libedit/libtirpc -- fine for the
development/test rollout, needs a per-item license audit (bundled
LICENSE file at minimum) before commercial release.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.
Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.
Also:
- test_apply_committed_entry: three new tests covering the skip for all
three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
command. capnpc-c deliberately never frees its compile-time structures
(one-shot tool); under the de-nix asan profile the wrap-built generator
carries -fsanitize=address and LSAN's end-of-process check failed the
generation step. The legacy nix build never propagated -fsanitize to
the subproject. Same exemption convention as every other one-shot/
never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
even though the build host resolves them from /usr/lib.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).
Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
subprojects under src/subprojects/ (source closure, ~500 files): the
upstream trees plus packagefiles/ overlays and diff_files patches
registering dependency/program overrides the nix .pc files used to
provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
Fresh-clone reproducibility verified: wiped trees + `meson subprojects
download` reproduce the committed state exactly (fixed librtti.wrap's
dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.
Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
the dangling build RPATH makes ldd drop the very lines the naive filter
needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
(kernel does not parse quotes in shebangs), ships systemd unit samples +
install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
both bin/ wrapper and venv script.
Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
loopback port so the CLI-error tests assert the no-controller failure
path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
sig_handler re-raises received signals via kill(0, sig) -- process GROUP
wide (strace -k evidence) -- and the child previously shared the
harness's pgrp, so an interrupted SSL handshake killed the whole suite
(exit 130).
Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).
Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes from the MPI parallel-job path review, each unit-tested in the
launch_type_dispatch suite (5 -> 8 tests):
B1 (controller_connector.c): handle_startjob's fan-out-failure path reported
xcecd_task_fail(jobid, body.runtaskmsg->clientidx, ...). In the startjob
request the body union arm is jobstartmsg, so runtaskmsg->clientidx aliases
the same storage as the start_info pointer's high half -- uninitialized
garbage passed to the controller. Pass -1 (the honest value; this start has
no client-side task index, job_ctx->clientidx is -1 downstream). The same
path also fell through to the local start after a failed fan-out; it now
returns the error so a job missing its remote legs is not reported healthy.
B2 (controller_connector.c): xcecd_job_remote_start leaked the taskhost
strings on every remote start -- xcecli_runtask serializes the task
synchronously but takes no ownership, so the caller must free. Add owned
copies for cmdline/host and free all seven heap copies after the call.
Verified clean under ASan with detect_leaks=1.
B3 (xrun.c): the mpirun plm_rsh_agent shim built full_command[4096] with
unbounded strcat from mpirun-passed argv -- a stack smash on any command
line over MAXLINELEN. Bounded append with quoting; overflow is rejected
with rc=1 (verified at the exact 4096-byte boundary).
Co-Authored-By: Claude Code <noreply@anthropic.com>
type=array_stats (xce jobs list -A) read store.ArrayJobs, which has no
production writer, so it always answered zero rows. Stats are now derived
at query time by scanning store.Jobs and aggregating per array base ID
(controllers/arraystats.go), mirroring hosts.go's computeHostSlotUsage
precedent: counters can never drift from the element rows.
- GetJobsInfo/GetJobByIDs array_stats branches run the derived rows
through the same ExecuteQuery filter/sort/paginate pipeline as store
queries; unique_job_id rides in the field list so processJobData can
annotate job_id_human (SelectFields drops unlisted fields).
- processJobData: new rowInt64 helper (uint64 bit-preserved + float64/
json.Number tolerant) replaces brittle bare type assertions for
unique_job_id/array_idx reads; dead code removed (StackLimit
wrong-source copy, unreachable index error).
- QueryData(TableArrayJob) now returns a guard error instead of always
answering zero rows from the never-written store.
- xcecli: ArrayJobStatsData got UnmarshalJSON via UnmarshalOrderMapRow —
the server's OrderMap wrapper ({Fields,Data}) decoded to all-zero
fields, which is why table mode showed 0/- rows against live arrays.
- analysis.md CL5 marked fixed with the real root cause.
Cluster-verified on t1/t2/t3: submit→PEND→RUN→DONE per element, kill
path, job_id filter, and per-status counters all live; all daemons
healthy. Unit suites: apiserver + xcecli green (CGO_ENABLED=0).
Co-Authored-By: Claude Code <noreply@anthropic.com>
API keys were a pure in-memory map (store/apikey.go) — every apiserver
restart silently revoked every service-account key (advisor, MCP
server) until re-issued, with only an inexplicable 401 to show for it.
Keys now persist to <top>/state/apikeys.json: hash → metadata only
(never the raw key, preserving the existing "a dump cannot leak usable
credentials" property), written atomically (temp file + fsync + rename,
0600) on each issuance, loaded at InitStores. The load also restores
the user→role mapping, without which a reloaded key would authenticate
but be denied every permission check. A missing file is a normal fresh
install; a corrupt one fails fast at startup instead of quietly
starting with zero keys. A failed save fails the issuance request —
the raw key is shown exactly once, so we must not hand out a key we
could not persist.
The memory-only path (no SetAPIKeyStatePath) is kept for unit tests.
Cluster-verified on t1-t3: issue key → restart apiserver → key still
authenticates (hosts OK, advisor:report POST published 1) — previously
impossible. Verify job submitted, ran, exited; all daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Case 2b documents `advisor compare` with the live-observed echo-vs-LLM
table (echo 0ms unverified "unknown" vs deepseek ~1.5s severe
cross-field root cause). Case 2 notes deepseek-v4-flash as the
preferred model after glm timed out twice back-to-back on
2026-10-20; pitfalls table updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
One snapshot through BOTH providers per fired scenario, side by side:
CompareRun rows (scenario, trigger, per-side model/latency/tokens/
findings/unverified). The LLM side errors are reported in-band
(llm_error) rather than degrading to echo, so a flaky router shows up
as data instead of a silent echo-only row. The compare path is pure
measurement: nothing logged, nothing pushed — the run stays the only
writer of the decision log.
Live data (pend_stall, cmp job on t1-t3 cluster):
echo 0ms 1 finding, unverified (reason "unknown" opaque)
ds-v4 ~1.6s cross-field catch: job PEND 876-902s while all
nodes idle + no dispatch -> severe, actionable
Echo answers in 0ms with a coarse degraded verdict; the LLM leg adds
cross-field correlation (pend vs host state vs dispatch) that the
static rules structurally cannot express. This is the quantified
incremental-value page for the advisor pitch.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor now installs to ~/bin/xce-advisor (survives /tmp cleanup and
reboots) and is managed by ~/bin/advisor-restart.sh: --check (status +
last log line), --stop, bare start (reads /tmp/advisor_apikey_v2, warns
if the apiserver is unreachable, self-checks liveness ~10s after launch,
refuses to double-start). Handbook pre-flight step 3, the pitfall table
row, and the design doc deployment line now reference the script instead
of a hand-rolled env nohup command.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The dedup disarm fix (e97f6b16) makes close → open → close fire on every
outage, so the handbook's caveat is replaced with a repeatable demo beat
(verified cycle: 11s fire / silent recovery / 22s re-fire).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Live-cluster gap found 2026-10-20 (t1/t2/t3): t3 went down, recovered,
went down again — and the second outage never reported. The dedup key
survived the healthy rounds, so the recurring situation looked
"unchanged" forever. Same hazard for pend_stall: a stall that empties
and later re-forms identically stays silent.
Fix: fire/clear pairing. When a detector's condition is fully absent
(all nodes healthy, no stalled jobs, no skew), it clears its scenario's
dedup key; the next occurrence is a fresh situation and fires. Unit
tests pin the outage→recovery→identical-outage cycle for worker_health
and the stall→resolve→re-form cycle for pend_stall.
exit_burst deliberately does NOT clear: RecentExits is a per-scan
window (consumed once), so an empty window is the steady state, not a
resolution — clearing there would re-fire bursts whose situation never
changed.
Cluster-verified cycle: t3 close → fired (glm, 11.1s); open → silent;
close again → fired again (22.1s, 2 findings). All detect tests green;
advisor restarted on the fix.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Four demo scenarios, all live-verified on the test cluster (2026-10-19/20):
1. pend_stall diagnosis (most reliable; includes the change-triggered
second-fire follow-up demo)
2. real-LLM cross-field inconsistency catch (glm queue-accounting finding,
reproduced twice)
3. worker_health node-down detection (fastest; documents the repeat-outage
dedup gap found while verifying: lastKey is never cleared on recovery,
so a second identical outage stays silent — one-line fix candidate)
4. MCP scheduler_advisor_findings query (zero-risk closer, auditability)
Plus pre-flight checks, advisor start command, known pitfalls table
(in-memory API keys die on apiserver restart, glm timeouts degrade to
echo, SSE listener timeout vs 5-min stall threshold), and cleanup
checklist.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor's findings previously lived only in its JSONL decision log on
its own host. P1 adds two observe-only outlets so diagnoses reach humans
and other agents live:
apiserver side:
- backend/broker.go: EventAdvisorFinding ("advisor.finding") event type +
AdvisorFinding struct + Event.AdvisorFindings field.
- controllers/events.go: ReportFindings — POST /xce/v1/advisor/findings
(generated_at/scenario/findings), publishes one broker event. Broadcast
only: the decision log remains the store of record.
- router/routers.go: route behind RequirePermission("advisor:report").
- middleware/whitelist.go: admit /xce/v1/advisor/findings in the POST URL
whitelist (the URL middleware 403s before the router's RBAC otherwise —
same shape as bug-1489).
- config/rbac.yaml: advisor_agent role (advisor:report + the six *:view
grants the snapshot needs) and advisor:report in the permission
whitelist. mcp_readonly stays strictly *:view. New
TestRBACYAML_AdvisorAgentRole_PinsReportSurface pins the role surface.
advisor side:
- internal/push: Pusher — best-effort POST of each round's findings, one
retry, nil/unconfigured/empty findings are no-ops; failures log and
never affect the observe loop.
- internal/agent: SetPusher wired after the decision-log append (the log
is authoritative; push is a fan-out).
- internal/config + cmd: push default-on to the same XCE_API_URL with the
same XCE_API_KEY (the key's role needs advisor:report); ADVISOR_PUSH=
off|false|0 disables.
mcp-server side:
- internal/advisor.go: lenient JSONL reader for the advisor's decision
log (ADVISOR_LOG env), tail-bounded, corrupt lines skipped.
- scheduler_advisor_findings tool: recent records with parsed findings;
friendly errors when the log is missing/unconfigured. Passive reader —
never talks to the advisor process.
Cluster verification (t1/t2/t3): redeployed apiserver closure, deployed
rbac.yaml, issued advisor_agent key. Forced-PEND job → pend_stall fired
on the real LLM (glm-5-3-flash, 19.4s, degraded=false) → 2 findings, all
evidence verbatim in the snapshot, unverified=false → SSE broadcast
captured on /xce/v1/events → MCP tool returned both records. Second PEND
job joining the stalled set fired a new decision within one round
(change-triggered semantics). Verification jobs killed; sanity job
submitted→ran→DONE; all daemons and 3 hosts healthy.
Design doc: architecture diagram gains the push branch; roadmap P1
marked done (CLI outlet not built).
Co-Authored-By: Claude Code <noreply@anthropic.com>
cJobToModel pre-filled main_reason_name with pendReasonName(main_reason),
so a detail-only record (main_reason 0, the allocator's ordinary shortfall
shape) arrived at rawJobToModel named "unknown(0)" — and the promotion that
derives the overall reason from the first detail row requires an EMPTY
name. The promotion's own tests passed because they build rawJobData
directly; the live C path never hit it. Found by the advisor P0
verification: jobs whose res_reasons named no_resource read
main_reason_name=unknown(0) over the API.
main_reason 0 now leaves the name empty for the promotion to fill; a
non-zero main reason keeps its mapped name unchanged.
Co-Authored-By: Claude Code <noreply@anthropic.com>
snapshot: list jobs with AllUsers + submit_time:desc + limit 200 — the
advisor must see every user's PEND jobs newest-first, or pend_stall sees
nothing on a shared cluster (the default list is the caller's own jobs,
oldest first).
detect: drop the per-scenario cooldown from the fire path. The original
cooldown-first order suppressed a CHANGED situation for up to 10 minutes —
on the live cluster a new job joining the stalled set went unreported
until the cooldown expired. Semantics now: an unchanged situation (same
dedup key) never re-evaluates; any changed situation fires immediately.
A thrashing situation (A→B→A) firing per change is honest reporting — a
flapping node or oscillating stalled set is real signal.
Verified on the t1/t2/t3 cluster: two forced-PEND jobs (unsatisfiable
select; 64-slot limit rejection) both diagnosed in the first scan, a
kill changing the stalled set re-triggered within one 30s round, and
identical situations stayed deduped.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two live-verification defects in the jobs-list path the advisor depends on:
1. The client never passed "sort", so the apiserver's default
unique_job_id:asc + limit returned the OLDEST jobs first — every
recent PEND job was invisible to any limited list. JobFilter.Sort now
forwards to the sort query param.
2. The apiserver's job list handler appends submit_user:eq:<caller> when
no submit_user filter is given (controllers/jobs.go), so even a
scope-all service key saw only its own jobs (0 rows on the live
cluster). JobFilter.AllUsers emits filter={submit_user:eq:all} to get
the cluster-wide view; an explicit Owner still wins.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Architecture rationale (LLM out of the hot path, perception/authority
split, uncalibrated confidence never gates), provider contract, detector
table, runbook-as-poor-man's-RAG, honest limitations of commodity LLMs,
and the P1/P2 path.
Appendix A records the TypeSafe Jev (System One) integration design for
future use only: single /v1/systemone endpoint, noul/choice/score
primitives, the pend_stall_v1 four-question pack and its Finding mapping,
confidence-gated act mode, jev-1.13 jaggedness risk. Deliberately not
wired in P0.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/agent: observe-mode loop — periodic scan + SSE job-event pull
(events only pull the next scan earlier, debounced 5s; job.finished
feeds exit_burst). Provider chain: primary (LLM) → echo fallback on
failure, recorded as degraded. VerifyEvidence firewall runs on every
LLM answer. Observe-only by construction: no write path reachable.
- internal/declog: JSONL decision log (scenario, trigger, provider, model,
latency, tokens, findings, raw output, state snapshot + digest,
degraded flag), size-rotated.
- internal/runbook: incident-pattern seed (embedded YAML) from the
cluster's real debugging history (bug-1494 stale config, IPC ABI skew,
bug-1513 rejoin crashes, bug-1521 reason race, first-fit bias,
dispatch-fail repend family), rendered into the provider prompt.
- cmd: advisor run (signal-handled observe loop) and advisor once
(single round, JSON report to stdout, no decision log).
- internal/config tests; full suite CGO_ENABLED=0 go test ./... clean.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/snapshot: compact cluster state via schedulerclient (nodes,
PEND top-N most-starved with parsed pending_reason, cluster sums, queue
garnish, recent exits). Field whitelists + stall-threshold filter keep
the state a few KiB. Module-local PEND-reason table mirrors
mcp-server/internal/pendreason.go (lift into schedulerclient is a P1
cleanup).
- internal/detect: pend_stall / worker_health / util_skew / exit_burst
triggers — threshold, then per-scenario cooldown, then dedup on a hash
of the situation (job set + reasons / node set) so an unchanged cluster
is never re-evaluated.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P0 of the scheduler advisor (observe-only LLM ops copilot). New standalone
Go module src/frontend/advisor (CGO_ENABLED=0, mcp-server layout, replaces
schedulerclient). No LLM executes anything: providers turn scenario
snapshots into structured Findings only.
- internal/config: env-driven config (XCE_API_URL/KEY, ADVISOR_LLM_*,
ADVISOR_INTERVAL, ADVISOR_LOG); LLM unset is valid (echo-only).
- internal/provider: Provider interface + Finding contract (kind/severity/
summary/evidence/confidence/suggestion). Confidence is display-only —
never calibrated for LLM backends, never gated on.
- echo: deterministic rules provider (fallback, test baseline, yardstick).
- openai: OpenAI-compatible chat-completions (GLM/Qwen/DeepSeek/local
vLLM), strict JSON response_format, enum-validated findings, bounded
429/5xx retries with backoff, 30s default timeout.
- VerifyEvidence: hallucination firewall — evidence elements must appear
verbatim in the state snapshot or the finding is demoted (unverified,
confidence 0.2) while staying in the log.
Jev (TypeSafe System One) deliberately not wired: recorded in the design
doc (later commit) as a future provider behind the same interface.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The per-job MIT-MAGIC-COOKIE-1 that unlocks an -X job's X11 forwarding
port rode every job reply that asked for it: any job:view caller could
read another user's cookie and connect straight to the worker's
forwarding port (6100-6199), bypassing the xattach relay's ownership
check. The SSE job stream carried it too, since its events embed the
full JobInfo row.
- apiserver: stripX11Cookie drops the cookie from GetJobsInfo,
GetJobByIDs and the SSE stream for every job the caller does not own
and holds no job:control grant over — the same rule XAttachJob
enforces. The row itself stays visible; only the secret goes.
- xcecli: preflightX11 checks DISPLAY, the Xauthority file and the
xauth binary before the job is submitted, so a misconfigured display
fails in one second instead of dispatching a job nobody can watch.
Verified end-to-end on t1: owner reads the cookie, a group mate with
job:view(self_ug)+job:control(self) gets the row with the cookie
stripped, an unrelated user is refused at the row level; controllers
tests cover the matrix plus the SSE path.
Co-Authored-By: Claude Code <noreply@anthropic.com>
