The sched_unit_test aggregate and 16 zero-#include standalone tests each
compiled all ~50 scheduler sources as their own TUs (~800 redundant
compiles of identical sources per clean build). Add a shared
sched_srcs_lib static archive (scheduler_srcs minus main.c) in the parent
meson.build; those 17 targets now link it and compile only their own test
TUs.
- Only extern-API consumers convert: these tests call nothing but symbols
declared in scheduler.h/core.h. Textual includers (white-box statics
access) keep their own compilation -- their TUs define the included
files' symbols and could collide with archive members; the 4
exclusion-list foreach blocks (snapshot_load_allocator_regression,
repl_trace_wiring, trace_error_paths, trace_bounds) are untouched.
- -Wl,--wrap= interposition is link-time symbol redirection and works
identically against archive members; the three --wrap tests
(publish_retry/backpressure/redelivery) are the sensitive probes and
pass.
- The aggregate keeps -Werror=incompatible-pointer-types and all baked
path defines; its src foreach is gone (the lib carries the sources).
Verified: fresh asan tree 1135 steps (was 1934), ninja green, meson test
134/134 including all converted targets; disk unchanged at ~1.3G after
the previous commit's capnp dedup.
Co-Authored-By: Claude Code <noreply@anthropic.com>
libxce_dep's sources:[gen_srcs] handed every consumer target the generated
.c output, so 143 targets each compiled controller.capnp.c independently:
a 9.5s / 293MB-RSS / 3.3MB-object TU under the asan profile, ~35% of a
clean build's serial work and ~470MB of duplicate objects on disk.
Index the dep's sources to gen_srcs[1] (the .h output) instead: the
custom_target edge still orders every consumer's compiles after codegen
(bug-1798's guarantee, unchanged), .h sources are never compiled, and the
.c is compiled exactly once inside libxce. libxce_src keeps the unindexed
custom_target for the same reason it always did.
Verified: fresh asan tree builds green (1934 steps vs 2076), meson test
134/134 including the three --wrap interposition tests; tree size 2.0G ->
1.3G.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Collapse the 28 zero-#include white-box scheduler tests (whose meson
executable blocks each duplicated the full scheduler_srcs list) into a
single sched_unit_test aggregate, following the controller
controller_test.c precedent: one executable, one process, one runner.
- sched_unit_runner.c: SUITES registry (28 entries) with per-suite
setUp/tearDown dispatch, -t <suite> filter, UnitySetTestFile per entry
- testsuite/meson.build: runner + 28 test TUs + every scheduler_srcs
entry except src/main.c as separate TUs; foreach over scheduler_srcs
so the aggregate can never drift from the production source list;
all baked-path defines (XCE_SCHED_*_PATH, XCE_SIM_*) move to the
aggregate's c_args; -Werror=incompatible-pointer-types carried
suite-wide (verified clean across all 28 tests + 50 src TUs)
- transformed tests drop main()/setUp()/tearDown() and local stubs that
now resolve against the aggregate's real src TUs
- test() registrations 126 -> 99 (126 - 28 + 1 aggregate); the three
--wrap= interposition tests (decision publish retry, backpressure,
decision redelivery) keep their standalone executables unchanged
- isolation libs (sched_core_isolation_lib, sched_sim_isolation_lib)
stay build-only and are wired as test()-level depends; the nm(1)
text-scan tests find them via XCE_CORE_LIB_PATH/XCE_SIM_LIB_PATH
Verified: meson test --suite scheduler 100/100 green x3 consecutive
runs (43s total, aggregate 3.3s); /tmp/xce_* sweep clean (the one
leftover is test_worker_launch_type_dispatch.c's by-design fixed-path
fake xrun launcher).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Deduped census insight: the earlier "730 warnings" figure counted per-TU
echoes of the same source line (white-box textual inclusion recompiles
scheduler.c/limit.c/etc. dozens of times, each copy re-reporting). The
distinct warning sites in our code were 24 unused-variable + 11
unused-function + 2 Wcomment. This commit removes all of them; a clean
full rebuild now reports zero warnings outside the format-truncation
family (125 sites, deferred to its own pass) and vendored capnpc-c.
Notable non-mechanical fixes (the rest is dead-code deletion):
- schedobj.c make_sched_obj() silently swallowed its sub-constructor's
rc (ENOMEM/-EINVAL) and returned 1 unconditionally -- a half-built
schedobj with a NULL union payload could escape to callers. It now
propagates the failure, and make_sched_job() frees and returns NULL
on it instead of continuing with a broken schedobj.
- scheduler.c make_allocs(): vget() rc was set-but-unused; the r==NULL
skip below is the intended handling, so the call is now (void)'ed with
a comment saying so.
- commit.c highest_appended_lsn(): kept but scoped #ifdef XCE_TESTING --
it backs the never-beyond-the-log assert; it had no production caller.
- limit.c get_vres_value(): deleted -- its only call site sits inside a
#if 0 block and its own body was #if 0'ed to a constant.
- group.c update_group_dirty(): deleted recursive dead subtree, no
callers outside itself.
- test_sim_reconcile_unaccounted.c: test_reconcile_respects_host_
capacity_under_contention() was never registered in RUN_TEST; removed
with an NB pointing at the registered bug-1836 coverage.
Verified: clean full rebuild = 0 unused/Wcomment/implicit warnings in
our code (89 unique sites remain, all -Wformat-truncation or vendored);
controller_unit_test 1342/1342, xcelib_test 758/758, 126 scheduler
suites and 18 worker suites all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Warning fixes (from a 730-warning full-build census; these were the
semantically risky ones):
- test_repl_majority.c:272: remove a broken "field-existence" loop whose
condition (i < ep != NULL) parsed wrong (-Wparentheses); the direct
pending_ack_count write it guarded already exists and still fails the
build if the field disappears.
- ha.c:8593: ha_rdhup_election_hook now takes const xce_candidate_t**
so RDHUP election can hand over candidates straight from the const
peers_cfg without stripping const. All 5 test-side hook impls only
(void)peers -- signature updated in ha.h + 3 test files.
- ha.c:726/872/10804 endpoint_connect: constify the host parameter of
endpoint_connect/socket_connect (netio.h/netio.c). The functions only
pass host to gethostbyname(); peer->hostname etc. can now flow in
without discarding const. All callers are read-only and unaffected.
Debug profile (src/meson.build): drop -gdwarf-4 -g3
-fvar-tracking(-assignments) in favor of -ggdb -g1. Debug objects were
~16x larger with full DWARF (scheduler.c: 1.3MB vs 0.08MB), pushing the
build tree to 3.9G; at -g1 it is 850M and a clean build drops from
6m43s to 4m36s. -g1 keeps file/line and backtraces; gdb variable/
macro inspection needs the asan profile. -fno-omit-frame-pointer kept.
Verified: controller_unit_test 1342/1342, xcelib_test 758/758, all 126
scheduler suites + 18 worker suites green; modified sources compile
with zero warnings.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Section 3 documented the retired nix path (nix.conf edits, nix build
.#xcecore, AppImage, nix develop self-wrap). Replace it with the actual
build as it now works on plain Ubuntu:
- 3.1 one-shot apt install (toolchain + 11 C libraries), pointing at
DEPS.md for the versioned dependency table and the wrap-subproject
source closure for everything apt does not carry
- 3.3 daily development via tools/xce-build.sh (profiles, --incremental
caveat), plus the plain meson commands it wraps; Go vendor refresh
without any special shell
- 3.4 tarball packaging via tools/xce-package.sh + install.sh (the
shipping distribution form, replacing the AppImage section)
- 3.5 scripts/ci-ubuntu.sh as the CI entry point
Co-Authored-By: Claude Code <noreply@anthropic.com>
ttyd is the interactive-job launcher for worker TTY/DISPLAY modes: the
worker resolves it next to its own binary (inter_job_utils.c
readlink(/proc/self/exe) + /ttyd, /opt/xce/bin/ttyd fallback), so a
tarball without bin/ttyd silently loses -I/--tty on the target host.
The packager now picks it up from $BUILD_DIR/worker/ttyd first, then
/opt/xce/bin/ttyd, and warns loudly if neither exists.
NB it is NOT an upstream release: this is rshen's modified ttyd (version
string 1.7.7-eccebc6), and the binary we bundle today is that same
modified build. Source lands in-tree later and moves to a meson target;
the packager already prefers a build-dir copy for that day.
DEPS.md: document the ttyd situation (including the MIT-compatibility
audit needed before commercial distribution) and record the user-flagged
license note on bundling libhwloc/libedit/libtirpc -- fine for the
development/test rollout, needs a per-item license audit (bundled
LICENSE file at minimum) before commercial release.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.
Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.
Also:
- test_apply_committed_entry: three new tests covering the skip for all
three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
command. capnpc-c deliberately never frees its compile-time structures
(one-shot tool); under the de-nix asan profile the wrap-built generator
carries -fsanitize=address and LSAN's end-of-process check failed the
generation step. The legacy nix build never propagated -fsanitize to
the subproject. Same exemption convention as every other one-shot/
never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
even though the build host resolves them from /usr/lib.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).
Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
subprojects under src/subprojects/ (source closure, ~500 files): the
upstream trees plus packagefiles/ overlays and diff_files patches
registering dependency/program overrides the nix .pc files used to
provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
Fresh-clone reproducibility verified: wiped trees + `meson subprojects
download` reproduce the committed state exactly (fixed librtti.wrap's
dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.
Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
the dangling build RPATH makes ldd drop the very lines the naive filter
needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
(kernel does not parse quotes in shebangs), ships systemd unit samples +
install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
both bin/ wrapper and venv script.
Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
loopback port so the CLI-error tests assert the no-controller failure
path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
sig_handler re-raises received signals via kill(0, sig) -- process GROUP
wide (strace -k evidence) -- and the child previously shared the
harness's pgrp, so an interrupted SSL handshake killed the whole suite
(exit 130).
Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).
Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes from the MPI parallel-job path review, each unit-tested in the
launch_type_dispatch suite (5 -> 8 tests):
B1 (controller_connector.c): handle_startjob's fan-out-failure path reported
xcecd_task_fail(jobid, body.runtaskmsg->clientidx, ...). In the startjob
request the body union arm is jobstartmsg, so runtaskmsg->clientidx aliases
the same storage as the start_info pointer's high half -- uninitialized
garbage passed to the controller. Pass -1 (the honest value; this start has
no client-side task index, job_ctx->clientidx is -1 downstream). The same
path also fell through to the local start after a failed fan-out; it now
returns the error so a job missing its remote legs is not reported healthy.
B2 (controller_connector.c): xcecd_job_remote_start leaked the taskhost
strings on every remote start -- xcecli_runtask serializes the task
synchronously but takes no ownership, so the caller must free. Add owned
copies for cmdline/host and free all seven heap copies after the call.
Verified clean under ASan with detect_leaks=1.
B3 (xrun.c): the mpirun plm_rsh_agent shim built full_command[4096] with
unbounded strcat from mpirun-passed argv -- a stack smash on any command
line over MAXLINELEN. Bounded append with quoting; overflow is rejected
with rc=1 (verified at the exact 4096-byte boundary).
Co-Authored-By: Claude Code <noreply@anthropic.com>
type=array_stats (xce jobs list -A) read store.ArrayJobs, which has no
production writer, so it always answered zero rows. Stats are now derived
at query time by scanning store.Jobs and aggregating per array base ID
(controllers/arraystats.go), mirroring hosts.go's computeHostSlotUsage
precedent: counters can never drift from the element rows.
- GetJobsInfo/GetJobByIDs array_stats branches run the derived rows
through the same ExecuteQuery filter/sort/paginate pipeline as store
queries; unique_job_id rides in the field list so processJobData can
annotate job_id_human (SelectFields drops unlisted fields).
- processJobData: new rowInt64 helper (uint64 bit-preserved + float64/
json.Number tolerant) replaces brittle bare type assertions for
unique_job_id/array_idx reads; dead code removed (StackLimit
wrong-source copy, unreachable index error).
- QueryData(TableArrayJob) now returns a guard error instead of always
answering zero rows from the never-written store.
- xcecli: ArrayJobStatsData got UnmarshalJSON via UnmarshalOrderMapRow —
the server's OrderMap wrapper ({Fields,Data}) decoded to all-zero
fields, which is why table mode showed 0/- rows against live arrays.
- analysis.md CL5 marked fixed with the real root cause.
Cluster-verified on t1/t2/t3: submit→PEND→RUN→DONE per element, kill
path, job_id filter, and per-status counters all live; all daemons
healthy. Unit suites: apiserver + xcecli green (CGO_ENABLED=0).
Co-Authored-By: Claude Code <noreply@anthropic.com>
API keys were a pure in-memory map (store/apikey.go) — every apiserver
restart silently revoked every service-account key (advisor, MCP
server) until re-issued, with only an inexplicable 401 to show for it.
Keys now persist to <top>/state/apikeys.json: hash → metadata only
(never the raw key, preserving the existing "a dump cannot leak usable
credentials" property), written atomically (temp file + fsync + rename,
0600) on each issuance, loaded at InitStores. The load also restores
the user→role mapping, without which a reloaded key would authenticate
but be denied every permission check. A missing file is a normal fresh
install; a corrupt one fails fast at startup instead of quietly
starting with zero keys. A failed save fails the issuance request —
the raw key is shown exactly once, so we must not hand out a key we
could not persist.
The memory-only path (no SetAPIKeyStatePath) is kept for unit tests.
Cluster-verified on t1-t3: issue key → restart apiserver → key still
authenticates (hosts OK, advisor:report POST published 1) — previously
impossible. Verify job submitted, ran, exited; all daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Case 2b documents `advisor compare` with the live-observed echo-vs-LLM
table (echo 0ms unverified "unknown" vs deepseek ~1.5s severe
cross-field root cause). Case 2 notes deepseek-v4-flash as the
preferred model after glm timed out twice back-to-back on
2026-10-20; pitfalls table updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
One snapshot through BOTH providers per fired scenario, side by side:
CompareRun rows (scenario, trigger, per-side model/latency/tokens/
findings/unverified). The LLM side errors are reported in-band
(llm_error) rather than degrading to echo, so a flaky router shows up
as data instead of a silent echo-only row. The compare path is pure
measurement: nothing logged, nothing pushed — the run stays the only
writer of the decision log.
Live data (pend_stall, cmp job on t1-t3 cluster):
echo 0ms 1 finding, unverified (reason "unknown" opaque)
ds-v4 ~1.6s cross-field catch: job PEND 876-902s while all
nodes idle + no dispatch -> severe, actionable
Echo answers in 0ms with a coarse degraded verdict; the LLM leg adds
cross-field correlation (pend vs host state vs dispatch) that the
static rules structurally cannot express. This is the quantified
incremental-value page for the advisor pitch.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor now installs to ~/bin/xce-advisor (survives /tmp cleanup and
reboots) and is managed by ~/bin/advisor-restart.sh: --check (status +
last log line), --stop, bare start (reads /tmp/advisor_apikey_v2, warns
if the apiserver is unreachable, self-checks liveness ~10s after launch,
refuses to double-start). Handbook pre-flight step 3, the pitfall table
row, and the design doc deployment line now reference the script instead
of a hand-rolled env nohup command.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The dedup disarm fix (e97f6b16) makes close → open → close fire on every
outage, so the handbook's caveat is replaced with a repeatable demo beat
(verified cycle: 11s fire / silent recovery / 22s re-fire).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Live-cluster gap found 2026-10-20 (t1/t2/t3): t3 went down, recovered,
went down again — and the second outage never reported. The dedup key
survived the healthy rounds, so the recurring situation looked
"unchanged" forever. Same hazard for pend_stall: a stall that empties
and later re-forms identically stays silent.
Fix: fire/clear pairing. When a detector's condition is fully absent
(all nodes healthy, no stalled jobs, no skew), it clears its scenario's
dedup key; the next occurrence is a fresh situation and fires. Unit
tests pin the outage→recovery→identical-outage cycle for worker_health
and the stall→resolve→re-form cycle for pend_stall.
exit_burst deliberately does NOT clear: RecentExits is a per-scan
window (consumed once), so an empty window is the steady state, not a
resolution — clearing there would re-fire bursts whose situation never
changed.
Cluster-verified cycle: t3 close → fired (glm, 11.1s); open → silent;
close again → fired again (22.1s, 2 findings). All detect tests green;
advisor restarted on the fix.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Four demo scenarios, all live-verified on the test cluster (2026-10-19/20):
1. pend_stall diagnosis (most reliable; includes the change-triggered
second-fire follow-up demo)
2. real-LLM cross-field inconsistency catch (glm queue-accounting finding,
reproduced twice)
3. worker_health node-down detection (fastest; documents the repeat-outage
dedup gap found while verifying: lastKey is never cleared on recovery,
so a second identical outage stays silent — one-line fix candidate)
4. MCP scheduler_advisor_findings query (zero-risk closer, auditability)
Plus pre-flight checks, advisor start command, known pitfalls table
(in-memory API keys die on apiserver restart, glm timeouts degrade to
echo, SSE listener timeout vs 5-min stall threshold), and cleanup
checklist.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The advisor's findings previously lived only in its JSONL decision log on
its own host. P1 adds two observe-only outlets so diagnoses reach humans
and other agents live:
apiserver side:
- backend/broker.go: EventAdvisorFinding ("advisor.finding") event type +
AdvisorFinding struct + Event.AdvisorFindings field.
- controllers/events.go: ReportFindings — POST /xce/v1/advisor/findings
(generated_at/scenario/findings), publishes one broker event. Broadcast
only: the decision log remains the store of record.
- router/routers.go: route behind RequirePermission("advisor:report").
- middleware/whitelist.go: admit /xce/v1/advisor/findings in the POST URL
whitelist (the URL middleware 403s before the router's RBAC otherwise —
same shape as bug-1489).
- config/rbac.yaml: advisor_agent role (advisor:report + the six *:view
grants the snapshot needs) and advisor:report in the permission
whitelist. mcp_readonly stays strictly *:view. New
TestRBACYAML_AdvisorAgentRole_PinsReportSurface pins the role surface.
advisor side:
- internal/push: Pusher — best-effort POST of each round's findings, one
retry, nil/unconfigured/empty findings are no-ops; failures log and
never affect the observe loop.
- internal/agent: SetPusher wired after the decision-log append (the log
is authoritative; push is a fan-out).
- internal/config + cmd: push default-on to the same XCE_API_URL with the
same XCE_API_KEY (the key's role needs advisor:report); ADVISOR_PUSH=
off|false|0 disables.
mcp-server side:
- internal/advisor.go: lenient JSONL reader for the advisor's decision
log (ADVISOR_LOG env), tail-bounded, corrupt lines skipped.
- scheduler_advisor_findings tool: recent records with parsed findings;
friendly errors when the log is missing/unconfigured. Passive reader —
never talks to the advisor process.
Cluster verification (t1/t2/t3): redeployed apiserver closure, deployed
rbac.yaml, issued advisor_agent key. Forced-PEND job → pend_stall fired
on the real LLM (glm-5-3-flash, 19.4s, degraded=false) → 2 findings, all
evidence verbatim in the snapshot, unverified=false → SSE broadcast
captured on /xce/v1/events → MCP tool returned both records. Second PEND
job joining the stalled set fired a new decision within one round
(change-triggered semantics). Verification jobs killed; sanity job
submitted→ran→DONE; all daemons and 3 hosts healthy.
Design doc: architecture diagram gains the push branch; roadmap P1
marked done (CLI outlet not built).
Co-Authored-By: Claude Code <noreply@anthropic.com>
cJobToModel pre-filled main_reason_name with pendReasonName(main_reason),
so a detail-only record (main_reason 0, the allocator's ordinary shortfall
shape) arrived at rawJobToModel named "unknown(0)" — and the promotion that
derives the overall reason from the first detail row requires an EMPTY
name. The promotion's own tests passed because they build rawJobData
directly; the live C path never hit it. Found by the advisor P0
verification: jobs whose res_reasons named no_resource read
main_reason_name=unknown(0) over the API.
main_reason 0 now leaves the name empty for the promotion to fill; a
non-zero main reason keeps its mapped name unchanged.
Co-Authored-By: Claude Code <noreply@anthropic.com>
snapshot: list jobs with AllUsers + submit_time:desc + limit 200 — the
advisor must see every user's PEND jobs newest-first, or pend_stall sees
nothing on a shared cluster (the default list is the caller's own jobs,
oldest first).
detect: drop the per-scenario cooldown from the fire path. The original
cooldown-first order suppressed a CHANGED situation for up to 10 minutes —
on the live cluster a new job joining the stalled set went unreported
until the cooldown expired. Semantics now: an unchanged situation (same
dedup key) never re-evaluates; any changed situation fires immediately.
A thrashing situation (A→B→A) firing per change is honest reporting — a
flapping node or oscillating stalled set is real signal.
Verified on the t1/t2/t3 cluster: two forced-PEND jobs (unsatisfiable
select; 64-slot limit rejection) both diagnosed in the first scan, a
kill changing the stalled set re-triggered within one 30s round, and
identical situations stayed deduped.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two live-verification defects in the jobs-list path the advisor depends on:
1. The client never passed "sort", so the apiserver's default
unique_job_id:asc + limit returned the OLDEST jobs first — every
recent PEND job was invisible to any limited list. JobFilter.Sort now
forwards to the sort query param.
2. The apiserver's job list handler appends submit_user:eq:<caller> when
no submit_user filter is given (controllers/jobs.go), so even a
scope-all service key saw only its own jobs (0 rows on the live
cluster). JobFilter.AllUsers emits filter={submit_user:eq:all} to get
the cluster-wide view; an explicit Owner still wins.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Architecture rationale (LLM out of the hot path, perception/authority
split, uncalibrated confidence never gates), provider contract, detector
table, runbook-as-poor-man's-RAG, honest limitations of commodity LLMs,
and the P1/P2 path.
Appendix A records the TypeSafe Jev (System One) integration design for
future use only: single /v1/systemone endpoint, noul/choice/score
primitives, the pend_stall_v1 four-question pack and its Finding mapping,
confidence-gated act mode, jev-1.13 jaggedness risk. Deliberately not
wired in P0.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/agent: observe-mode loop — periodic scan + SSE job-event pull
(events only pull the next scan earlier, debounced 5s; job.finished
feeds exit_burst). Provider chain: primary (LLM) → echo fallback on
failure, recorded as degraded. VerifyEvidence firewall runs on every
LLM answer. Observe-only by construction: no write path reachable.
- internal/declog: JSONL decision log (scenario, trigger, provider, model,
latency, tokens, findings, raw output, state snapshot + digest,
degraded flag), size-rotated.
- internal/runbook: incident-pattern seed (embedded YAML) from the
cluster's real debugging history (bug-1494 stale config, IPC ABI skew,
bug-1513 rejoin crashes, bug-1521 reason race, first-fit bias,
dispatch-fail repend family), rendered into the provider prompt.
- cmd: advisor run (signal-handled observe loop) and advisor once
(single round, JSON report to stdout, no decision log).
- internal/config tests; full suite CGO_ENABLED=0 go test ./... clean.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- internal/snapshot: compact cluster state via schedulerclient (nodes,
PEND top-N most-starved with parsed pending_reason, cluster sums, queue
garnish, recent exits). Field whitelists + stall-threshold filter keep
the state a few KiB. Module-local PEND-reason table mirrors
mcp-server/internal/pendreason.go (lift into schedulerclient is a P1
cleanup).
- internal/detect: pend_stall / worker_health / util_skew / exit_burst
triggers — threshold, then per-scenario cooldown, then dedup on a hash
of the situation (job set + reasons / node set) so an unchanged cluster
is never re-evaluated.
Co-Authored-By: Claude Code <noreply@anthropic.com>
P0 of the scheduler advisor (observe-only LLM ops copilot). New standalone
Go module src/frontend/advisor (CGO_ENABLED=0, mcp-server layout, replaces
schedulerclient). No LLM executes anything: providers turn scenario
snapshots into structured Findings only.
- internal/config: env-driven config (XCE_API_URL/KEY, ADVISOR_LLM_*,
ADVISOR_INTERVAL, ADVISOR_LOG); LLM unset is valid (echo-only).
- internal/provider: Provider interface + Finding contract (kind/severity/
summary/evidence/confidence/suggestion). Confidence is display-only —
never calibrated for LLM backends, never gated on.
- echo: deterministic rules provider (fallback, test baseline, yardstick).
- openai: OpenAI-compatible chat-completions (GLM/Qwen/DeepSeek/local
vLLM), strict JSON response_format, enum-validated findings, bounded
429/5xx retries with backoff, 30s default timeout.
- VerifyEvidence: hallucination firewall — evidence elements must appear
verbatim in the state snapshot or the finding is demoted (unverified,
confidence 0.2) while staying in the log.
Jev (TypeSafe System One) deliberately not wired: recorded in the design
doc (later commit) as a future provider behind the same interface.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The per-job MIT-MAGIC-COOKIE-1 that unlocks an -X job's X11 forwarding
port rode every job reply that asked for it: any job:view caller could
read another user's cookie and connect straight to the worker's
forwarding port (6100-6199), bypassing the xattach relay's ownership
check. The SSE job stream carried it too, since its events embed the
full JobInfo row.
- apiserver: stripX11Cookie drops the cookie from GetJobsInfo,
GetJobByIDs and the SSE stream for every job the caller does not own
and holds no job:control grant over — the same rule XAttachJob
enforces. The row itself stays visible; only the secret goes.
- xcecli: preflightX11 checks DISPLAY, the Xauthority file and the
xauth binary before the job is submitted, so a misconfigured display
fails in one second instead of dispatching a job nobody can watch.
Verified end-to-end on t1: owner reads the cookie, a group mate with
job:view(self_ug)+job:control(self) gets the row with the cookie
stripped, an unrelated user is refused at the row level; controllers
tests cover the matrix plus the SSE path.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Render -X jobs' windows on the user's own X server with zero client
install, mirroring the ssh -X experience: the job command starts with
DISPLAY pointed at a per-job TCP listener on the worker, and the user's
xcecli relays that listener's byte stream to their local X server.
- xcecli: new -X/--x11 flag (interactive=2 + exdata "x11":1). After
submit it fetches the per-job MIT-MAGIC-COOKIE-1, registers it in the
user's Xauthority, then bridges one xattach websocket to the local X
server address parsed from $DISPLAY (unix socket or TCP). The cookie
entry is removed when the session ends. Pre-flight checks report
missing DISPLAY/xauth before submit.
- apiserver: GET /xce/v1/jobs/:id/xattach relays websocket binary frames
to the worker's forwarding port (raw TCP, no ttyd). Close codes match
the ttyd attach (4000 unreachable, 4001 ended). The job-finished event
ends the relay. Teardown closes both hops as soon as either pump
exits, so a client disconnect propagates instead of deadlocking in a
blocking read.
- worker: XJOB jobs with exdata x11=1 skip VNC, listen on 6100-6199,
generate a per-job xauth file (/tmp/job_<id>_xauth, worker-side since
the worker has no session), export DISPLAY/XAUTHORITY to the command,
and bridge the app connection (loopback) with the relay connection
(non-loopback; roles classified by source address, not arrival order).
A 60 s no-client watchdog reaps jobs nothing ever attached to; the
v1 single-connection boundary refuses a second X connection loudly.
t1-verified per the plan matrix: window rendering (xclock/xeyes on
Xvfb), normal end -> DONE + xauth removed, no-client 60 s reap (137),
kill -> immediate teardown with no residue, -I/-I --browse regression,
daemon health, missing-DISPLAY error, second-connection refusal.
(--xjob VNC on t1 fails for lack of a vncserver binary; pre-existing.)
Co-Authored-By: Claude Code <noreply@anthropic.com>
An attach job (xcecli -I without --browse) that no client ever connects
to sat in RUN forever: ttyd only starts the job command on the first
websocket client, so with no client the command never runs, ttyd never
exits, and the job holds its ttyd, port and per-job worker indefinitely.
New exdata key "attach":1, set by the CLI for TTY jobs that are not
--browse and carried through the apiserver's extendDataMap to the
worker's transfer_info->attach_mode. For such jobs the worker starts a
1 s watchdog when the job process starts:
- no client attached within 60 s -> reap ttyd (SIGKILL to the monitored
tree), job EXITs with status message "No client attached within 60 s"
- client came and the command ran and exited -> give ttyd 5 s to stop
on its own (-Q), then reap: "Command exited; ttyd did not stop on its
own"
The reap lands in the regular process_exit_callback path (JOB_EXITED,
status reason, post-exec, worker exit), so kill and cleanup behave like
any other job termination.
Field-critical detail: uv_proc_get_pid returns the "/bin/sh -c ttyd ..."
wrapper pid, and sh does not reliably exec -- on the field ttyd runs as
a child of the wrapper. Judging the wrapper's children would see ttyd
itself as "the command has started" and never reap. The watchdog
resolves the real ttyd pid first (/proc/<pid>/comm walk down through
the sh wrapper, capped depth) and judges client activity by the real
ttyd's children; killing still goes through the monitored wrapper tree
so both topologies (exec'd or forked) are covered.
Web-viewed sessions (--browse / DISPLAY) are untouched: no attach
exdata, no watchdog -- ttyd stays until a browser connects or the job
is killed, as before.
Attach failure on the CLI side prints that the job will be cleaned up
by the worker if no client attaches within 60 s.
Tests: xcecli (attach flag matrix), apiserver controllers (attach
exdata end-to-end in ExtendData), worker native suites; live t1
cluster: normal attach completes DONE with no residue, unattached job
EXITs at ~60 s with the reason in the worker log and port released,
Ctrl-C-then-wait completes when the command ends, --browse job stays
RUN past 60 s and kills cleanly.
Co-Authored-By: Claude Code <noreply@anthropic.com>
waitForInteractiveURL polled PEND every 5s, adding ~4.4s of dead time
to every interactive submit (the job typically reaches RUN with its
session URL ~0.6s after submit once the apiserver applies the
controller's jobPidInfo push), and the RUN-but-no-URL-yet round waited
another 2s. Front-load fast 500ms polls (10 rounds) so a normally
dispatched job attaches within ~1s, then back off to 2s so a job stuck
in a long queue costs the server a query every couple of seconds
instead of a permanent 2 qps; the RUN race polls 500ms too.
checkJobStatus also failed the wait outright on any state other than
PEND/RUN: treat PSUSP like PEND (still queued), and USUSP/SSUSP/
TRANSIENT as keep-waiting states — the allocation is held, a resume
(or failover settle) may follow, and tearing down the attach on a
transient state is wrong.
Cluster-verified on t1: submit-to-attach-attempt 0.56-1.08s across
three runs (previously ~6s even after the jobPidInfo fix), full
submit/run/finish lifecycle clean, daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
handleNotifyData had no case for NotifyData_data_jobPidInfo, so the
controller's start-report push (apiserver_job_start_fun, fired from
HOOK_DETAIL_JOB_START the moment WAL_JOB_START commits) was silently
dropped. interactive_job_url then stayed invisible until the next
periodic rusage-driven job UPDATE re-delivered the full row — up to
10s later (job_rusage_collect_interval), the dominant chunk of an
-I --tty job's submit-to-first-output latency (measured ~6s on t1,
server-side work itself is ~0.6s).
Add OnJobPidInfo to the notification interface and DefaultHandler:
it merges the worker-reported pids/pgids and the exdata {"url": ...}
onto the already-stored job record (get-mutate-put, mirroring
OnJobFinished) and publishes a job.updated event. The CGO decode
side is factored into cPidInfoToRaw/pidInfoJSONB, shared with the
existing xce_job_t conversion.
Cluster-verified on t1: URL is pollable ~0.6s after submit
(previously ~11s), full submit/run/finish lifecycle clean, all
daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three attach-mode (-I) fixes, verified end-to-end on the t1 cluster:
1. Session hang on job finish (job_attach.go): the relay's client->worker
pump blocked forever in ReadMessage after the worker hop closed, so the
old wg.Wait()-before-Close teardown never ran and the client hung, then
died with an abnormal 1006. A broker watcher now subscribes to
EventJobFinished and on job end sends the 4001 policy close and closes
both hops, unblocking the pumps; the teardown follows normally. The
client (attach.go) treats 4001/1000/1001 -- and 1006-after-output as a
final safety net -- as a clean session end.
2. "<<Starting on t1 >>" printed repeatedly (submit.go): the status poll
printed the banner on every transition; now gated on a reportedStart
flag. The URL is also no longer printed in attach mode -- it only makes
sense for browse (-I --tty --browse) and DISPLAY jobs.
3. ttyd left behind after an attach job exits (xcecd_worker.c): a new
tty_stop_attach_session(), called from process_exit_callback for TTY
jobs, kills the per-job ttyd right away. This is insurance -- the sh -c
exec means uv_proc monitors ttyd's own pid, so ttyd is normally gone
before the callback fires -- but it guarantees no orphan when the
timing lines up badly. Web-viewed sessions are untouched: they keep
ttyd and output alive for later browsing, per design.
Co-Authored-By: Claude Code <noreply@anthropic.com>
-I without --xjob now means -I --tty: the submit attaches the interactive
session right in the calling terminal through the apiserver relay. The
previous bare -I mapping (interactive=3, readonly DISPLAY web view) had no
interactive input and gave the user no hint; --browse still opens the web
URL for TTY jobs and --xjob keeps the VNC path. The API's interactive=3
semantics are unchanged -- only this CLI's default remaps.
Help text updated: -I/--tty/--xjob descriptions and the long-help examples.
Co-Authored-By: Claude Code <noreply@anthropic.com>
xjob_start_vnc's four failure points (XCE_SERVER_DIR unset, pre-exec spawn
failure, DISPLAY export failure, port-wait timeout) called exit(1), taking
down the whole per-job worker with no user-visible reason and leaving the
job RUNNING forever (the master never learned of the death until its
instance reap path). It now returns int; the worker_job_callback call site
reports the standard failed-start pattern -- JOB_FAILED + status_message
"Failed to start VNC server (check worker log)" + worker_exit_flag so the
master reaps the instance and starts a fresh one for the next job.
xjob_stop_vnc is cleanup-only: its failures now log and return instead of
exit(1) -- a failed stop must not kill the worker or clobber the job's
exit code.
generate_url sized its snprintf with CMD_MAX_LENGTH (4096) while the url
buffer is URL_MAX_LENGTH (256). Latent since import: the old build's
non-fortified snprintf silently truncated, but any _FORTIFY_SOURCE build
aborts with SIGABRT (__chk_fail) on the FIRST interactive job -- found via
core dump after t1 bare -I jobs died ~1s after start ("*** buffer overflow
detected ***" in generate_url). Size both snprintfs with URL_MAX_LENGTH.
Verified on t1: -I --xjob on a host without vncserver fails in ~1s with
the reason in worker/master logs, worker exits cleanly, next batch job
completes; bare -I submits, dispatches, attaches via the relay and runs
until killed; daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Step 2 of the interactive-job UX work: `-I --tty` now attaches the session
to the submitting terminal by default instead of opening a browser.
Producer/consumer shape (user clusters cannot reach worker ttyd ports):
xcecli (-I --tty, raw terminal)
<-> GET /xce/v1/jobs/:id/attach (JWT auth + job:control ownership)
<-> ws://worker:ttyd-port/ws (ttyd)
apiserver (controllers/job_attach.go):
- new websocket relay endpoint, route gated by RequirePermission("job:control");
owner may always attach, others need a job:control scope covering the owner
- JWT accepted from the Authorization header or ?token= query param
(browser WebSocket API cannot set headers)
- bidirectional frame pump with policy close codes: 4000 = cannot reach the
session on the worker, 4001 = session ended
- the worker hop dials with the "tty" websocket subprotocol: ttyd 1.7.7
completes the handshake without it but then stays silent (verified against
the deployed binary)
xcecli (cmd/jobs/attach.go):
- attach client: dials the relay, raw mode via golang.org/x/term, ttyd
wire format (TEXT hello {"AuthToken",columns,rows}; binary frames
'0'+input, '1'+resize JSON, output '0' to stdout), SIGWINCH resize
forwarding, terminal restored on every exit path
- submit.go: --browse flag opens the web URL instead of attaching (TTY jobs
attach by default; XJOB/VNC keeps the browser path); the URL line is
always printed as a record
- waitForInteractiveURL no longer fails when the job is RUN but the worker's
start report (carrying the URL) has not landed yet -- poll and retry
Verified on t1: raw-client and real-xcecli attach round trips through the
relay (echo both ways), kill -> EXIT with URL retained, batch jobs unaffected,
all daemons healthy. Suites: meson 'apiserver backend' 1/1, CGO_ENABLED=0
controllers/router and xcecli tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The worker packs the interactive session URL (ttyd/XJOB) into the start
report's exdata as {"url": ...}; the controller stores it on
job->job.pidinfo->exdata and the capnp Job wire carries pidinfo (pidinfo@9,
JobPIDInfo.exdata@3) — but apiserver's cJobToModel never mapped pidinfo, so
models.JobInfo.InteractiveJobURL (declared only in the query-field whitelist
JobAdditionalFields) had no producer and interactive_job_url was permanently
empty in API responses. xcecli's waitForInteractiveURL/openBrowser flow could
never fire.
- models.JobInfo: add InteractiveJobURL (json interactive_job_url, matching
the JobAdditionalFields whitelist name).
- convert_pure.go: rawPidInfo (pids/pgids/exdata) + interactiveURLFromExdata;
rawJobToModel marshals pids/pgids into PIDInfo (pid_info is a JSONBJFields
entry, parsed by processJobData) and promotes the exdata URL to
InteractiveJobURL.
- convert.go: cJobToModel extracts job.pidinfo via pidinfo_pid/pidinfo_pgid
cgo helpers (CGO cannot index the C arrays directly).
Cluster-verified end to end on t1 (fresh apiserver deployed): tty job
(xcecli -I --tty equivalent, exdata {"interactive":1}) now returns
interactive_job_url=http://<worker>:9000 and a populated pid_info; non-tty
jobs return ""/no pid_info. All daemons healthy, zero errors in window.
Co-Authored-By: Claude Code <noreply@anthropic.com>
test_sim_reconcile_unaccounted.c captures stdout through mkstemp() files in
/tmp/xce_simreconcile_*, but only the shared capture_stdout_around() helper
unlinked its file after reading it back -- the 4 inlined capture sites left
theirs behind on every passing run (the mkdtemp shim does not intercept
mkstemp, so nothing else removed them). Add the same close(fd); unlink(tmp)
the file family's other capture helpers use to each of the 4 sites.
Verified: scheduler sim reconcile unaccounted test passes and leaves no
/tmp/xce_simreconcile_* files behind.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
A job without a project and without a group default goes to the cluster
default project, the built-in `default` (members all) unless configured
otherwise; it is never rejected for lacking a project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A share may be owned at a location (host group) as well as anywhere.
Per resource, located host groups and the rest of the cluster form
disjoint pools; usage is charged to the host's pool and every pool follows
the share rules on its own, so located ownership is real. Pool capacity
counts registered hosts including down and closed ones. No-default
submits list the user's projects; a per-user default project is a
client-side xcecli setting; members see their ancestors' usage, not
siblings' details.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- `jobs` is a built-in resource for shares and member quotas: the job
limits per project and user; pending caps go to pending-job-limits.
- Jobs may be charged to inner projects, with a warning and pend reason
when their unassigned part is empty.
- Project aliases keep jobs through renames; running jobs follow them.
- Changing a pending job's project is part of job-modify.
- New project-accounting brief: durable finished-job records, outside the
WAL, before controller-job-cleanup drops a job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fairshare weights and usage history live on the project tree; queues keep
priority between queues and only choose fairshare or FCFS ordering. The
per-queue `fairshare:` user lists in queues.yaml go away.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A project's `share` (absolute or a percentage of its parent's) is carved
from the parent; siblings never add up to more than the parent. v1 is
strict: a project holds at most its share. A project's own jobs and its
children without a share draw from its unassigned part. Borrowing idle
shares above a project's own (`max`) and reclaim are future work, in the
renamed project-borrowing brief.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Revocation and expiry close live connections authenticated with the key,
since authentication happens only at join. Lists the design questions:
where the trusted list is changed, keeping revoked keys revoked, bootstrap
keys, token registration for elastic hosts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Per-host identity without a CA; only public keys move, and the trusted
list can ride on the replicated cluster config. Recommended over the
phased per-role-keys/PKI plan; decision still deferred.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
